Your Bottle application behind Gunicorn is likely failing to shut down gracefully because the worker processes are not handling the SIGTERM signal correctly, often due to blocking operations or missing signal handlers. This guide explains how to diagnose the hang and implement a robust shutdown sequence that ensures your server terminates cleanly when receiving termination requests from process managers like systemd or Docker.
Understanding Signal Propagation in Gunicorn
When you send a
SIGTERM to the Gunicorn master process, it forwards this signal to all active worker processes. In a standard synchronous setup, workers should catch this signal and begin their shutdown routine immediately. However, if your Bottle application contains long-running loops, database connections that do not respect timeouts, or global state that prevents immediate exit, the workers may ignore the signal or hang indefinitely. The master process will eventually kill the workers with
SIGKILL after its timeout expires, but this results in data loss or corrupted states. Understanding this propagation chain is critical for debugging why your application appears unresponsive during deployment updates or container restarts.
The Role of Worker Classes
The behavior of signal handling depends heavily on the worker class you have selected. The default sync workers are simple and execute one request at a time per process. If your code blocks the main thread, the signal handler cannot interrupt the execution flow until the current operation completes. Asyn workers like
gthread or
gevent offer better concurrency but introduce complexity in signal handling. If you are using async frameworks, ensure that your event loop is properly configured to handle interrupts. For most Bottle applications, which are typically synchronous, sticking to sync workers is recommended unless you have specific performance requirements that necessitate asynchronous processing.
Detecting Blocking Code in Your Application
The most common cause for a non-responsive shutdown is blocking I/O within your Bottle routes. If a route performs a heavy computation, waits on an external API without a timeout, or holds a lock indefinitely, the worker process remains occupied and cannot process the incoming
SIGTERM. To identify these bottlenecks, review your application logic for any long-running tasks. Look for functions that call external services without specifying a timeout parameter. Additionally, check for infinite loops or recursive calls that might never return control to the interpreter. By isolating and optimizing these sections, you allow the signal handler to execute promptly when the shutdown command is received.
Implementing Graceful Shutdown Hooks
Bottle provides a mechanism to register callbacks that run before the application shuts down. You can use the
@hook('shutdown') decorator to clean up resources such as database connections or file handles. This hook ensures that your application releases locks and flushes buffers before exiting. It is essential to keep these hooks lightweight to avoid delaying the shutdown process. If your shutdown hook itself hangs, it will prevent the worker from terminating. Test your shutdown logic independently by simulating a termination signal in a development environment to verify that all cleanup tasks complete within a reasonable timeframe.
Configuring Gunicorn Timeouts and Settings
Gunicorn offers several configuration options to manage how it handles slow workers and shutdown signals. The
timeout setting defines how long a worker can be idle before being killed. While this helps prevent memory leaks, it does not directly address
SIGTERM responsiveness. Instead, focus on the
graceful_timeout parameter, which specifies how long Gunicorn waits for workers to finish their current request after receiving a shutdown signal. Setting this value appropriately ensures that Gunicorn gives your application enough time to clean up without hanging forever. Adjust these settings in your Gunicorn configuration file or via command-line arguments to match the expected shutdown duration of your application.
Monitoring Process States
Use system tools to monitor the state of your Gunicorn workers during a shutdown attempt. Commands like
ps aux | grep gunicorn can help you see if workers are stuck in a specific state. Checking logs for error messages related to signal handling can also provide clues. If workers are not dying, inspect the kernel logs for signs of deadlock or resource contention. Regularly monitoring these states allows you to detect issues early and adjust your configuration or code accordingly.
For more insights on Python debugging techniques, you might find our guide on
Troubleshooting pprint Usage in Python helpful for logging complex object states during troubleshooting.
Advanced Troubleshooting Steps
If basic configurations do not resolve the issue, consider upgrading Gunicorn to the latest version, as newer releases include improvements in signal handling and worker management. Additionally, review your operating system's ulimit settings to ensure that file descriptors and process limits are not restricting Gunicorn's ability to manage workers. In some cases, running Gunicorn under a different user account with appropriate permissions can resolve permission-related hangs. Always test changes in a staging environment before applying them to production to avoid unexpected downtime.
Frequently Asked Questions
Why does my Gunicorn master process die but workers remain?
This usually happens when the master process receives a signal that causes it to exit, but the workers are blocked and cannot respond to the subsequent termination signals sent by the master. Ensure that your application code does not block indefinitely and that graceful timeouts are configured correctly in Gunicorn.
How can I force Gunicorn to kill hung workers?
You can increase the
graceful_timeout value to give workers more time, but if they remain unresponsive, the master will eventually send a
SIGKILL after the
timeout period expires. To manually intervene, you can send a
SIGQUIT to the master process, which triggers a quicker shutdown sequence than
SIGTERM.
Does changing the worker class affect signal handling?
Yes, async worker classes like
gevent or
eventlet handle signals differently than sync workers. They may require additional configuration to ensure that signals are propagated correctly to the green threads. Switching to sync workers can simplify signal handling if you do not require high concurrency.
What is the best way to log shutdown events?
Use the
@hook('shutdown') decorator in your Bottle application to log cleanup activities. This ensures that logs are written before the process exits. Avoid using heavy logging libraries in this hook to prevent delays that could hinder the shutdown process.
Can Docker signals cause issues with Gunicorn?
Docker sends
SIGTERM to the main process of the container. If Gunicorn is not the entry point or if the entry point script does not forward signals correctly, Gunicorn may not receive the shutdown command. Ensure that your Dockerfile uses Gunicorn as the CMD or ENTRYPOINT and that it is running in the foreground.