NGINX Server Failing to Handle Certain Number of New Connections on Ubuntu VM

NGINX failing to accept new connections on an Ubuntu VM is typically caused by exhausted system file descriptors, kernel network limits, or application-level timeouts rather than a DDoS attack. The solution involves increasing ulimit values, adjusting net.core.somaxconn, and verifying upstream health checks to ensure the server can sustain the expected load without dropping packets.

Understanding Connection Limits in NGINX

When an NGINX server stops accepting new connections, it often returns a 502 Bad Gateway or 504 Gateway Timeout error, or simply hangs until the client times out. This behavior indicates that the web server process itself is still running but has reached a bottleneck preventing it from establishing new sockets. On an Ubuntu Virtual Machine (VM), these bottlenecks are frequently related to resource allocation limits imposed by the operating system or the hypervisor, rather than malicious traffic. It is crucial to distinguish between a genuine Distributed Denial of Service (DDoS) attack and legitimate traffic hitting infrastructure limits. While DDoS attacks can overwhelm servers, many administrators incorrectly diagnose simple connection drops as attacks when the root cause is actually misconfigured system limits. Idle resources or high latency do not necessarily indicate an active attack; they often signal that the kernel’s network stack is saturated or that file descriptor tables are full. Therefore, troubleshooting should begin with local configuration audits before implementing complex firewall rules or DDoS mitigation services. For more context on managing NGINX configurations on Ubuntu, you might find it useful to read about Why is my domain URL still rendering a 404 page after uninstalling nginx? which covers common service state issues. Additionally, understanding how your React and Express applications communicate through NGINX can help isolate whether the issue lies in the proxy layer or the backend, as discussed in Troubleshooting React/Express on Nginx/Ubuntu.

Increasing File Descriptor Limits

One of the most common reasons for connection failures is hitting the maximum number of open files allowed by the OS. Each incoming connection requires a file descriptor. If the limit is too low, NGINX will refuse new connections even if CPU and memory are available. First, check the current limit for the NGINX user by running:

cat /proc/$(pidof nginx)/limits | grep "Max open files"

If this value is low (e.g., 1024), you need to increase it. Edit the /etc/security/limits.conf file and add the following lines:

nginx soft nofile 65535
nginx hard nofile 65535

You must also update the systemd service definition to override these defaults. Create or edit /etc/systemd/system/nginx.service.d/override.conf and add:

[Service]
LimitNOFILE=65535

After saving, reload the systemd daemon and restart NGINX:

sudo systemctl daemon-reload
sudo systemctl restart nginx

Verify the change by checking the process limits again. This adjustment allows NGINX to handle thousands of simultaneous connections without exhausting the file table.

Adjusting Kernel Network Parameters

Beyond file descriptors, the Linux kernel has specific parameters controlling how many pending connections can be queued for acceptance. If this queue fills up, new TCP handshakes are dropped immediately. Check the current setting for somaxconn using:

sysctl net.core.somaxconn

The default is often 128, which is insufficient for high-traffic servers. Increase this value to 4096 or higher by adding the following line to /etc/sysctl.conf:

net.core.somaxconn = 4096

Apply the changes immediately with:

sudo sysctl -p

Additionally, ensure that the backlog queue in your NGINX configuration matches this kernel limit. In your server block or http block, set:

listen 80 backlog=4096;

This alignment ensures that the OS and NGINX agree on the maximum queue depth, preventing silent packet drops during traffic spikes.

Troubleshooting Upstream Timeouts

Sometimes the issue is not with NGINX itself but with the backend application it proxies to. If the upstream server (like Node.js, Python, or PHP-FPM) takes too long to respond, NGINX may close the connection or return a timeout error, making it appear as though it cannot handle new requests. Review your NGINX configuration for proxy_read_timeout and proxy_connect_timeout directives. Default values are often set to 60 seconds, which may be too short for complex backend operations. Increase these values if necessary:

proxy_read_timeout 300s;
proxy_connect_timeout 300s;

Monitor the backend logs to see if processes are crashing or hanging. If the backend is restarting frequently due to memory leaks, NGINX will fail to connect because the upstream socket is unavailable. Use tools like htop or free -m to monitor VM memory usage, ensuring the guest OS has sufficient RAM allocated. For tips on optimizing your Ubuntu VM environment, consider reviewing Run Ubuntu VM Guest without Installation to understand resource constraints better.

Verifying Firewall and Security Groups

Finally, ensure that the underlying firewall is not interfering with new connections. On Ubuntu, ufw (Uncomplicated Firewall) is commonly used. Check its status with:

sudo ufw status

Ensure that port 80 (HTTP) and 443 (HTTPS) are allowed. If you are using a cloud provider, verify that the security group or firewall rules associated with the VM instance permit inbound traffic on these ports. Misconfigured security groups can sometimes allow existing connections while blocking new ones, leading to intermittent failures.

Frequently Asked Questions

How do I know if my NGINX server is under a DDoS attack?

A true DDoS attack usually shows up as a massive spike in traffic from many different IP addresses, often with identical request patterns. If you see high CPU usage combined with normal traffic levels and connection refusals, it is likely a configuration limit issue rather than an attack.

What causes the "Too Many Open Files" error in NGINX?

This error occurs when the operating system reaches its limit for simultaneously open file descriptors. Since each network connection consumes a file descriptor, exceeding this limit prevents NGINX from accepting new clients. Increasing the nofile limit in limits.conf resolves this.

Why does my NGINX return 502 Bad Gateway errors randomly?

Random 502 errors often indicate that the upstream backend server is temporarily unavailable or overloaded. This can happen if the backend crashes, runs out of memory, or fails to respond within the configured timeout period. Checking backend logs is essential to confirm this.

Can I fix connection issues without restarting NGINX?

While some settings like sysctl parameters can be applied dynamically without a full restart, changing file descriptor limits or major configuration blocks usually requires reloading or restarting the service. Always test changes in a staging environment first to avoid downtime.

How does the backlog parameter affect NGINX performance?

The backlog parameter defines the maximum length of the queue for partially completed TCP connections. If the backlog is too small, new connections may be rejected during traffic spikes even if the server has capacity. Aligning this with net.core.somaxconn ensures optimal handling of concurrent requests.