Unable to permanently adjust hard/soft open files limit on Debian 12 with kernel 6.1.0-32-amd64

Permanently adjusting the open files limit on Debian 12 requires modifying /etc/security/limits.conf and ensuring PAM initialization occurs for all login sessions. The system must also respect the kernel’s maximum hard limit defined in /proc/sys/fs/nr_open. This approach resolves issues where changes appear to take effect only temporarily or are ignored by specific services.

Understanding the Limit Hierarchy

Linux manages file descriptors through a hierarchy of limits enforced by the kernel and the Pluggable Authentication Modules (PAM) system. When a user logs in, PAM reads configuration files to set resource limits for that session. If you modify limits using ulimit in an interactive shell, those changes apply only to the current process tree and are lost upon logout. To make them permanent, you must configure the system-wide defaults that PAM applies during authentication. The relevant settings are the soft limit, which is the default limit imposed on new processes, and the hard limit, which acts as the ceiling that the soft limit cannot exceed without root privileges. In Debian 12, the PAM stack includes the module pam_limits.so, which is responsible for reading these configurations. If this module is not properly loaded or if the syntax in the configuration file is incorrect, the limits will remain at their default values, typically 1024 for both soft and hard limits. It is crucial to understand that there is a global kernel constraint known as nr_open. This value defines the maximum number of file descriptors any single process can open. Even if you set a very high limit in limits.conf, the effective limit cannot exceed nr_open. You can check this value by reading the file /proc/sys/fs/nr_open. On most modern Debian systems, this is set to 1048576, which is sufficient for almost all use cases. If you attempt to set a limit higher than this, the system will silently cap it or fail to apply it correctly.

Configuring /etc/security/limits.conf

The primary method for setting persistent limits is editing the /etc/security/limits.conf file. This file uses a specific syntax that distinguishes between users, groups, and wildcards. To adjust limits for a specific user, you add lines to this file. For example, to set the soft limit to 65536 and the hard limit to 65536 for the user myuser, you would add the following line:

myuser soft nofile 65536
myuser hard nofile 65536

You can also apply these limits to all users by using the asterisk wildcard character (*). However, be cautious when using wildcards, as this affects every account on the system, including service accounts that might not require such high limits. It is generally safer to target specific users or groups. For instance, to apply limits to a group named developers, you would prefix the username field with an at symbol (@):

@developers soft nofile 65536
@developers hard nofile 65536

After saving the file, the changes do not take effect immediately for existing sessions. You must log out and log back in for PAM to re-read the configuration and apply the new limits. You can verify the change by running ulimit -n in a new shell session. If you are configuring limits for systemd services, the process is slightly different. Systemd does not always source limits.conf in the same way interactive shells do. For systemd units, you should use the LimitNOFILE directive in the service’s unit file. This ensures that the service starts with the correct file descriptor limits regardless of how the administrator logged into the system.

Verifying Kernel Constraints

Before applying user-level limits, ensure that the kernel allows them. Check the current maximum allowed value by running:

cat /proc/sys/fs/nr_open

If this value is lower than your desired limit, you must increase it. You can do this temporarily by writing to the proc file:

echo 1048576 | sudo tee /proc/sys/fs/nr_open

To make this change persistent across reboots, add the following line to /etc/sysctl.conf:

fs.nr_open = 1048576

Then run sudo sysctl -p to apply the changes. This step is often overlooked but is critical because even with correct PAM configuration, the kernel will reject any attempt to allocate more file descriptors than nr_open permits.

Common Pitfalls and Debugging

One common issue is that changes to limits.conf do not seem to apply. This is often due to PAM not being invoked for the specific type of login. For example, SSH sessions usually invoke PAM, but some background services or cron jobs might not. If you are testing limits via SSH, ensure you completely log out and log back in. Another pitfall involves the interaction between different PAM modules. Ensure that pam_limits.so is included in the /etc/pam.d/common-session file. It should typically appear as:

session required pam_limits.so

If this line is missing or commented out, the limits configuration file will be ignored entirely. You can also check the system logs for errors related to PAM if the limits are still not applying after a reboot. For detailed verification of open file descriptors, avoid using tools like ss, which display network socket statistics. Instead, use lsof or inspect the /proc/[pid]/fdinfo directory to see exactly which files are open and how many descriptors a process is consuming. This provides accurate insight into whether your limit adjustments are functioning as intended.

Frequently Asked Questions

Why did my ulimit changes disappear after a reboot?

Changes made with the ulimit command are temporary and only affect the current shell session. To make them permanent, you must edit /etc/security/limits.conf or configure systemd unit files. Additionally, ensure that PAM is correctly initialized for your login method.

How do I check the current open file limit for a running process?

You can view the open file descriptors for a specific process ID by listing the contents of /proc/[pid]/fd. Alternatively, use the lsof -p [pid] command to list open files. Do not use ss as it only shows network sockets, not general file descriptors.

Can I set the open files limit higher than nr_open?

No. The nr_open value in the kernel represents the absolute maximum number of file descriptors any single process can have. If you try to set a limit in limits.conf that exceeds nr_open, the system will either cap it at nr_open or fail to apply the setting. You must increase nr_open first via sysctl.

Do systemd services respect /etc/security/limits.conf?

Generally, no. Systemd services start independently of user login sessions and may not trigger the PAM limits module. To set file descriptor limits for systemd services, you should use the LimitNOFILE= directive directly within the service’s unit file located in /etc/systemd/system/.

What happens if I set the soft limit higher than the hard limit?

The system will enforce the hard limit as the maximum. A soft limit cannot exceed the hard limit. If you attempt to set a soft limit higher than the hard limit, the kernel will typically ignore the soft limit setting or cap it at the hard limit value. Always ensure the soft limit is less than or equal to the hard limit.