Linux Kernel cgroup v2: SCFS Scheduler

Throttled time accounting in Linux cgroup v2 applies to the entire control group, not individual threads. When a cgroup reaches its CPU bandwidth limit, all tasks within that group share the same throttling window. This ensures fair resource distribution across multiple threads running in the same hierarchy.

Understanding Cgroup V2 CPU Bandwidth Control

The Linux kernel’s cgroup v2 subsystem provides robust mechanisms for controlling CPU usage through bandwidth limits. Unlike earlier versions, cgroup v2 simplifies the interface while maintaining precise control over resource allocation. The core mechanism relies on two parameters: cgroup.max.cpu.weight for priority-based sharing and cgroup.max.bandwidth for hard limits. When you set a bandwidth limit, such as max 500ms 1000ms, you are defining a quota period. The kernel schedules work for all tasks in the cgroup until the allocated time (quota) is exhausted within that period. Once the quota is used up, the cgroup becomes throttled. No new scheduling credits are granted until the next period begins. This global throttle state affects every task inside the group simultaneously.

The Role of SCFS in Modern Kernels

It is important to clarify that "SCFS" is not a standard scheduler component in mainstream Linux kernels. Some users may confuse this with the Completely Fair Scheduler (CFS), which is the default scheduler for general-purpose workloads. CFS handles the actual process scheduling decisions, while cgroups provide the constraints. If you encountered references to SCFS, it might be from a specific patchset, a vendor-specific modification, or a misunderstanding of terminology. For standard distributions like Ubuntu, Debian, or RHEL, you are interacting with CFS under cgroup v2 management. For more details on how CPU throttling interacts with multiple threads in cgroup v2, see our guide on Linux Kernel cgroup v2 and SCFS Scheduler: CPU Throttling and Multiple Threads. Note that while the title mentions SCFS, the underlying principles of cgroup throttling remain consistent across standard CFS implementations.

How Throttling Affects Multiple Threads

A common misconception is that each thread gets an independent throttle period. In reality, the throttle period is a property of the cgroup, not the task. If your application spawns ten threads within a cgroup limited to 50% of one CPU core, those ten threads collectively consume that 50%. Here is how the accounting works:

1. Global Quota: The cgroup has a single usage_usec counter that tracks total CPU time consumed by all tasks combined.
2. Scheduling Decision: When the scheduler runs, it checks if the cgroup has remaining budget. If yes, any runnable task in the cgroup can run.
3. Throttle Trigger: Once usage_usec exceeds the quota, the cgroup is marked as throttled. The scheduler stops picking tasks from this cgroup until the period resets.
4. Period Reset: After the specified interval (e.g., 1000ms), the quota refreshes, and tasks can resume execution.

This design prevents a single thread from monopolizing resources while other threads starve. It also simplifies monitoring because you only need to watch one metric per cgroup to understand overall utilization.

Monitoring Throttling Events

To observe this behavior, you can inspect the cgroup.stat file located in your cgroup directory. Look for fields like nr_throttled and throttled_time. These counters increment when the cgroup hits its limit. If you see rapid increases in nr_throttled, it indicates that your workload is exceeding the allocated bandwidth. You can also use bpftrace or perf to trace scheduler events. Specifically, look for sched_stat_sleep and sched_switch events filtered by cgroup ID. This will show you how long tasks wait due to throttling versus waiting for I/O or locks.

Best Practices for Multi-Threaded Applications

When designing multi-threaded applications that require strict CPU limits, consider these strategies:

• Group Related Tasks: Place all threads of a single service into the same cgroup. This allows them to share the bandwidth pool efficiently.
• Avoid Nested Limits: Do not set strict limits on child cgroups if the parent already enforces a tight cap. This can lead to unpredictable behavior and increased overhead.
• Monitor Aggregate Usage: Use tools like systemd-cgtop or cgroups-ng to view aggregate CPU usage per cgroup rather than per process.
• Adjust Periods Wisely: Shorter periods (e.g., 100ms) allow for quicker recovery but increase scheduling overhead. Longer periods (e.g., 1000ms) smooth out bursts but delay response to changes.

For information on maintaining kernel stability while managing these configurations, refer to our article on Upgrading Linux Kernel Version without Compiling Everything. Keeping your kernel updated ensures you have the latest fixes for cgroup accounting bugs.

Common Pitfalls and Debugging

One frequent issue arises when developers assume that setting nice values overrides cgroup limits. While nice affects priority within the available bandwidth, it does not bypass the hard cap. If a cgroup is throttled, high-priority tasks must still wait. Another pitfall involves mixing cgroup v1 and v2 controllers. Ensure your system is using unified hierarchy mode. You can check this by verifying that /sys/fs/cgroup shows no separate cpu or memory directories. If they exist separately, you are likely in legacy mode, which behaves differently regarding throttling. If you encounter errors during configuration, such as permission denied messages, ensure your user has appropriate capabilities. Root access or specific CAP_SYS_ADMIN privileges are often required to modify cgroup settings. For troubleshooting package-related issues that might interfere with kernel modules, see our guide on Fixing E: Sub-process /usr/bin/dpkg returned an error code (1) on Kali Linux.

Frequently Asked Questions

Does each thread get its own throttle period?

No, the throttle period is shared across all tasks within the same cgroup. The kernel tracks total usage for the group, so once the collective quota is exhausted, all threads are paused until the next period starts.

Can I assign different quotas to individual threads?

Not directly within a single cgroup. To give individual threads different limits, you must place each thread in its own cgroup. However, this adds complexity and overhead, so it is usually better to manage limits at the group level.

What happens if a thread blocks on I/O while throttled?

I/O blocking is independent of CPU throttling. If a thread waits for disk or network I/O, it does not consume CPU time, so it does not contribute to the throttle counter. It can resume immediately when I/O completes, provided the cgroup has remaining budget.

Is SCFS a standard Linux scheduler?

No, SCFS is not a standard component of the mainline Linux kernel. You are likely referring to the Completely Fair Scheduler (CFS). Standard distributions use CFS for general task scheduling under cgroup v2 constraints.

How do I reset throttling counters?

Throttling counters reset automatically when the bandwidth period expires. You do not need to manually clear them. However, you can read the current values from cgroup.stat to monitor cumulative usage since the last boot or cgroup creation.