Linux Kernel cgroupv2: SCFS Scheduler - Throttled usec Accounting for Multiple Threads
In Linux kernel, cgroupv2 (control groups version 2) is a powerful hierarchical framework that enables system administrators and developers to manage and isolate system resources, such as CPU, memory, and I/O, among others. One essential aspect of cgroupv2 is its integration with the Completely Fair Scheduler (CFS), which fairly allocates CPU time among tasks running in different cgroups. This article will focus on how the CFS scheduler handles throttled usec accounting for multiple threads within a single cgroup.
Completely Fair Scheduler (CFS)
The Completely Fair Scheduler (CFS) is the default scheduling algorithm in Linux kernel. Its primary goal is to provide fairness - meaning, equally allocating CPU time among tasks. The scheduler implements this by assigning a virtual runtime to each task and then arranging them in a red-black tree, ordered by their vruntime. The task with the smallest vruntime will be allocated CPU time until it reaches a quantum time slice or gets preempted by a higher-priority task.
cgroupv2 and CFS Integration
cgroupv2 extends CFS functionality by introducing resource controllers (such as CPU, memory, and I/O) and hierarchically organizing them. Through this hierarchy, resources can be allocated and isolated, and tasks can belong to various groups based on the administrator's chosen criteria.
Throttling usec Accounting
cgroupv2 enables administrators to limit resources, usually CPU bandwidth, by applying a throttle expressed in microseconds (usec). However, when multiple sibling threads within the same cgroup reach the throttle limit, the kernel uses a sophisticated mechanism for allocating the remaining throttled time. This mechanism prevents starvation, where sibling threads get stuck and wait for resources while consuming more throttled time than necessary.
throttled_usec Tracking
For each cgroup, the kernel maintains a global throttled_usec variable that tracks throttled time in microseconds. The throttled_usec value represents the total amount of time the cgroup has spent beyond its throttle limit.
Each time a task belonging to a throttled cgroup gets scheduled, the kernel checks whether the throttle limit is active. If it is, the scheduler subtracts the throttle limit from the task's runtime quantum. This ensures that the task stays within the throttle limit. If a task consumes less time than its adjusted quantum, the remaining time is contributed back to the throttled_usec variable.
Remaining time Allocation for Multiple Threads
When there are multiple sibling threads sharing the same throttled cgroup and the throttle limit is reached, the cgroup's throttled_usec is distributed among them. By dividing the throttled_usec variable by the total number of runnable tasks in the throttled cgroup, the scheduler assigns each task a proportional share of throttled time. This distribution guarantees that threads within the same cgroup receive a fair chance of executing.
The Linux kernel cgroupv2's CFS scheduler offers an elaborate system for throttling usec accounting while ensuring multiple threads within the same cgroup receive a fair chance of resource utilization. The throttled_usec variable tracks the cumulative throttled time across all threads, while the remaining throttled time is shared among runnable tasks based on a fair proportional distribution.