To optimize low latency systems on Linux, you must align your application’s CPU affinity with the physical Non-Uniform Memory Access (NUMA) topology. By binding threads to specific cores within a single NUMA node and disabling hyperthreading where appropriate, you eliminate cross-node memory traffic and cache thrashing, which are primary causes of unpredictable latency spikes in high-performance computing environments.
Understanding NUMA Topology
Non-Uniform Memory Access (NUMA) architectures divide the system into nodes, each containing one or more CPUs and their local memory controllers. In a NUMA system, accessing memory attached to a local CPU is significantly faster than accessing memory attached to a remote CPU. For low latency applications, such as high-frequency trading engines or real-time data processing pipelines, this difference in access time can be critical.
You can inspect the NUMA topology using the
numactl --hardware command. This output reveals how many nodes exist, how many CPUs belong to each node, and the memory distribution. Understanding this layout is the first step in designing an efficient core allocation strategy. If your application spans multiple NUMA nodes, it may suffer from increased latency due to inter-node communication overhead. Therefore, the goal is often to confine the application's workload to a single NUMA node whenever possible.
Selecting the Right CPU Cores
Once you understand the topology, you need to select the optimal set of CPU cores for your application. The selection process involves balancing between physical cores and logical processors (hyperthreads). Hyperthreading allows a single physical core to execute two threads simultaneously by duplicating certain parts of the CPU. While this increases throughput for general-purpose workloads, it can introduce jitter and latency variability for latency-sensitive tasks because both threads compete for the same execution resources.
For strict low-latency requirements, it is often recommended to disable hyperthreading or to bind threads to physical cores only. You can identify physical cores versus logical threads by examining the CPU information in
/proc/cpuinfo or using tools like
lscpu. Look for the
core id and
cpu cores fields. Physical cores share the same
core id, while logical threads have unique IDs but share the same core ID. Binding to unique core IDs ensures that your threads do not compete for execution units on the same physical core.
Additionally, consider the relationship between CPU caches and NUMA nodes. L3 caches are often shared among cores within a NUMA node. Keeping related threads close to each other within the same node maximizes cache locality, further reducing latency. This is particularly important for applications with heavy data sharing between threads.
Implementing CPU Affinity and Memory Policies
The Linux kernel provides mechanisms to enforce CPU affinity and memory policies. The
numactl utility is essential for launching processes with specific NUMA preferences. Using
numactl --cpunodebind, you can restrict a process to run only on CPUs belonging to a specific NUMA node. Similarly,
--membind ensures that memory allocations come from the local node’s memory banks.
For finer control, you can use the
SCHED_SETAFFINITY system call within your application code to pin individual threads to specific CPU masks. This is useful when different threads have different latency requirements or computational characteristics. For example, a thread handling network I/O might be pinned to a different core than a thread performing complex calculations, allowing them to operate independently without interference.
It is also crucial to configure the kernel scheduler correctly. You may want to isolate cores from the general scheduler using the
isolcpus boot parameter. This prevents the kernel from scheduling background tasks, such as interrupts or daemon processes, on the cores reserved for your low-latency application. This isolation reduces context switches and interrupt-induced latency spikes. Note that managing isolated cores requires careful configuration to ensure that necessary system services still have access to available CPUs. For more details on kernel task isolation, see our guide on
Ubuntu Kernel Sending Task to Isolated Core.
Monitoring and Validation
After configuring your system, validate the performance gains using monitoring tools. Tools like
perf,
htop, and
numastat provide insights into CPU usage, cache misses, and memory access patterns. Specifically, look for a reduction in remote memory accesses and cache miss rates after implementing NUMA binding. A successful configuration should show that most memory accesses are local to the NUMA node hosting the executing thread.
If you observe unexpected latency spikes, check for interrupt migration. Interrupts can be migrated between CPUs, potentially moving away from the isolated cores. Use
irqbalance or manually configure IRQ affinities to keep interrupts on the isolated cores if they are part of your workload, or off them entirely if they cause contention. Additionally, ensure that power management settings do not downclock the CPUs unexpectedly. Power saving states can introduce latency during frequency transitions. You may need to set the CPU governor to
performance using
cpupower or similar tools. Be aware that power settings can vary based on user privileges; for instance, checking why
cpupower reports different speeds and settings depending on if I'm root is a common troubleshooting step when power management behaves inconsistently.
Frequently Asked Questions
How do I check my current NUMA topology?
Run
numactl --hardware in the terminal to display the available NUMA nodes, their associated CPUs, and memory ranges. This output helps you map out which cores are physically close to each other and share memory bandwidth.
Is hyperthreading always bad for low latency?
Not always, but it often introduces jitter. Hyperthreading shares execution resources, which can lead to unpredictable delays if two threads contend for the same functional units. For deterministic low latency, disabling it or strictly binding to physical cores is usually safer.
What does isolcpus do exactly?
The
isolcpus kernel parameter removes specified CPUs from the default scheduler domain. This prevents the OS from scheduling non-real-time tasks on those cores, reducing noise and context switches for your dedicated low-latency threads.
Can I change NUMA bindings at runtime?
Yes, you can use
taskset to modify CPU affinity for running processes or implement
SCHED_SETAFFINITY calls within your application. However, changing memory policies dynamically is more complex and may require restarting the application or using advanced memory management APIs.
Why are my latency spikes still occurring despite proper binding?
Check for interrupt handling overhead. If hardware interrupts are being handled on the same cores as your application, they can cause significant latency. Consider migrating IRQs to different cores or using dedicated interrupt handling threads to decouple them from your critical path.