CPU, NUMA, and Data Locality: Hidden Bottlenecks in Multi-GPU Training

A common attempt to improve throughput is to increase num_workers in PyTorch or TensorFlow. That can help, but only when the additional loader workers are scheduled efficiently.

Operating systems generally schedule processes for overall balance, not for GPU-local data delivery. In NUMA systems, this means data-loader workers may be spread across cores and memory domains without regard to where the target GPU is attached.

Why this matters

If loader workers are physically distant from the memory controller or PCIe root that serves the target GPU, the system does more work to prepare each batch:

• •preprocessing becomes less locality-aware
• •pinned-memory buffers may still be populated remotely
• •host-side contention increases
• •inter-step idle gaps can become more frequent

Practical implication

Increasing worker count can improve throughput, but it can also reduce it if those workers amplify cross-node memory traffic. More parallelism on the host side is not always better when topology is working against locality.

Pinned memory helps improve transfer efficiency, but it does not by itself solve poor CPU affinity or remote memory allocation. Those remain topology problems.

Spec sheets emphasize device memory bandwidth—how fast each GPU can read and write its own VRAM (HBM or GDDR), often quoted in TB/s. That number describes on-device tensor traffic: moving activations and weights that already live in GPU memory. It is not the same quantity as GPU-to-GPU throughput—the rate at which tensors move between GPUs during collectives, gradient sync, or peer copies over NVLink, PCIe peer-to-peer, or paths that involve the host.

A system can have excellent per-GPU VRAM bandwidth and still scale poorly across devices when the interconnect is the limiting factor. Conversely, a fast link between GPUs does not fix host-side staging, NUMA crossings, or slow storage—the bottlenecks earlier sections of this article describe.

Practical distinction

When reasoning about multi-GPU performance, keep three layers separate:

• •Device VRAM bandwidth: on-package memory to SMs (per-GPU tensor work on resident data)
• •GPU-to-GPU bandwidth: links and topology between devices (collectives, P2P, NVLink where present)
• •Host memory bandwidth: DDR channels and NUMA—how the CPU feeds the pipeline (next section)

Training runs can shift which layer dominates. If you only optimize for the first, you may miss the second—often the missing link when scaling from one GPU to several.

Core count is often overemphasized in workstation planning. For many multi-GPU training workloads, host memory bandwidth and channel architecture are more consequential than raw CPU compute capacity.

Loader workers, pinned-memory staging, decompression, and I/O coordination all generate sustained traffic through the CPU memory subsystem. Platforms with more memory channels are generally better able to feed dense GPU configurations without starving the host side of bandwidth.

System-level effect

When memory bandwidth becomes constrained:

• •loader throughput can flatten
• •staging buffers refill less consistently
• •GPUs spend more time waiting for prepared batches
• •scaling degrades as more devices are added

This is why CPU selection in AI systems should be evaluated partly as a memory and I/O design decision, not only as a compute choice. A processor with many cores but limited memory bandwidth can support light workloads adequately while still underfeeding a dense multi-GPU configuration.

NUMA and locality problems are difficult to reason about if the topology is treated as opaque. In practice, the first step is to make the layout visible.

Useful tools include:

nvidia-smi topo -m
numactl --hardware
lscpu

These tools help answer questions such as:

• •which GPUs share closer paths to one another
• •how NUMA nodes are exposed to the OS
• •how CPUs, memory domains, and devices are arranged

For execution, NUMA-aware binding can reduce unnecessary cross-node traffic. For example:

numactl --cpunodebind=0 --membind=0 python train.py

This does not guarantee optimal placement for every workload, but it provides a controlled starting point by keeping execution and allocation local to one NUMA node.

What to watch for

Common warning signs of topology-related inefficiency include:

• •uneven step times across GPUs
• •poor scaling when moving from one GPU to several
• •high reported utilization with disappointing throughput
• •one CPU domain much busier than another
• •transfer behavior that becomes more erratic under sustained load

Several patterns recur in underperforming multi-GPU systems:

• •treating system RAM as if it were uniformly local to all workers
• •assuming PCIe slot population alone guarantees efficient device locality
• •increasing loader worker counts without checking affinity or memory placement
• •selecting CPUs primarily by core count while ignoring memory channels
• •equating high per-GPU VRAM bandwidth (TB/s) with strong multi-GPU scaling without checking GPU-to-GPU paths
• •using GPU utilization as the main measure of training efficiency

These mistakes do not always produce obvious errors. More often, they show up as lower-than-expected throughput, unstable scaling, or synchronization overhead that is difficult to explain from GPU metrics alone.