Numa Programming

NUMA Programming

Purpose

Guide agents through NUMA-aware programming: topology detection with numactl and sysfs, libnuma API (numa_alloc_onnode, mbind, set_mempolicy), process binding with numactl, NUMA-aware data structures, remote access diagnosis with perf stat, and lstopo visualization.

When to Use

• Multi-socket server shows poor scaling despite low CPU utilization
• Memory bandwidth saturation on one NUMA node
• Binding database or cache process to local memory
• Designing per-node freelists or sharded allocators
• Measuring remote vs local memory access latency
• Tuning HPC, DPDK, or custom allocator for socket locality

Workflow

1. Topology detection

# Hardware topology summary
numactl --hardware

# Detailed topology with distances
lstopo --of console

# sysfs nodes
ls /sys/devices/system/node/
cat /sys/devices/system/node/node0/meminfo
cat /sys/devices/system/node/node0/cpulist

Typical output:

available: 2 nodes (0-1)
node 0 cpus: 0-15
node 0 size: 65536 MB
node 1 cpus: 16-31
node 1 size: 65536 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10

Distance 10 = local, higher = remote (cross-socket).

2. Process binding with numactl

# Bind to node 0 CPUs and memory
numactl --cpunodebind=0 --membind=0 ./myapp

# Interleave memory across all nodes
numactl --interleave=all ./myapp

# Preferred node (fallback if full)
numactl --preferred=0 ./myapp

# Show process NUMA policy
numactl --show
cat /proc/self/numa_maps

3. libnuma API

#include 
#include 
#include 

int main(void) {
    if (numa_available() free_list[node];
    if (blk) {
        p->free_list[node] = *(void **)blk;
        return blk;
    }
    return numa_alloc_onnode(BLOCK_SIZE, node);
}

Pin threads to cores on the same node as their pool.

5. Thread affinity alignment

# Pin thread 0 to CPU 0 (node 0), allocate on node 0
numactl --cpunodebind=0 --membind=0 ./worker --id 0
numactl --cpunodebind=1 --membind=1 ./worker --id 1

#include 
#include 

cpu_set_t cpuset;
CPU_ZERO(&cpuset);
CPU_SET(target_cpu, &cpuset);
pthread_setaffinity_np(pthread_self(), sizeof(cpuset), &cpuset);

6. Remote access diagnosis

# Cache misses often spike with remote memory
perf stat -e cache-misses,cache-references,node-load-misses \
    numactl --cpunodebind=0 --membind=1 ./myapp

# Compare local vs remote binding
perf stat numactl --cpunodebind=0 --membind=0 ./myapp
perf stat numactl --cpunodebind=0 --membind=1 ./myapp

# NUMA hit/miss stats (if available)
perf stat -e node-loads,node-load-misses,node-stores ./myapp

7. Measuring remote penalty

// Microbenchmark: touch 1GB on local vs remote node
clock_t start = clock();
for (size_t i = 0; i < size; i += 4096)
    sum += ((char *)mem)[i];

Expect 1.5–3x slowdown for remote access depending on interconnect (QPI/UPI/Infinity Fabric).

8. lstopo visualization

# Graphical (if X11)
lstopo

# Text with memory/PCI
lstopo --of ascii

# Export for documentation
lstopo file.png

Shows: NUMA nodes, cores, caches, PCI devices — essential for DPDK NIC placement.

9. Decision tree

Poor scaling on multi-socket?
├── Check numactl --hardware
├── Verify thread and memory on same node
├── perf stat node-load-misses
├── Remote misses high?
│   ├── numactl --membind=local
│   └── Per-node data partitioning
└── Still slow → memory bandwidth bound; reduce sharing

Common Problems

| Symptom | Cause | Fix | |---------|-------|-----| | OOM on one node despite free RAM elsewhere | MPOLBIND too strict | Use --preferred or interleave | | 2x slower after scaling threads | Remote memory access | numactl --membind matching CPU node | | Inconsistent benchmark results | OS migrated pages | mbind MPOLBIND; mlock if needed | | DPDK NIC on wrong socket | PCI far from CPU | lstopo; bind EAL to local socket | | libnuma not found | Package not installed | apt install libnuma-dev | | First-touch policy surprise | Alloc on node 0, run on node 1 | Allocate from bound thread |

Related Skills

• skills/allocators/custom-allocators — per-node pool allocators
• skills/async-io/dpdk — rte_malloc_socket, NIC NUMA locality
• skills/hpc/mpi — MPI process binding per NUMA node
• skills/profilers/hardware-counters — cache miss measurement
• skills/profilers/linux-perf — perf NUMA events
• skills/low-level-programming/cpu-cache-opt — cache locality fundamentals

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mohitmishra786
• Source: mohitmishra786/low-level-dev-skills
• License: MIT
• Homepage: https://www.lowleveldevskills.com

Install and usage instructions live in the source repository linked above.