Memory and Performance Guide for Low-Latency C++

In this post, I will read and share what I learnt from Ulrich Drepper's Memory Guide. I will also present foundational concepts and C++ code examples for low latency development.

RAM Fundamentals

SRAM vs DRAM

SRAM uses six transistor cells for cross coupled inverters: ultra-fast (~1 ns), no refresh needed, but low density and high cost. DRAM uses one transistor plus one capacitor: slower (~50 ns), requires periodic refresh, but offers high density and low cost.

// DRAM latency measurement (approximate)
#include 
#include 
volatile uint32_t* dram = reinterpret_cast(0x80000000);
auto start = std::chrono::high_resolution_clock::now();
uint32_t v = dram[0];  // DRAM read
auto end = std::chrono::high_resolution_clock::now();
std::cout << "DRAM read latency: "
          << std::chrono::duration_cast(end - start).count()
          << " ns\n";

6‑T SRAM Cell           1‑T DRAM Cell
     Vdd                    _____
      |                     |   |
   +‑[M1]‑+                 | C |
   |      |                 |___|
  [M2]   [M3]                 |
   |      |                   [T]
   +‑[M4]‑+                     |
  BL/BL   WL                   GND

DRAM Organization
DRAM is arranged in banks of rows and columns. Accessing a different row incurs a precharge, then RAS (row select), then CAS (column select), each adding delay.
```
Bank 0        Bank 1
+------+      +------+
|Row0  |      |Row0  |
|Row1  |      |Row1  |
| ...  |      | ...  |
+------+      +------+
```

Memory Subsystem Architecture

Traditional PC architecture: CPU <-> FSB <-> Northbridge (memory controller) <-> RAM; Northbridge <-> Southbridge <-> I/O devices. Modern CPUs integrate the memory controller on die, replacing the FSB with point to point links (Example: QPI/HT).

    +-------+      +--------+
    |  CPU  |<---->| Memory |
    +-------+      +--------+
         |              ^
     FSB |              |
         v      +---------------+
    +------------+ Northbridge  |
    | Southbridge|–– PCIe/SATA–—+
    +------------+
     /   |    \
   USB  PCIe  SATA

// Example: list cache directories on Linux
system("ls /sys/devices/system/cpu/cpu0/cache");

CPU Cache Hierarchy

Hierarchy from fastest to slowest:

[ Registers ]
     ↓
[  L1 Cache ]  (32 KB, 1–3 cycles)
     ↓
[  L2 Cache ]  (256–512 KB, ~10 cycles)
     ↓
[  L3 Cache ]  (2–16 MB, ~30–50 cycles)
     ↓
[   DRAM    ]  (GB-scale, ~50–200 ns)

Intel’s L3 cache is inclusive (holds copies of all L1/L2 lines), whhile AMD’s is non‑inclusive (evicts lines loaded into L1/L2).

#include 
alignas(64) char buf[64];
// Evict from L1 to force a miss
_mm_clflush(buf);

Cache Mapping & Associativity

Caches are set-associative: addresses are split into tag, set_index, and block_offset. Each set holds multiple "ways".

Set 42 (8-way):
+--------------------------------+
| Way 0 | Tag:0x1234 | Data       |
| Way 1 | Tag:0x5678 | Data       |
| ...                            |
+--------------------------------+

constexpr int BLOCK_BITS = 6;  // 64-byte lines
constexpr int SET_BITS   = 7;  // 128 sets
size_t addr = reinterpret_cast(&buf[0]);
size_t set  = (addr >> BLOCK_BITS) & ((1<> (BLOCK_BITS + SET_BITS);

Cache Coherence Protocols

Coherence protocols (MESI/MOESI/MESIF) maintain consistency across cores:
M (Modified), E (Exclusive), S (Shared), I (Invalid),
O (Owned, MOESI), F (Forward, MESIF). Avoid false sharing by aligning per thread data to cache line boundaries.

        +------+      +------+
   [S]  | CPU0 |<---->| CPU1 |  Shared clean
        +------+      +------+
                 ↑
          Snoops/sync messages

struct alignas(64) Counter { std::atomic value; };
Counter counters[NUM_THREADS];

Virtual Memory & TLB

Virtual addresses are translated via multi‑level page tables (up to 4 levels); without a TLB hit, each access may incur multiple memory references. The TLB (Translation Lookaside Buffer) caches VA→PA mappings to avoid table walks.

Virtual Addr [31:22] [21:12] [11:0]
     |         |        |
     v         v        v
 Page Dir   Page Tbl  Offset
     \        /
      Physical Page Base + Offset → Physical Addr

// Map a 2 MB huge page on Linux
void* p = mmap(nullptr, 2*1024*1024,
               PROT_READ|PROT_WRITE,
               MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB,
               -1, 0);

NUMA Considerations

On multi socket servers, local memory access is approximately 2× faster than remote. Use memory policies and OS APIs to control allocation:

Node 0       Node 1
+------+     +------+
| CPU0 |     | CPU1 |
| Mem0 |     | Mem1 |
+------+     +------+
   \\           //
    \\ interconnect

// Allocate 1 MB on NUMA node 1
#include 
void* mem = numa_alloc_onnode(1024*1024, 1);

Memory policies (Linux/libnuma): MPOL_BIND, MPOL_PREFERRED, MPOL_INTERLEAVE, MPOL_DEFAULT

Performance Optimization Techniques

Data Locality: Keep related data contiguous (arrays of structs) to maximize spatial & temporal reuse; align to 64 B cache lines.

Software Prefetch:

for (int i = 0; i < N; i += 16) {
  _mm_prefetch(&data[i+32], _MM_HINT_T0);
  // Process data[i]…data[i+15]
}

Non Temporal Stores: Use streaming stores for write only buffers:
```
#include 
_mm_stream_ps(dst + i, src_vec);
```

Programming Interfaces & Tools

Cache information via /sys/devices/system/cpu/cpu*/cache (sysfs).
NUMA APIs: mbind(), set_mempolicy(), sched_getcpu(), libnuma functions.
Huge pages (2 MB/1 GB) via hugetlbfs to reduce TLB misses.
Profiling & analysis: perf, valgrind, pahole, STREAM benchmark, custom microbenchmarks.
```
perf stat -e cache-misses,cache-references ./my_app
```

Modern Relevance:

Learning Ulrich Drepper’s memory guide is still super relevant today because the fundamentals of how memory interacts with the CPU have not really changed at all.
Stuff like cache hierarchies, memory latency, false sharing, and data alignment still directly affect performance especially in systems programming, game engines, or anything that is considered performance critical.
Even with newer hardware the principles he explains help you write code that doesn’t accidentally become bottlenecked just because of bad memory access patterns.
This is basically a very beginner friendly guide on how the CPU works under the hood, very useful.

← Back to blog