cpu transformer training on zen 4 with avx512-bf16

the amd ryzen 7 7840hs supports avx-512 instructions, including avx512_bf16 for bfloat16 matrix work. that support makes cpu training less impractical than it would be with fp32 alone. it does not make this laptop processor competitive with the tested gpus.

this page reports measurements from three otherwise identical zen 4 machines. it also records two easy ways to spoil a cpu benchmark: using an unthreaded basic linear algebra subprograms (blas) library and letting unrelated work consume a core.

hardware

the tests used three identical machines with an amd ryzen 7 7840hs: 8 cores, 16 threads, a measured boost clock of 5.14 ghz, 16 mb of l3 cache, dual-channel ddr5-5600 memory, and ubuntu 26.04. amd specifies a maximum boost clock of 5.1 ghz; small differences in reported clocks can come from the measurement method.¹

relevant cpu flags: avx512f, avx512bw, avx512vl, avx512_bf16, avx512_vnni, avx512vbmi.

using a measured all-core clock near 4.2 ghz gives an estimated fp32 ceiling of 1.07 tflop/s. this is an arithmetic ceiling, not expected application throughput.

the packaging trap

before benchmarking pytorch, a plain sgemm through the distribution’s numpy gave:

the flat result across thread counts is consistent with an unthreaded reference blas. at about 1.3% of the estimated fp32 ceiling, it says more about the installed library than the processor. setting OMP_NUM_THREADS or OPENBLAS_NUM_THREADS changes nothing when the linked blas has no threading.

before trusting a cpu benchmark, confirm that a large general matrix multiplication (gemm) scales across threads. the pytorch wheel used here included onednn and its own threading support, so the training test did not use numpy’s slow blas path.³

measured training throughput

the test used transformer encoders with 4,871,585 and 21,308,033 parameters, a batch size of 16, and a sequence length of 256. training used the AdamW optimizer and PyTorch 2.9.1+cpu. the table reports tokens per second.

bf16 is worth using

PyTorch’s automatic mixed-precision context, autocast, delivered 1.62-1.87 times the fp32 throughput across the three machines. it was the largest configuration gain measured in this test and requires wrapping the step in torch.autocast(device_type='cpu', dtype=torch.bfloat16).

unlike fp16, bf16 has the same 8-bit exponent as fp32. it has less precision, but its wide numeric range usually removes the need for loss scaling.⁴

smt reduces throughput

at 8 threads (one per physical core) the 5m model reaches 13,255 tokens/s; at 16 threads it drops to 12,734. the pattern holds on all three machines and in both dtypes.

the measurements show that a second hardware thread did not help the bf16 kernels. scheduling and cache contention are plausible causes, but these tests did not isolate either one. start with one thread per physical core, then measure. PyTorch’s own tuning guide also recommends avoiding cpu oversubscription and controlling OpenMP thread counts.²

OMP_NUM_THREADS=8 python train.py

the fp32 numbers show a weaker version of the opposite trend, which is consistent with fp32 being less able to saturate the vector units in the first place.

governor and background load

the machine using the performance governor was about 5% faster than one comparable machine using powersave (13,255 versus 12,577 tokens/s). because this comparison used different hosts, treat 5% as an observed difference, not a controlled estimate of the governor’s effect.

background load matters much more. one machine ran a busy-wait shell loop with no sleep:

# burns a full core indefinitely
until grep -q MARKER /dev/null 2>/dev/null; do :; done

with that loop running, 16-thread throughput fell from 7,740 to 2,301 tokens/s, a 70% loss. always check uptime and top processes before recording a cpu benchmark.

comparison to gpu

the same benchmark on an rtx 4060 ti reached 197,924 tokens/s at 5m and 80,716 at 21m, versus 13,255 and 5,178 on zen 4 — roughly 15x. against an rtx 5070 ti running a full production pipeline the gap widens to 15-50x depending on how much of the pipeline is included.

projected onto a complete 3,029,577,143-token epoch, cpu training lands at 2.6-5.4 days for the 5m model and 6.8-22.7 days for the 21m model, against 4.27 hours and 17.2 hours respectively on an rtx 5070 ti.

where cpu is still useful

for this workload, the measured gap makes cpu pretraining a poor use of time. spare cpu machines can still handle adjacent work:

• evaluation passes on frozen checkpoints, which are inference-only and small
• dataset materialization, tokenization, and corpus census, which are cpu and i/o bound anyway
• manifest construction and validation

moving that work to spare cpu machines frees gpu memory and host cores on the training box. for programs where evaluation is queued behind training because it cannot share gpu memory with a live trainer, this is the practical win.

one caveat: evaluating checkpoints on cpu produces slightly different floating-point results than gpu. as long as every checkpoint in a given comparison is evaluated on the same backend, the comparison remains internally consistent; mixing backends within one comparison does not.

reproducing

# confirm the blas actually threads before trusting any number
python -c "
import numpy as np, time
a = np.random.randn(8192, 2048).astype(np.float32)
b = np.random.randn(2048, 2048).astype(np.float32)
a @ b
t = time.perf_counter()
for _ in range(5): a @ b
print('%.1f GFLOP/s' % (2*8192*2048*2048*5/(time.perf_counter()-t)/1e9))
"

# verify the bf16 instructions are present
grep -o 'avx512_bf16\|avx512_vnni' /proc/cpuinfo | sort -u

references

[1] amd. zen 4 architecture.

[2] pytorch. cpu inference and training optimizations.

[3] oneapi. onednn documentation.

[4] google. bfloat16: the secret to high performance on cloud tpus.