Benchmark environment — getting interpretable results

Before you start optimizing code, you need to make sure your benchmark results are actually meaningful. On a modern multi-core server or laptop, the operating system and hardware introduce so much noise that two consecutive runs of the exact same binary can differ significantly. That just makes it hard to interpret and explain your results. This page explains the tools and settings that eliminate that noise.

Why benchmarks are noisy

Modern CPUs and operating systems do several things that are excellent for everyday usability but terrible for micro-benchmarks:

• CPU frequency scaling — The kernel adjusts per-core clock speed dynamically (turbo boost, powersave governor). Your benchmark may run at 2.2 GHz on one iteration and 4.8 GHz on the next.
• Process migration — The Linux scheduler freely moves your benchmark process between cores. Different cores may have different cache states, different neighboring processes, and even different microarchitectures (e.g. Laptops with performance cores and power-efficient Atom cores).
• Background activity — Disk I/O, network interrupts, systemd timers, and other processes compete for CPU time, causing involuntary preemption.
• Real-time priority inversion — If any background thread has a higher nice value or scheduling priority than your benchmark, it can steal CPU time mid-measurement.

If you ignore these factors, you will measure the operating system’s scheduling behavior in addition to your algorithm’s performance.

taskset — pin to a specific CPU core

taskset binds a process to a given set of CPU cores, preventing the scheduler from migrating it.

# Pin to core 1 only
taskset -c 1 ./my_benchmark

# Pin to cores 2 and 3
taskset -c 2,3 ./my_benchmark

You can also chain it with e.g. perf with then calls your executable:

taskset -c 1 perf stat ./my_benchmark

Why this matters

Without taskset, your benchmark process may land on a core sharing a last-level cache with a memory-intensive background process, or may migrate mid-run, invalidating warm-cache assumptions. Worse, on a system with performance and power-efficient cores, it may migrate between them, making results completely unpredictable. Pinning ensures every iteration sees the same hardware topology.

chrt — change scheduling policy and priority

chrt lets you change the Linux scheduling policy of a process. The default policy is SCHED_OTHER (normal time-sharing). For benchmarks, you typically want SCHED_FIFO (first-in-first-out real-time):

# Run with SCHED_FIFO at priority 10 (low RT priority)
chrt -f 10 ./my_benchmark

# Check current policy
chrt -p $$

With SCHED_FIFO, once your process gets the CPU it keeps it until it voluntarily yields (or a higher-priority RT task preempts it). This eliminates involuntary preemption from normal-priority background tasks.

Priority value

Priorities 99–100 are reserved. Using very high RT priorities (e.g. 90+) for a long-running process can starve the entire system and make it unresponsive. For short benchmarks, low RT priorities (1–10) are sufficient and safe.

schedtool — combine both in one command

schedtool wraps both chrt and taskset into a single invocation. This is the tool I recommend to add in your CMakeLists.txt.

schedtool -F -p 10 -a 2 -e ./my_benchmark

Breaking down the flags:

This single line gives you RT scheduling, elevated priority, and CPU pinning simultaneously.

Adjust your CMakeLists.txt

Prepend schedtool -F -p 10 -a 2 -e to the invocation of the benchmark in add_benchmark.

Multithreaded benchmarks

Since the above schedtool call restricts the benchmark to a single core, multi-thread benchmarks will execute on a single core. Keep that in mind when you want to play with multi-threading.

Also, if you remove the core restriction, a run-away benchmark can hang your system, as it starves all user processes.

Disabling CPU frequency scaling — benchmarking.sh

Even with schedtool, if the CPU changes its clock frequency between iterations, your timing measurements are meaningless. You can download my benchmarking.sh script.

./benchmarking.sh on

This script performs three actions:

1. Sets the CPU governor to performance — Writes performance to every /sys/devices/system/cpu/cpufreq/policy*/scaling_governor. This locks each core at its maximum base frequency.

2. Disables Turbo Boost — Depending on the CPU, writes either:
   • echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo (Intel)
   • echo 0 > /sys/devices/system/cpu/cpufreq/boost (AMD)
3. Handles permissions — If needed, uses sudo to grant group write access to the sysfs files, so subsequent invocations do not require root. The provided laptops already have permissions set up as needed, so this is not needed there.

To restore normal operation:

/home/mkretz/bin/benchmarking.sh off

For convenience, there is an interactive mode:

/home/mkretz/bin/benchmarking.sh

Why disable turbo boost?

Turbo boost is triggered by temperature, power draw, and active core count. During a benchmark run, the kernel may enable or disable turbo at arbitrary moments depending on thermal headroom. This means identical code can execute at different frequencies across iterations, inflating variance. Disabling turbo ensures every measurement runs at a known, fixed clock rate.

Note: This means your benchmark measures performance at the base frequency, not the turbo frequency. For comparing algorithms this is exactly what you want — consistency beats absolute peak numbers.

When not to disable turbo boost

There can be good reasons to keep turbo boost on. E.g., if you want to measure best-case throughput for a certain operation / transformation. With lower core frequency, the ratio of memory latency to instruction throughput is skewed in favor of less cache-efficient codes.

On my laptop, the achievable peak FLOP differ by more than a factor 2!

Putting it all together

For the most reliable benchmark results, combine all layers:

In one terminal window keep interactive benchmarking.sh open. You want to turn the benchmark mode off for normal interactive usage of the system (faster compiles).

/home/mkretz/bin/benchmarking.sh

And after adding the schedtool call to CMakeLists.txt, all your benchmarks invoked via the make run_<name> target automatically use real-time priority without CPU migration.

However, remember to use schedtool when invoking manually, e.g. for running in perf.

schedtool -F -p 10 -a 2 -e perf stat ./my_benchmark

Or equivalently with separate tools:

chrt -f 10 taskset -c 2 perf ./my_benchmark

Quick checklist for interpretable benchmarks

Verify each of these before trusting your numbers:

• CPU governor is performance
• Turbo boost is disabled
• Process is pinned to a single performance core
• Process uses real-time scheduling (SCHED_FIFO)
• No heavy background workloads running (check top)
• Built in Release or RelWithDebInfo mode (benchmarking Debug builds seldom makes sense)

References

• Linux schedtool man page
• Linux chrt man page
• Linux taskset man page
• Kernel documentation — CPUFreq sysfs interface