Write a benchmark (part 1)

In the src/exercises directory:

CMake

• Edit CMakeLists.txt and add a function add_benchmark, similar to add_runnable.
• For benchmarks, however, we can automatically link against the Google Benchmark library, via PRIVATE benchmark::benchmark.
• The run_${name} target can then add --benchmark_counters_tabular=true --benchmark_perf_counters=CYCLES,INSTRUCTIONS to the benchmark invocation. These arguments are parsed by Google Benchmark. (Execute a benchmark without these arguments to see the difference.)
• Finally, add add_benchmark(peakflop peakflop.cpp) at the bottom.

If you don’t want to figure it out yourself 😉

function(add_benchmark name)
  add_executable(${name} ${ARGN})
  target_link_libraries(${name} PRIVATE benchmark::benchmark)
  add_custom_target(run_${name}
    COMMAND $<TARGET_FILE:${name}> --benchmark_counters_tabular=true --benchmark_perf_counters=CYCLES,INSTRUCTIONS
    COMMENT "benchmarking ${name}")
endfunction()

Code

Create a new file peakflop.cpp and add the following boilerplate:

#include <benchmark/benchmark.h>

void peak(benchmark::State &state)
{
  float x = 1;
  for (auto _ : state) {
    x = x * 3 + 1;
  }

  // compute FLOP/s and FLOP/cycle
  constexpr double flop_per_iteration = 2;
  state.counters["FLOP"] = {flop_per_iteration,
                            benchmark::Counter::kIsIterationInvariantRate};
  if (state.counters.contains("CYCLES")) {
    state.counters["FLOP/cycle"] = {flop_per_iteration / state.counters["CYCLES"],
                                    benchmark::Counter::kIsIterationInvariant};
  }
}

// Register the function as a benchmark
BENCHMARK(peak);

// Run the benchmark
BENCHMARK_MAIN();

Run the benchmark

Call ninja run_peakflop to compile and execute the benchmark.

Are the numbers correct?

Anything wrong with the benchmark?

Inspect the binary

Inspect the binary with

vir_inspect.sh peakflop peak

Inspect with Compiler Explorer

Inspect the example we benchmarked using Compiler Explorer. (Remove the FLOP/s computation.) CE link

TIP

Use, e.g., std::vector<int> in place of benchmark::State to simplify the asm on CE.

Fix the benchmark

Modify the benchmark to produce believable results.

Documentation

benchmark::DoNotOptimize(x)

Inline assembly barrier

Or invoke inline assembly yourself:

asm volatile("" : "+x"(x));

It is different from benchmark::DoNotOptimize. Is it better? More correct? Discuss.

Documentation

GCC’s documentation for asm

Local Compiler Explorer

Of course you can achieve a very similar result locally, using e.g. the following command. Compiler Explorer has the added feature of better annotation of the assembler output and easy testing of different compilers and compiler flags.

CXXFLAGS=-std=c++26 -O2 -DNDEBUG -g0
watch "ccache g++ $CXXFLAGS -c -S -o - -masm=intel myfile.cpp|grep -vE '^\s+\.'|c++filt"

Drop ccache if you don’t have it available. But since watch recompiles every 2s, caching recompiles of unchanged code is not such a bad idea. 😉

We will continue with this benchmark until we reach Peak-FLOP — but not today.

References

• vir_inspect.sh
• preventing optimization (Google Benchmark)
• asm