Skip to content

Question about benchmarking #1

@Epixu

Description

@Epixu

Didn't see a discussion tab, so I'm asking here:
Catch2 included some rudimentary benchmarking, but didn't include patterns for checking benchmarking results in order to detect performance regresses - have you thought about something along these lines, or do you think this is out of your scope here?
It will probably need benchmarking data to be recorded, merged, and carried over between test runs.

Here's a cobbled up example from the top of my head to give more context:

auto result = BENCHMARK_ADVANCED("default construction") (timer meter) {
   some<uninitialized<T>> storage(meter.runs());
   meter.measure([&](int i) {
      return storage[i].construct();
   });
};

REQUIRE(result.within(0.1f));
REQUIRE(result.under(5s));
REQUIRE(result.faster_than(result2));
//etc...

I'm not familiar how other projects do their benchmarking and performance regress tests, but I suppose this is one way to do it?
I might be able to work on it, if you decide this is a feature you want. With C++23's chrono improvements, I believe it can be made quite painless.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions