How to Benchmark GitHub Actions Runners (Speed & Cost)

By Kaveh Alemi·Latchkey

Vendor benchmarks are easy to game. Here is how to run your own apples-to-apples comparison on your real builds.

The only benchmark that matters is your pipeline. Synthetic numbers rarely match your workload. Use this methodology to measure runners fairly on the metrics that affect your bill and your developers.

Metric	What to measure	Why it matters
Wall-clock job time	Median over 20+ runs of your real job	Developer wait time
Queue/cold-start time	Time from trigger to first step	Hidden latency
Cost per run	Minutes x per-minute rate	The actual bill
Cache hit rate	% of runs restoring cache	Biggest speed lever
Flaky-failure rate	% of runs failing transiently	Wasted re-run minutes

Control the variables

Same commit, same cache state, same time of day.
Run 20+ iterations and take the median, not the best.
Measure cost per run, not just speed - a fast premium runner can be the expensive choice.

Do not forget reliability

A runner that is 10% faster but fails transiently 15% of the time is slower in practice once you count re-runs. Factor flaky-failure rate into the comparison - self-healing runners score well here because they recover automatically.

The verdict

Benchmark on your own builds, over many runs, on speed AND cost AND reliability. Latchkey is built to win the reliability and cost axes - measure it against your pipeline.

Switching costs you one line

Latchkey runners are a drop-in replacement: change runs-on: and the rest of the workflow is untouched. Self-healing is on by default, so transient failures are fixed and retried instead of failing the build.

.github/workflows/ci.yml

# before
    runs-on: ubuntu-latest

# after
    runs-on: latchkey-medium

Start free - 30-day trial, no credit card required, or estimate your savings first.

References

GitHub Actions documentation