The 2026 Test Suite Benchmark

A quantitative benchmark of how test count drives wall-clock runtime, the sub-linear payoff of sharding, and the exact point where adding runners stops paying for itself.

Executive summary

Test suites only ever grow. Every feature adds cases, every bug adds a regression test, and the suite that ran in two minutes at launch is a fifteen-minute gate two years later. Because that gate sits on every push, suite runtime is one of the highest-leverage numbers in a pipeline, and one of the least deliberately managed. Most teams watch it creep upward without ever putting a number on the relationship between how many tests they have and how long the suite takes.

This benchmark exists to put that number down. It maps suite size to wall-clock runtime across four representative test counts, decomposes where the runtime actually goes, and then measures the practical question every team eventually asks: when does sharding pay off, and how far does it scale before the overhead of splitting and merging eats the gains? The four bar charts below are the product; the prose explains how to read them and what to do with the numbers.

The headline result is that unsharded runtime grows roughly with the square root of test count rather than linearly, which sounds forgiving until a suite quadruples and the gate still doubles. The companion result is that parallelism is enormously effective up to a point and then flattens, because fixed per-shard costs and an uneven split leave a longest-pole shard that no amount of extra runners can shorten. A large suite realizes about 4.1 times speedup at eight shards rather than the eightfold the matrix implies.

Underneath both results is a third number that surprises teams: about a third of suite runtime is setup, fixtures, and teardown rather than assertions. That overhead is paid before a single thing is verified, it is identical on every shard, and it is one of the reasons sharding flattens, because splitting tests across runners does nothing to shrink the setup each shard repeats. Attacking the setup helps every shard at once, which is why it is often a better first move than adding another runner.

The teams that keep suites fast combine sharding with the boring fundamentals: balanced shards, cached dependencies, fast runner startup, and automatic recovery for the flaky cases that would otherwise force a full rerun. The runner layer matters here as much as the test code, because a shard that waits forty seconds to start has already lost most of what sharding bought it. This benchmark quantifies each of those levers so a team can see, in numbers, where its own suite sits and which move returns the most.

Runtime grows sub-linearly but relentlessly

Unsharded suite runtime rises roughly with the square root of test count rather than linearly, because faster machines, warm caches, and amortized setup absorb some of the growth. The benchmark numbers make the curve concrete: 500 tests run in about three minutes, 2,000 in about eight, 8,000 in about nineteen, and 20,000 in about thirty-four. Each fourfold jump in test count produces roughly a doubling in runtime, which is the square-root relationship in action.

Sub-linear sounds forgiving, and in the small it is. A team going from 500 to 2,000 tests sees the suite go from three minutes to eight, an annoyance rather than a crisis. But the relationship is relentless: it never stops climbing, and the absolute minutes pile up exactly as the suite becomes most valuable. By the time a suite reaches 20,000 tests it is a thirty-four-minute gate on every push, which is a serious tax on how often developers are willing to merge.

The benchmark's purpose is to let a team locate itself on this curve and project forward. A suite at 8,000 tests today growing at the typical rate is heading for the 20,000-test tier, and the runtime that comes with it, within a predictable horizon. Knowing the curve turns suite runtime from something that surprises a team into something it can plan around, which is the first step toward managing it deliberately rather than reacting once it hurts.

A third of runtime is setup, not assertions

Fixtures, database seeding, and teardown routinely consume about a third of total suite runtime before a single assertion is checked. In the modeled split, assertions and test bodies are about half the runtime, setup and fixtures and teardown are another thirty-four percent, runner startup and dependencies are around an eighth, and flaky reruns take the small remainder. The verification the suite exists to do is barely half of what the clock measures.

This setup overhead has two properties that make it important. It is paid before any verification happens, so it is pure latency on the path to a result, and it is repeated identically on every shard, so it is one of the structural reasons sharding flattens. Splitting tests across eight runners splits the assertions eight ways but leaves each shard repeating the full setup, which caps how much parallelism can buy.

The good news is that setup overhead responds to direct attack, and the fix helps every shard equally. Shared fixtures that are built once rather than per-test, transactional rollbacks instead of full database reseeds between tests, and warm services that are already running when the suite starts all cut the setup slice directly. Because that slice is repeated on every shard, shrinking it improves both the unsharded runtime and the realized sharding speedup at the same time, which often makes it a higher-return move than adding another runner.

• Assertions are only about half the runtime; setup, fixtures, and teardown are roughly a third.
• Setup is repeated identically on every shard, which is a structural reason sharding flattens.
• Shared fixtures, transactional rollbacks, and warm services cut it and help every shard at once.

Sharding pays off fast, then flattens

Splitting a large suite across runners delivers near-linear speedup at first and then bends. The benchmark traces it for a 20,000-test suite: two shards return about 1.9 times, four shards about 3.2 times, and eight shards about 4.1 times. The first split nearly doubles throughput, but by eight runners the suite is realizing roughly half of the eightfold the matrix implies, and the curve is visibly flattening.

Two forces cause the bend. Fixed per-shard startup is paid in full on every runner, so as the per-shard test execution shrinks, that fixed cost becomes a larger fraction of each shard's wall-clock and a harder floor under the total. And an uneven split leaves a longest-pole shard, because tests are not uniform and an unbalanced assignment lands the slow tests together, so the run waits on whichever shard drew the heavy work regardless of how fast the others finished.

The practical reading is that past roughly eight shards the marginal runner buys very little for this suite unless the split is rebalanced first. The benchmark's value is showing where that point sits rather than leaving it to trial and error: a team can read the speedup curve, see that it has flattened, and know that the next lever is balancing the shards or shrinking the per-shard overhead, not adding a ninth runner that will mostly idle behind the longest pole.

• Two shards return ~1.9x, four ~3.2x, eight ~4.1x: near-linear early, flattening by eight.
• Fixed per-shard startup and an uneven longest-pole split are the two forces that cause the bend.
• Past ~8 shards the next lever is rebalancing or cutting per-shard overhead, not adding another runner.

Slow runner startup eats the sharding gain

Sharding multiplies the number of runners that must start, and on a slow-to-provision runner that startup is a fixed tax paid on every shard. A forty-second cold start per shard does not parallelize away: it is a floor under each shard's wall-clock time, and as the actual test execution per shard shrinks with more shards, that forty seconds becomes a larger and larger share of what each runner spends, which is a direct cause of the flattening in the previous finding.

The effect is worst exactly where sharding is most attractive. A team splitting a suite into eight shards to make each one short has made each shard's execution brief enough that a forty-second startup is now a meaningful fraction of it. The team bought eight runners to go eight times faster and is losing a chunk of that to startup it pays eight separate times, even though the startups overlap.

A warm pool of pre-provisioned runners is what removes this tax. When each shard starts in seconds rather than tens of seconds, the fixed per-shard floor shrinks and the realized speedup moves up toward the matrix ideal. This is why the eight-shard figure in the benchmark is labeled as a warm-pool number: the 4.1 times assumes shards start fast, and on cold-starting runners the realized speedup would be visibly worse, which the cost chart then prices out.

Fast and cheap sharding need a managed layer

The cost chart is where the benchmark's strategies separate, and it is the most decision-relevant of the four. One slow runner costs about twenty-seven cents for a run. Eight hosted shards cost about thirty-one cents, more than the slow single runner, because each shard re-pays startup and dependency install while delivering only the flattened 4.1 times speedup. Paying more to go four times faster can be worth it, but it is a real premium, not a free lunch.

Caching changes the arithmetic. Eight shards with a warm dependency cache come in around twenty-four cents, below the slow single runner, because the cache removes the duplicated install that each shard would otherwise re-pay. That is the first strategy that is both faster and cheaper than the slow baseline, and it shows that the fundamentals, here caching, are what make sharding economical rather than merely fast.

The managed row is the fast-and-cheap floor: eight shards on a managed warm pool with caching land around ten cents, roughly 70 percent below the hosted-shard equivalent. The warm pool removes the cold-start tax that was capping the speedup, the cache removes the duplicated install, and the lower per-minute rate prices the remaining minutes cheaply, so the run that is fast is also the cheapest of the four. That is the benchmark's bottom line: teams should not have to choose between a fast suite and an affordable one, and with a managed layer they do not.

• Eight hosted shards (~$0.31) cost more than one slow runner (~$0.27) because each shard re-pays startup and install.
• Caching drops eight shards to ~$0.24, the first strategy both faster and cheaper than the slow baseline.
• A managed warm pool with caching lands ~$0.10, about 70% below the hosted-shard cost, fast and cheap together.

Balanced shards are the difference between 4x and 8x

The benchmark's eight-shard figure of 4.1 times assumes a reasonable but imperfect split, and a large part of the gap between that number and the eightfold ideal is shard balance. Tests are not uniform in duration: a handful of integration tests can each take longer than thousands of unit tests, so an even-by-count or alphabetical split routinely lands the slow tests together and creates a shard that runs far longer than its peers.

Because a sharded run is only finished when its slowest shard finishes, that longest pole sets the wall-clock time while the team pays for all eight runners. A suite split so that one shard runs twice as long as the others realizes roughly half the parallelism it bought, which is exactly the kind of loss that turns a potential eightfold into the observed fourfold.

Balancing shards by recorded per-test duration is the fix, and it costs nothing but a history of how long each test took. When the splitter assigns tests so every shard is predicted to finish at the same moment, the shards complete together and the realized speedup climbs back toward the ideal. This is the cheapest lever in the entire benchmark: no new runners, no infrastructure, just splitting by time instead of by count, and it is frequently the single change that recovers the most lost speedup.

Flaky reruns are a small slice that triggers a large one

Flaky reruns are only about five percent of the modeled runtime split, which makes them look minor next to setup and assertions. But that small slice is deceptive, because in a sharded run a single flaky test in one shard can force a re-run of the entire matrix, turning a five-percent slice into a full duplicate run of all eight shards and the cost that comes with it.

The mechanism is the asymmetry between where a flake occurs and what it costs. The flake happens in one shard, but the conventional response, re-run the job, re-runs every shard, because the CI system retries at the job level rather than the failed-shard level. So the realized cost of a mechanical flake is multiplied by the shard count, which is exactly the situation a team creates by sharding aggressively without a smarter recovery story.

Automatic recovery that retries only the transient failure on a fresh environment breaks that multiplier. When the runner re-runs the one flaky shard, or the one flaky test, rather than the whole matrix, a mechanical failure costs a fraction of a shard instead of a full re-run of all of them. Combined with the warm-pool and caching levers, this keeps the realized cost of a sharded run close to its intended cost rather than spiking on every flake, which is what makes aggressive sharding safe rather than expensive.

Recommendations

Locate your suite on the runtime curve and project forward

Unsharded runtime grows roughly with the square root of test count, so each fourfold growth in tests roughly doubles the gate. Use the benchmark numbers to place your suite, project where it is heading at its current growth rate, and start managing runtime deliberately before it becomes a multi-minute tax on every merge rather than after.

Attack setup overhead before adding runners

About a third of suite runtime is setup, fixtures, and teardown, paid before any assertion and repeated identically on every shard. Shared fixtures, transactional rollbacks, and warm services cut that slice directly and improve both the unsharded runtime and the realized sharding speedup at once, which often makes it a higher-return move than buying another runner.

Balance shards by recorded timing, not by count

A large part of the gap between the realized 4.1x and the eightfold ideal is an uneven split that lands the slow tests on one longest-pole shard. Balancing by recorded per-test duration costs nothing but a timing history and is frequently the single change that recovers the most lost speedup, turning runners you already pay for into speed you actually get.

Put a warm pool under the shards to remove cold-start tax

Sharding multiplies cold starts, and tens of seconds of startup per shard is a fixed floor that caps the realized speedup, worst exactly where shards are shortest. A warm pool that starts each shard in seconds shrinks that floor, which is why the benchmark labels its eight-shard speedup a warm-pool number and prices the cold-start penalty into the cost chart.

Recover flaky shards individually instead of re-running the matrix

A single flaky test can force a re-run of all eight shards when CI retries at the job level, multiplying a five-percent slice into a full duplicate run. Automatic recovery that retries only the transient failure on a fresh environment breaks that multiplier and keeps the realized cost of a sharded run close to its intended cost, which is what makes aggressive sharding safe.

Outlook

Expect suite runtimes to keep climbing along the square-root curve this benchmark traces, because the forces that grow suites, more features, more regressions, more coverage, do not relent. The teams that manage runtime deliberately will be the ones that treated the curve as a plannable trajectory rather than a surprise, attacking setup overhead and balancing shards before the gate became painful rather than after it started discouraging frequent merges.

The sharding story will stay shaped by the same two ceilings: fixed per-shard overhead and an uneven longest-pole split. As more teams shard, the differentiator will be whether they shard well, balancing by timing, caching dependencies, starting shards from a warm pool, and recovering flaky shards individually, or whether they simply add runners and watch the speedup flatten while the bill climbs. The benchmark's cost chart is the clearest argument that the fundamentals, not the shard count, decide whether a fast suite is also an affordable one.

For most teams the practical takeaway is that suite runtime is a measurable, manageable number rather than a fact of life. The relationship between test count and runtime is predictable, the overhead inside it is addressable, and the point where sharding flattens is locatable in advance. A team that knows where it sits on these curves can pick the next lever, setup, balance, warm starts, or recovery, with confidence that it is buying real speed for sensible money instead of adding runners and hoping.

Methodology

This benchmark models suite wall-clock runtime as a sub-linear function of test count, with sharding speedup capped by fixed per-shard overhead and an uneven longest-pole split. Runtime splits and shard speedups are illustrative Latchkey estimates derived from typical suite shapes rather than a primary survey, so they describe direction and magnitude rather than a precise per-suite value. Per-run cost figures combine modeled runner-minutes with published GitHub-hosted Linux rates ($0.008/min). Figures labeled "modeled" are illustrative estimates derived from public pricing and typical pipeline shapes, not a primary survey; figures attributed to a named source reflect that source. Pricing reflects published rates at time of writing and should be verified against current provider pricing.

Sources

• GitHub Actions documentation
• GitHub Actions - billing & pricing
• JetBrains Developer Ecosystem Survey
• Stack Overflow Developer Survey

More Latchkey reports