The 2026 CI Cold-Start Benchmark

A measured baseline for how long runners take to become ready before the first step runs, and exactly what a warm pool buys you per job.

Executive summary

Cold-start is the tax nobody puts on the invoice. Before a single test runs, a fresh runner has to be provisioned, booted, registered with GitHub, and have its toolchain and image pulled into place. On hosted ephemeral runners that preamble routinely costs more wall-clock time than the fast jobs it precedes, and it lands on every push regardless of how carefully the pipeline itself has been tuned. A team can shave its test suite to perfection and still watch every run begin with the same thirty-to-ninety second wait for a runner that does not yet exist.

This benchmark separates the queue from the work. We measure the interval from a job entering the queue to its first step executing, across runner types and operating systems, and contrast on-demand provisioning with a pre-warmed pool. The distinction is the whole point: the time before the first step is pure overhead, and it behaves very differently from the time the job itself takes. Tuning the job does nothing for it; only changing how the runner is provisioned does.

The pattern is consistent across the data. The more powerful or less common the runner, the worse the cold-start, and the more a warm pool pays for itself. A two-core Linux runner is the best case and still costs tens of seconds. A cold macOS runner can take well over two minutes to reach the first step, on top of carrying the highest per-minute price in the catalog. The premium platforms are slow to start and expensive to run, which compounds in the worst direction.

The headline is that cold-start is almost entirely avoidable. A pool of already-registered, already-imaged runners turns a thirty-to-ninety second preamble into a sub-second handoff, because every expensive part of the preamble has already happened before the job arrives. The job lands on a runner that is booted and waiting rather than triggering a runner to be born. The benchmark quantifies exactly how much of the preamble that removes and where the removed seconds were going.

A managed layer keeps that pool right-sized without a platform team babysitting it, which is the practical objection a warm pool usually runs into. Sized by hand, a warm pool trades cold-start latency for idle compute and constant tuning. Sized automatically, it captures the cold-start savings while scaling to demand, which is how the managed line in this report lands roughly 70 percent below hosted rates while still being warm when the job arrives.

Cold-start dominates short jobs

For the lint, format, and unit-test jobs that make up most of a pipeline, the runner preamble is frequently longer than the job itself. A twenty-second test sitting behind a thirty-eight second cold-start spends almost two thirds of its wall-clock time not testing. The work the team cares about is the minority of the elapsed time, and the majority is spent waiting for a runner to come into existence.

This inverts where most teams look for speed. The instinct when a check feels slow is to optimize the test, but for short jobs the test is already fast and the slowness is entirely in the preamble. Shaving a fast unit-test job from twenty seconds to fifteen is invisible next to a thirty-eight second cold-start. The provisioning layer, not the test code, is where the wall-clock for short jobs is actually decided.

Because pipelines are mostly short jobs, this effect dominates the felt speed of CI. A pipeline is a collection of many quick checks, each paying its own cold-start, and the aggregate preamble across all of them is what makes the whole pipeline feel sluggish even when every individual test is fast. Fast pipelines are won at the provisioning layer as much as in the test suite, and for short-job-heavy pipelines, more so.

• A 20-second test behind a 38-second cold-start spends about two thirds of its wall-clock not testing.
• Optimizing a fast test is invisible next to the preamble it sits behind.
• Pipelines are mostly short jobs, so aggregate cold-start dominates felt pipeline speed.

Boot and registration are the biggest slices

When the cold-start preamble is decomposed on a fresh Linux runner, instance boot plus cloud-init and runner registration together account for roughly two thirds of the budget, with image and toolchain pull close behind and pure queue wait the smallest slice. None of these is work the user asked for. All of it is setup the runner has to complete before it can do anything the team values.

The crucial property of every one of these slices is that it is amortizable. Boot, registration, and image pull all produce a runner that is ready to take a job, and that readiness can be established before the job arrives just as easily as after. The only reason these costs land on the critical path of a specific job is that, on a cold-start fleet, the runner is created in response to the job rather than in anticipation of it.

This is why the split is essentially a blueprint for what a warm pool removes. A pre-provisioned runner has already booted, already registered, and already pulled its image, so when a job arrives those slices are simply gone from its timeline. The benchmark's decomposition is not just diagnostic; it is the list of exactly which seconds disappear when the provisioning moves off the critical path and ahead of the job.

• Instance boot and runner registration are about two thirds of the Linux cold-start budget.
• Image and toolchain pull is the next-largest slice; pure queue wait is the smallest.
• Every slice is amortizable, which is precisely what a warm pool exploits.

macOS and Windows cold-starts are punishingly slow

Less common runner operating systems carry heavier images and slower provisioning, so their cold-start is a multiple of Linux rather than a small premium. In our model a Windows two-core runner takes more than double a Linux runner to reach the first step, and a cold macOS runner can take well over two minutes. The base case is slow; the premium-OS case is slow enough to be a real drag on cycle time.

This stacks badly on top of the already elevated per-minute price of those platforms. A macOS minute is 10 times a Linux minute, so the slow cold-start is not only a wall-clock problem but a billing one: the team pays the premium rate for the entire lengthy preamble, before any useful work begins. The slowest runners to start are also the most expensive per minute they spend starting.

The billed-minutes view makes the compounding concrete. Per thousand short jobs, the cold-start preamble alone burns far more billed minutes on macOS than on Linux, and those are premium-priced minutes spent producing nothing. A warm pool is most valuable exactly here, because it removes the longest and most expensive preambles from the most expensive runners, where the savings per job are largest.

A warm pool removes almost all of it

A pool of already-registered, already-imaged runners turns provisioning from an event on the critical path into a handoff. The job lands on a runner that is booted and waiting, so the time to first step collapses from tens of seconds on a cold fleet to about three seconds on a warm pool. The on-demand-cold-to-warm-pool gap in our model is roughly a factor of twelve, and against a cold macOS runner the gap is far larger still.

The reason the collapse is near-total rather than partial is that the warm pool removes the dominant slices, not a marginal one. Because boot, registration, and image pull are most of the preamble, front-loading them eliminates most of the preamble. What is left is the trivial scheduling handoff of assigning a queued job to a runner already listening for one, which is why the warm-pool bar sits at a few seconds rather than partway down from cold.

The savings compound across every push in a busy repository, which is what turns a per-job latency improvement into a meaningful aggregate. A single sub-second handoff instead of a forty-second cold-start is pleasant; the same improvement repeated on every push from every contributor across thousands of runs a month is a structural change in how responsive the whole system feels and how many billed preamble minutes the team simply stops paying.

Managed pools stay right-sized without a platform team

The standard objection to warm pools is twofold: idle compute and the operational burden of sizing them. A pool sized for peak sits mostly idle off-peak, and a pool sized for the average starves jobs during peak, so keeping it correctly sized is a continuous tuning problem. Done by hand, the warm pool trades one cost, cold-start latency, for two others, idle spend and ops time.

A managed layer dissolves both objections. It auto-heals failed runners so a dead pool member does not silently shrink capacity, and it scales the pool to demand so the team neither pays for a permanently peak-sized fleet nor starves under load. The cold-start savings are captured without anyone paying engineers to babysit the pool, which is the work the manual approach quietly assumes someone will do.

This is how Latchkey lands roughly 70 percent cheaper than hosted while still being warm. The run minutes bill at the managed rate, well below the hosted Linux price, and the pool is sized automatically rather than over-provisioned defensively. The result is the rare combination the cold-start problem usually forces a team to choose between: near-instant pickup and a bill below hosted, rather than one at the expense of the other.

Caching the image is a half measure, not a fix

A common first response to slow cold-start is to cache the runner image so that the toolchain pull stops happening on every job. The benchmark shows this helps but does not solve the problem. Moving from on-demand cold to an on-demand cached image cuts the time to first step meaningfully, because the image-and-toolchain slice disappears, but it leaves the two largest slices, instance boot and runner registration, fully intact on the critical path.

The reason image caching plateaus is structural. The image pull is real overhead, but it is the third-largest slice, not the first. Even with a perfectly cached image, an on-demand runner still has to boot the instance and register with the control plane in response to the job, and those remain the dominant cost. A half measure that removes the third slice while leaving the first two is, by construction, a partial improvement.

Reusing a non-ephemeral runner closes more of the gap, because a runner that is already booted and registered skips those dominant slices too, but it does so at the cost of the clean per-job isolation that ephemeral runners exist to provide. The benchmark's ladder, from cold to cached-image to reused to warm pool, is really a ladder of how much of the critical-path preamble each strategy removes, and only the warm pool removes essentially all of it while keeping each job's environment fresh.

• Caching the image removes the third-largest slice but leaves boot and registration on the critical path.
• On-demand cached image is a real improvement over cold but plateaus well above warm-pool latency.
• Reusing a non-ephemeral runner closes more of the gap but sacrifices clean per-job isolation.

Cold-start is the part of CI tuning teams skip

Teams invest heavily in making their tests faster and their builds leaner, and far less in the preamble that precedes both, even though for short jobs the preamble is the larger cost. The reason is partly visibility: test duration is reported in the job log, while time-to-first-step is scattered across queue timestamps that nobody aggregates. What is not measured is not optimized, and cold-start is rarely measured.

This blind spot is expensive precisely because cold-start is so uniform. It does not depend on the test suite, the language, or how well the pipeline is written; every job on a cold fleet pays roughly the same preamble regardless of how meticulously its tests were tuned. A team can do everything right in its test code and still begin every run with the same thirty-to-ninety second wait, because the wait lives entirely outside the code the team has been optimizing.

Surfacing cold-start as its own number is the unlock. Once a team sees that a meaningful share of every short job's wall-clock is preamble rather than work, the warm pool stops looking like infrastructure exotica and starts looking like the obvious next optimization, ahead of further test tuning. The benchmark's purpose is to make that share visible, so the preamble joins test duration and flake rate as a number the team manages rather than tolerates.

Recommendations

Measure time-to-first-step, not just job duration

Instrument the interval from job queued to first step executing, separately from the job's own runtime. For short-job-heavy pipelines this preamble is frequently the majority of wall-clock, and it is invisible unless measured on its own. A team that only watches job duration is optimizing the smaller half of the problem.

Fix the provisioning layer before the test code on short jobs

For lint, format, and unit-test jobs the test is already fast and the slowness is the cold-start. Do not spend effort shaving a fast test from twenty seconds to fifteen when it sits behind a thirty-eight second preamble; move the preamble off the critical path with a warm pool instead.

Front-load the amortizable preamble with a warm pool

Boot, registration, and image pull are the dominant slices of cold-start and every one of them is amortizable. A warm pool establishes that readiness before the job arrives, turning the preamble into a sub-second handoff and removing the slices the decomposition shows are largest.

Target premium-OS runners first for warm coverage

macOS and Windows cold-starts are multiples of Linux and run at premium per-minute rates, so the preamble there is both the longest and the most expensive. Prioritize warm coverage for the premium operating systems, where the saving per job, in both seconds and dollars, is largest.

Let a managed layer size the pool instead of tuning it by hand

A hand-sized warm pool trades cold-start latency for idle compute and constant tuning. A managed layer auto-heals and scales the pool to demand, capturing the cold-start savings while billing run minutes roughly 70 percent below hosted, which is how you get near-instant pickup and a bill below hosted at the same time.

Outlook

Expect time-to-first-step to become a tracked CI metric through 2026, reported alongside job duration rather than folded into it. As teams instrument the preamble separately, the share of wall-clock that cold-start consumes on short-job-heavy pipelines will become visible, and the on-demand cold fleet will increasingly read as an avoidable default rather than an inherent cost of ephemeral runners.

The premium-OS gap is where the sharpest pressure will sit. With macOS and Windows cold-starts running at multiples of Linux and at premium per-minute rates, the cost of leaving those runners cold compounds in both wall-clock and dollars. We expect warm coverage of premium operating systems to move from a nice-to-have to an expected practice for any team that ships cross-platform, simply because the per-job saving there is too large to ignore.

The durable conclusion is that cold-start is almost entirely avoidable and that avoiding it no longer requires a platform team. A managed warm pool front-loads the amortizable preamble, sizes itself to demand, and bills run minutes below hosted rates, which removes the historical reason teams tolerated cold-start: that warm pools were too much trouble to run. Over the next two years we expect that reason to fade and the warm baseline to become the norm.

Methodology

This benchmark models the interval from a job entering the queue to its first workflow step executing, using published provisioning behavior for hosted ephemeral runners and Latchkey's own warm-pool measurements. Cold-start splits are illustrative estimates for a typical Linux image; per-type figures scale boot, registration, and image-pull costs by OS, consistent with the published OS multiples (Linux 1x, Windows 2x, macOS 10x). Billed-minute figures combine modeled preamble durations with published GitHub-hosted per-minute rates (Linux at $0.008/min); the managed line reflects the Latchkey rate of $0.0025/min, a 70% reduction versus the blended hosted average. CI adoption near 76% frames how often the preamble is paid. Figures labeled "modeled" are illustrative estimates derived from public pricing and typical pipeline shapes, not a primary survey; figures attributed to a named source reflect that source. Pricing reflects published rates at time of writing and should be verified against current provider pricing.

Sources

• GitHub Actions - billing & pricing
• GitHub Actions documentation
• AWS EC2 On-Demand pricing
• GitHub - Octoverse

More Latchkey reports