Self-healing: what it does

How Latchkey runners detect and fix transient CI failures during the run, what they will never touch, and how heal proposals become reviewable pull requests.

Every Latchkey runner ships with self-healing built in, enabled by default. When a workflow step fails, the runner diagnoses the failure locally and, when it is a known transient class, fixes it and retries the step in place. Your build goes green without a human re-run, and every intervention is recorded for you to inspect.

The three-stage diagnosis cascade#

Diagnosis runs fastest-first, so the common cases cost almost nothing:

What it catches, by category#

npm ERR! code ETIMEDOUT
npm ERR! network request to https://registry.npmjs.org/react failed
npm ERR! network This is a problem related to network connectivity.

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Error: ENOSPC: no space left on device, write

/usr/bin/bash: line 1: ffmpeg: command not found

npm ERR! `npm ci` can only install packages when your package.json and
package-lock.json are in sync.

The fix toolbox#

Every fix is applied inside the ephemeral runner only, and the runner is destroyed after the job:

A heal, end to end#

Here is one real heal, an npm registry timeout, from failing step to green job. Your step output streams unchanged, in real time; during a heal the runner adds two diagnostic lines of its own while the on-runner heal service is consulted:

$ npm ci
npm ERR! code ETIMEDOUT
npm ERR! syscall connect
npm ERR! network request to https://registry.npmjs.org/lodash failed, reason: connect ETIMEDOUT 104.16.92.83:443
npm ERR! network This is a problem related to network connectivity.
[latchkey-bash-wrapper] BEGIN sidecar POST (boot_wait=30s max_time=260s url=http://localhost/diagnose socket=/run/latchkey-self-heal/sock)
[latchkey-bash-wrapper] END sidecar POST ok (attempts=1 http=200)
$ npm ci
added 1291 packages, and audited 1292 packages in 42s
found 0 vulnerabilities

1. The step fails. npm ci exits non-zero after npm exhausts its own internal retries. The runner captured the step output as it streamed, so the diagnosis starts with the full picture already in hand.
2. Stage 1, exit code. Exit 1 is not decisive on its own, so the cascade moves on.
3. Stage 2, pattern match. The ETIMEDOUT lines match a known npm network signature in the pattern library: verdict heal, category network, no AI involved. The two [latchkey-bash-wrapper] lines are the only trace of this consultation in your logs.
4. Retry with backoff. The plan allows up to 3 attempts with an escalating delay starting at 2 seconds. The runner waits, then re-runs the step on the same machine, in the same workspace: the second npm ci in the log is that retry.
5. Green. The retry passes and the job continues as if the timeout never happened. On success nothing more is added to your logs; heals that change the machine announce it (for example [latchkey-bash-wrapper] installed package: ffmpeg), so no intervention is invisible.

The narrative you did not see in the logs is on the dashboard: the Recent Heals feed on the Runners page shows a Healed row in the network category with a Pattern match pill and the action taken ("Retried the npm command with exponential backoff after a network timeout"), the Heal Details drawer carries the stage-by-stage story, and the run keeps a green Healed badge in Pipeline Performance.

Heal pull requests#

Some root causes deserve a durable fix in your repository, not just an in-run rescue: a setup step missing from the workflow, a missing engines pin in package.json, a too-low job timeout. When a successful heal traces back to a structural cause like these, self-healing produces a structured proposal and opens a heal PR with a typed, deterministic edit.

• Each PR is titled "Latchkey heal: <error summary>"; the body explains the Error, the Root cause, and the Fix, and links back to the workflow run, so review takes minutes.
• Proposal PRs carry a verification note: Verified by run when the PR's own workflow run passed, or Proposed fix, not verified by a passing run when it did not.
• When retries rescued the failing runs, the proposal quantifies it ("Latchkey auto-retried this workflow N times; M passed on the retry"); auto-retries pause when a workflow's retry pass rate is poor.
• Heal PRs are never auto-merged, and pull requests from forks are never touched.
• PR creation uses the GitHub App's permissions; if you never merge a heal PR, nothing in your repository changes.

Declined fixes: stop proposing#

Not every proposal class is welcome on every repository. Self-heal proposal findings on the AI Insight page carry a Stop proposing action (owners and admins): Latchkey then permanently stops opening PRs for that failure class on that repository until you undo it under Settings, Self-Healing, Declined fixes. Self-healing still runs on those failures; it just stops proposing workflow changes.

What it will and will never do#

Hand off to your coding agent#

Self-healing fixes the environment, never your source. When the real problem is a bug in your own code, the build fails truthfully, and the failure arrives as a complete, structured bundle served over the Latchkey MCP server, ready for your own coding agent to fix from. The bundle hands your agent what it would otherwise reconstruct by hand:

• The root cause, in plain language.
• The failing step's exit code and the exact source file where the error surfaced.
• The full, untruncated logs of the failing step, including output GitHub hides in its log viewer, with secrets stripped before they leave Latchkey.
• What self-healing already investigated and why it stood down, plus the workflow definition.

In Claude Code, the built-in /mcp__latchkey__fix command does the whole round trip in one step: it grabs the most recent unfixed failure and gets to work. Any MCP-capable agent (Cursor, Codex, and others) can read the same bundles through the server's tools. Setup, API keys, and the ready-made connect command are in Connect your AI agent.

Observability: every heal is on the record#

• Recent Heals (on the Runners page) lists every intervention with its category, verdict, and a plain-language description of the action taken; AI-diagnosed heals include the agent's reasoning iterations.
• Healed runs are flagged in the Pipeline Performance runs table and deep-link to their heal report.
• Heal KPIs and trends on the Runners page show how often healing saves a run and which categories dominate, which is itself a signal about your infrastructure.

Controls#

• Workspace-level switch in Settings, Self-Healing (owners and admins), on by default for new workspaces; changes take effect within about a minute.
• There is no per-repository toggle; the workspace switch applies to all monitored repositories.
• Declined fixes, also under Settings, Self-Healing, lists the proposal classes you have stopped, with the option to undo each one.

Common questions#

To see which real-world failures healing covers, browse the self-healing pattern library on Learn - each entry shows the manual fix next to what the runner does automatically. For anything currently failing your builds, start from the full CI/CD error library.