GitLab "Job failed (system failure): lost connection to runner"

By Kaveh Alemi·Latchkey

The job was running, then the runner stopped reporting. GitLab marks it a system failure because the runner crashed, was scaled down, or lost network mid-job - not because your script failed.

What this error means

A previously-progressing job ends with a system failure noting the runner connection was lost. The log cuts off abruptly; re-running on a healthy runner usually succeeds.

ERROR: Job failed (system failure): aborted: terminated
# or
WARNING: Job failed: failed to update job: couldn't execute PUT ... connection reset by peer

Common causes

Runner crashed or was scaled down

An autoscaled or spot runner removed while the job ran, or a crashed runner process, severs the connection and aborts the job.

Network drop between runner and GitLab

A transient network failure stops the runner from updating the job, so the coordinator declares the connection lost.

How to fix it

Retry on runner system failure

Let GitLab automatically re-run jobs aborted by a lost runner.

default:
  retry:
    max: 2
    when:
      - runner_system_failure

Stabilize the runner fleet

1. For autoscaled/spot runners, use graceful draining so jobs are not killed mid-run.
2. Run runners under systemd so a crashed process restarts.
3. Investigate network stability between the runner host and GitLab.

How to prevent it

• Add retry: when: runner_system_failure for autoscaled fleets.
• Drain spot/autoscaled runners gracefully before termination.
• Monitor runner host and network stability.

Frequently asked questions

Related guides

References

• GitHub Actions documentation