ARC "EphemeralRunner failed too many times" in CI
ARC retries a failing EphemeralRunner a fixed number of times, then stops. "failed more than 5 times" means the same startup error recurs on every attempt, so a fix is needed rather than another retry.
What this error means
The controller logs that an EphemeralRunner failed more than the retry limit and marks it Failed. The job stays queued because ARC will not keep recreating a runner that cannot start.
ERROR EphemeralRunner has failed more than 5 times, marking as Failed
{"ephemeralrunner": "arc-runner-set-xxxx"}Common causes
A persistent startup failure
The same error (bad image, failed registration, missing sidecar) recurs each attempt, so every retry fails identically.
A resource or scheduling problem that never clears
If pods cannot schedule or are OOMKilled every time, ARC exhausts its retries without a single healthy runner.
How to fix it
Find the recurring failure in the pod logs
- Describe the failed EphemeralRunner for the recorded reason.
- Read the runner and any init/dind container logs for the repeated error.
- Fix the root cause, then delete the failed EphemeralRunner so ARC recreates it.
kubectl get ephemeralrunner -n arc-runners
kubectl logs -n arc-runners <runner-pod> --all-containers --previousAddress the underlying startup error
Resolve the image pull, registration, DinD readiness, or resource issue the logs point to; the retry limit will not clear until the real cause is fixed.
How to prevent it
- Validate runner images and credentials before rolling out.
- Right-size resources so pods do not OOM or stay Pending.
- Watch for repeated EphemeralRunner failures as an early signal.