Kubernetes "Liveness probe failed" restarting the pod in CI - Fix it
A liveness probe answers "is this container still healthy?" When it fails, the kubelet kills and restarts the container. A common deploy-time false positive is a liveness probe that runs before a slow-booting app is ready, restarting a process that would have come up fine.
What this error means
kubectl describe pod Events show Liveness probe failed: HTTP probe failed with statuscode: 503 (or connection refused) and Killing container ... failed liveness probe. The pod restarts repeatedly during the rollout.
Warning Unhealthy kubelet Liveness probe failed: Get "http://10.1.2.3:8080/healthz":
dial tcp 10.1.2.3:8080: connect: connection refused
Normal Killing kubelet Container api failed liveness probe, will be restartedCommon causes
Probe fires before the app is up
initialDelaySeconds/timeoutSeconds are too tight for the app boot time, so the probe fails during normal slow startup and the kubelet restarts it before it can serve.
The health endpoint is genuinely failing
The /healthz path is wrong, the port is misconfigured, or the app is actually unhealthy (deadlock, dependency down).
How to fix it
Separate startup from liveness
Use a startupProbe to cover slow boot, and keep the liveness probe lenient for steady state.
startupProbe:
httpGet: { path: /healthz, port: 8080 }
failureThreshold: 30
periodSeconds: 5
livenessProbe:
httpGet: { path: /healthz, port: 8080 }
initialDelaySeconds: 0
periodSeconds: 10Verify the endpoint and port
- Confirm the probe path/port match what the app actually serves.
- Curl the health endpoint from inside the pod (
kubectl exec) to confirm it responds. - Loosen timeoutSeconds/failureThreshold if the check is slow but valid.
How to prevent it
- Always pair a startupProbe with liveness for apps that take more than a few seconds to boot.
- Make the health endpoint cheap and dependency-light.
- Tune probe timings against measured boot time, not defaults.