GitLab Runner Kubernetes Executor - Pod Pending / ImagePullBackOff / OOMKilled

By Kaveh Alemi·Latchkey

With the Kubernetes executor, each job runs in a build pod. Jobs fail when that pod cannot be scheduled, cannot pull its image, or is OOMKilled - all cluster-side problems, not your script.

What this error means

The job fails in preparation or mid-run with a Kubernetes pod error surfaced in the log: the pod stays Pending (unschedulable), hits ImagePullBackOff, or is OOMKilled. Your script may never start.

Job log

ERROR: Job failed (system failure): prepare environment:
waiting for pod running: pod status is Pending
(0/3 nodes are available: 3 Insufficient memory)

Common causes

Pod cannot be scheduled

Insufficient CPU/memory on nodes, taints without tolerations, or a missing node selector leave the build pod Pending until capacity appears.

Image pull fails in the cluster

The build or helper image cannot be pulled - wrong registry, missing imagePullSecrets, or a transient registry outage - yielding ImagePullBackOff.

Container OOMKilled

Memory limits set too low in the runner’s Kubernetes config cause the build container to be OOMKilled when the job exceeds them.

How to fix it

Set sensible pod resources and scheduling

Configure requests/limits and any node selectors/tolerations in the runner’s Kubernetes executor config.

config.toml

[runners.kubernetes]
  image = "alpine:3.20"
  cpu_request = "500m"
  memory_request = "512Mi"
  memory_limit = "2Gi"
  [runners.kubernetes.node_selector]
    "kubernetes.io/arch" = "amd64"

Diagnose with kubectl

Find the build pod and run kubectl describe pod <pod> to see scheduling or pull events.
For ImagePullBackOff, fix the image reference or add imagePullSecrets for the registry.
A transient unschedulable/pull blip on an autoscaling cluster clears once a node or the registry recovers - retry the job.

How to prevent it

Right-size pod requests/limits to node capacity and real job usage.
Configure imagePullSecrets for private images used by jobs.
Use cluster autoscaling and retry: runner_system_failure for transient capacity blips.

Frequently asked questions

What causes "K8s executor pod errors"?