CI "Out of memory: Killed process" in dmesg - Reading the OOM Report

By Kaveh Alemi·Latchkey

When a CI step dies with a bare Killed, the kernel log holds the proof. The Out of memory: Killed process line names the victim and its memory at death - the definitive way to confirm an OOM rather than guessing.

What this error means

A step ends abruptly (often exit 137) and dmesg shows an oom-kill summary plus Out of memory: Killed process <pid> (<name>). The line reports the process’s RSS, which tells you how much memory it was using when the kernel stepped in.

Kernel log (dmesg)

[12345.678] oom-kill:constraint=CONSTRAINT_NONE,...,task=node,pid=4821
[12345.679] Out of memory: Killed process 4821 (node) total-vm:6291456kB,
anon-rss:4194304kB, file-rss:0kB, shmem-rss:0kB

Common causes

A process exceeded available memory and was chosen as the OOM victim

The kernel selects the task with the highest OOM score (largest memory footprint, usually) and kills it. The anon-rss field shows how much resident memory it held - the number to size the runner against.

The constraint field tells you the scope

constraint=CONSTRAINT_NONE means the whole host ran out; CONSTRAINT_MEMCG means a cgroup/container limit was hit. Reading it tells you whether to raise the runner size or a container limit.

How to fix it

Read the OOM report

Pull the kill line and the victim’s resident memory to confirm the cause and the size needed.

Terminal

dmesg -T | grep -i -E 'oom-kill|out of memory: killed'
# anon-rss in the killed-process line ≈ memory the victim needed

Size the runner or limit to the victim’s footprint

Move to a runner with more RAM than the victim’s peak anon-rss, with headroom.
If constraint=CONSTRAINT_MEMCG, raise the container/cgroup memory limit instead of the host size.
Lower the victim’s peak (smaller heap, fewer workers) if a bigger runner is not an option.

How to prevent it

Capture dmesg on failure so OOM kills are confirmable, not guessed.
Size runners against the observed peak RSS of the heaviest process.
Distinguish host-wide OOM from cgroup OOM via the constraint field.

Frequently asked questions

Why is dmesg the source of truth for an OOM?

A SIGKILL gives the process no chance to log why it died, so the CI log only shows Killed. The kernel, which did the killing, records the reason and the victim’s memory in dmesg - that is the authoritative record.

References

GitHub Actions documentation