CI Container "memory cgroup out of memory" - cgroup Limit Hit
The OOM kill came from a cgroup, not the whole host. A container or systemd scope with a memory ceiling was exceeded, so the kernel killed a process inside that cgroup - even though the host may have had free RAM.
What this error means
The kernel log shows Memory cgroup out of memory: Killed process .... A containerized step dies (often exit 137) while free -m on the host still shows available memory - the limit is the cgroup’s, not the machine’s.
Memory cgroup out of memory: Killed process 4821 (node)
total-vm:6291456kB, anon-rss:2097152kB ...Common causes
The container memory limit is too low
A --memory/-m flag, a compose mem_limit, or a Kubernetes pod memory limit caps the cgroup. Exceeding it triggers a cgroup-scoped OOM kill regardless of host free memory.
The job legitimately needs more than the cap
A build or test that grew over time now peaks above a previously-sufficient cgroup limit. Nothing is wrong with the code; the ceiling is simply too tight now.
How to fix it
Find and raise the cgroup limit
Read the effective limit, then raise it where the limit is set.
cat /sys/fs/cgroup/memory.max # cgroup v2
cat /sys/fs/cgroup/memory/memory.limit_in_bytes # cgroup v1
# raise it where set, e.g. docker run -m 6g ...Lower the cgroup’s peak usage
- Cap the in-container heap below the cgroup limit (e.g.
--max-old-space-size,-Xmx). - Reduce build/test parallelism inside the container.
- Split a heavy step so no single process needs the full ceiling at once.
How to prevent it
- Set container memory limits with headroom above real peak usage.
- Keep language heap limits strictly below the cgroup limit.
- Alert on cgroup memory high-water marks so caps stay ahead of growth.