Daemonless build "no space left on device" in CI

By Kaveh Alemi·Latchkey

Daemonless builders unpack the full base image and every layer onto the job filesystem. On a small ephemeral disk, or when Buildah uses vfs (which copies rather than shares layers), the disk fills and writes fail with no space left on device.

What this error means

A Kaniko or Buildah build fails mid-way with "write ...: no space left on device" or "mkdir ...: no space left on device" while unpacking or running a step.

error building image: error building stage: failed to execute command:
extracting fs: write /usr/lib/.../libLLVM.so: no space left on device

Common causes

The ephemeral job disk is too small for the image

Large base images plus layers exceed the runner disk, especially with big toolchains or ML images.

vfs multiplies disk usage

Buildah on vfs copies the whole filesystem per layer instead of sharing, so disk use balloons and fills the volume.

How to fix it

Give the build more disk and use overlay

1. Increase the job or pod ephemeral storage request.
2. Switch Buildah off vfs to overlay/fuse-overlayfs to stop copying every layer.
3. Slim the base image and clean apt/pip caches within the build.

resources:
  requests:
    ephemeral-storage: "20Gi"

Reduce what the build writes

Use a smaller base image and remove package caches in the same RUN so layers stay small.

RUN apt-get update && apt-get install -y --no-install-recommends libpq-dev \
  && rm -rf /var/lib/apt/lists/*

How to prevent it

• Right-size ephemeral storage for the image being built.
• Use overlay/fuse-overlayfs, not vfs, to avoid copying every layer.
• Keep base images slim and clean caches within RUN steps.

Frequently asked questions

Related guides

References

• Docker documentation
• Docker build cache
• GitHub Actions documentation