Python "OSError: [Errno 24] Too many open files" in CI

By Daniel Zoghalchali·Latchkey

The process hit its open-file-descriptor limit. Either code is leaking file handles or sockets without closing them, or the runner’s ulimit -n is too low for legitimate concurrency.

What this error means

A long run, a test suite, or a high-concurrency task fails with OSError: [Errno 24] Too many open files. It often appears only after many iterations or under load, and may surface from sockets, temp files, or subprocess pipes.

OSError: [Errno 24] Too many open files: 'data/chunk_4821.bin'
# or from sockets
OSError: [Errno 24] Too many open files
  File ".../http/client.py", line 941, in send

Common causes

Leaked file or socket handles

Files, sockets, or subprocess pipes opened without a with block (or never closed) accumulate until the descriptor table is exhausted.

ulimit -n too low for the workload

The runner’s soft limit on open files (often 1024) is too small for genuinely high concurrency, so even leak-free code can exhaust it.

How to fix it

Close handles with context managers

Use with so files and sockets are closed deterministically, even on error.

with open(path, "rb") as f:
    data = f.read()
# requests: use a session and close it, or 'with requests.Session() as s:'

Raise the descriptor limit

ulimit -n 65535
python run.py
# inside Python (soft up to hard limit):
# import resource; resource.setrlimit(resource.RLIMIT_NOFILE, (65535, 65535))

Find the leak

1. Print len(os.listdir("/proc/self/fd")) periodically to confirm growth.
2. Audit code paths that open files/sockets without a with or explicit close().
3. Cap parallelism (pool size, concurrent connections) to a sane bound.

How to prevent it

• Always open files/sockets with context managers.
• Bound concurrency and connection-pool sizes.
• Raise ulimit -n for genuinely high-fan-out workloads.

Frequently asked questions

Related guides

References

• Python documentation
• pip documentation
• GitHub Actions documentation