PyTorch "CUDA error: an illegal memory access was encountered" in CI
A CUDA kernel accessed memory outside a valid allocation and corrupted the context (CUDA error 700). Like a device-side assert, the async reporting can blame a later op; the real cause is usually an out-of-range index, a shape mismatch, or mixed-device indexing.
What this error means
A step fails with "RuntimeError: CUDA error: an illegal memory access was encountered", after which every later CUDA call also errors because the context is poisoned.
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the
stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.Common causes
An out-of-range index into a GPU tensor
A gather, index_select, or embedding lookup with an index beyond the tensor bounds reads invalid GPU memory and faults.
A shape mismatch or stale pointer in a custom op
A custom CUDA kernel launched with the wrong dimensions, or operating on a freed tensor, writes outside its allocation.
How to fix it
Localize the fault with blocking launches
- Set
CUDA_LAUNCH_BLOCKING=1so the error surfaces at the real kernel. - Reproduce on CPU where an out-of-range index gives a precise message.
- Check the indices and shapes feeding the named op for out-of-bounds values.
CUDA_LAUNCH_BLOCKING=1 python train.pyValidate indices and shapes before GPU ops
Bound-check index tensors and assert expected shapes so the fault is caught with a readable error instead of a GPU fault.
assert idx.max().item() < table.size(0)
assert idx.min().item() >= 0How to prevent it
- Keep CUDA_LAUNCH_BLOCKING=1 in CI for precise GPU tracebacks.
- Bound-check index/gather tensors before the GPU op.
- Add a CPU smoke test that catches index errors deterministically.