Python subprocess Hangs on Large Output: Fix PIPE Deadlocks with communicate

If a Python subprocess hangs only when it produces more output, check whether the parent creates stdout=PIPE or stderr=PIPE and then calls wait() before reading. For a batch job with bounded output, use subprocess.run(capture_output=True, check=True, timeout=...) or communicate() to consume output while waiting for completion. This article reproduces the pipe stall on Linux and compares timeout cleanup with handling larger output.

Concept illustration of unread output accumulating in a pipe and two output channels being drained. It is not an exact process diagram.

Written and documentation checked: November 4, 2026. Execution environment: Linux, CPython 3.12.14. The explanation targets the Python 3.12 documentation. These are functional checks using synthetic child processes, not performance measurements of a real service or the author's workload.

A parent waiting for exit and a child waiting to write

A pipe is not an unlimited store. A blocking write to a full pipe cannot proceed until a reader makes room. Do not assume a fixed default capacity. Linux pipe(7)

In the first example, the child writes 2 MiB to stdout while the parent waits for exit without reading. The child must finish writing before it can exit; the parent must see exit before moving on. Without a timeout, this circular wait persists. Python's wait warning describes this problem with unread pipes.

Reading stdout to completion and then reading stderr is also unsafe. The example child writes stdout and then stderr, keeping stdout open in the meantime. If stderr fills while the parent waits for stdout EOF, neither side can proceed. Both streams must be serviced.

Reproduction: separate waiting, collection, failure, and timeout

Save the following as subprocess_demo.py and run python3 subprocess_demo.py on Linux. No external packages are required. This is a Linux-only reproduction using fcntl and pipesize, and assumes Python 3.10 or later.

The first case sets both pipes to 4,096 bytes instead of relying on the default capacity. The 0.2-second timeout is a test value for observing the stall, not a recommended production deadline. The later five-second limit is for verifying the synthetic command. Every child exits or is explicitly cleaned up, and the final temporary file is deleted when closed.

Actual output:

Check 1 timed out in wait() while the child remained alive. Check 2 resumed handling that same child with communicate(), collected exactly 2,097,152 bytes from each stream, and verified exit code 0. Check 3 handled the same workload with run(). Check 4 verified exit code 7 and preservation of stderr. Check 5 verified that the child was still alive immediately after the communicate() timeout. Check 6 checked that the temporary file containing both streams was 4,194,304 bytes long.

Use run for bounded batch output, communicate for direct management

For a real batch wrapper, follow check 3: pass the command as an argument list and handle its exit status, deadline, and output together. This example compares bytes and does not decode text. If human-readable logs are needed, first verify the actual tool's output encoding.

When managing Popen directly, the exception path in check 5 matters. A communicate(timeout=...) timeout does not itself kill the child, so the example calls kill() and then communicate() again to finish termination and output handling. In contrast, run(timeout=...) kills and reaps its direct child before raising the timeout exception. Python's communicate and run documentation

This distinction affects execution policy. Prefer run() for simple batches, and make ownership of child cleanup explicit in custom Popen paths, including exceptions. Record nonzero exit status separately from exceeding the deadline.

Choose output handling by volume

communicate() and capture_output=True accumulate output in memory. A time limit is not a byte limit: a command can emit substantial logs quickly. Python's memory warning

Check 6 removes the buffer that would retain all output in a Python result object. File writes still need operating-system cache and storage space. File redirection does not solve full disks or slow storage. Combining the streams also loses their identity; use separate files when that distinction matters.

The -u option reduces the effect of the Python child's own output buffering in this reproduction. It does not fix a parent that fails to read its pipes. Increasing pipe capacity likewise postpones a finite stall rather than replacing continuous consumption.

Scope and remaining checks

These checks target one Python child that creates no descendants. If a real tool starts further processes or passes pipe write handles to them, killing only the direct child may still leave EOF or overall job completion delayed. Verify process-tree termination policy separately.

The code was not tested with FFmpeg, a TTS engine, or a cloud batch job. Interactive stdin protocols, live-consumer implementations, and memory caps for unlimited output are outside its verified scope. Before production use, run the actual command with bounded input and check stdout/stderr size, exit status, and any work remaining after the deadline.

한국어 버전: Python subprocess가 출력이 많을 때 멈추는 이유: wait와 communicate로 파이프 처리하기