Fix Split UTF-8 Characters in Python Streams with an Incremental Decoder
Reassemble Korean text and emoji across byte chunks with Python’s incremental decoder. Test every split point and handle finalization, empty chunks, and invalid input.
Hun Jang Nov 11, 2026
If Korean text breaks or raises UnicodeDecodeError while reading UTF-8 in chunks from a network or file, check whether the code calls bytes.decode() separately on each chunk. A character’s bytes can span two chunks, so keep one incremental decoder per stream and check the remaining bytes with final=True when the stream ends normally. The Python example tests every two-part split of Korean text and an emoji, and shows how to detect incomplete input.
Separate cyan data fragments pass through an amber buffer and join into a continuous ribbon
Why chunk boundaries differ from character boundaries
The UTF-8 representation of the Korean character 한 is ed 95 9c. If the first chunk contains only ed, it cannot form a complete character. Even if the next chunk starts with 95 9c, independent decoding has no access to the leading byte from the previous chunk. Four-byte characters such as emoji have the same issue.
errors="ignore" discards invalid sequences, and errors="replace" inserts a replacement character. Applying either policy when a valid character merely spans chunks can lose the original text. An incremental decoder retains incomplete bytes and processes them with the next input. Python codecs documentation · Error handlers
Keep one decoder per stream
Save the following as incremental_utf8.py and run python3 incremental_utf8.py. It uses only the standard library. decode_utf8_chunks is a generator that accepts a finite iterable of byte chunks and yields completed text fragments.
Use final=False during iteration, then pass empty bytes with final=True when the input iterable ends normally. That last call detects an incomplete character left at the end. With strict, invalid bytes and characters that remain incomplete at the end raise UnicodeDecodeError. Python IncrementalDecoder.decode documentation
Verify split positions and stream termination
This output comes from a run on Linux with Python 3.12.14 on November 11, 2026. 한글🙂ABC occupies 13 UTF-8 bytes. The example splits it at all 14 positions, including the start and end, and checks that decoding restores the original text. It also tests one-byte chunks and an empty chunk inserted while the decoder holds pending bytes.
The empty chunk in the example is temporary empty data yielded by the input iterable. An empty input with final=False does not tell the decoder that the stream has ended. When a real socket’s recv() returns empty bytes, however, it means the connection has closed. The receive loop must end iteration and perform the final check at that point. Python socket HOWTO
Connect the decoder to TTS, JSON, or SSE processing
A text fragment emitted by the decoder may still be only part of a sentence or JSON object. After restoring the text, handle the line, event, or message boundaries of the relevant protocol separately. For TTS input, apply sentence buffering after that step. Keep bytes from different responses or connections in separate decoder instances.
The example verifies decoding with chunks constructed in memory. It does not test a live TTS API or HTTP client. If a client already supplies decoded strings, check that interface’s buffering and error handling rules first.
The generator must be consumed to completion for finalization to run. Handle cancellation or transport errors separately from normal termination. Some text may already have reached the consumer before an error is detected at the end. If a storage operation requires the entire document to be valid, commit the result after full validation.
BlogPro로 만든 사이트