Go bufio.Scanner: Fix token too long and Enforce Line-Length Limits

If a Go program using bufio.Scanner stops processing after a long line from a file or standard input, check Scanner.Err() after the loop first. Exceeding the default token buffer limit makes Scan() return false, so code that ignores errors can mistake the failure for normal EOF. Set an appropriate buffer before scanning, and enforce the application's actual line-length limit separately after receiving each token.

A continuous paper ribbon and differently sized intake slots illustrate long lines and reading limits. This is conceptual artwork, not Go's actual memory layout.

Written and documentation checked: 2026-11-02. Execution environment: Go 1.24.7, Linux amd64, standard-library bufio. The results below come from string-based reproductions, not user logs, production services, or execution on the latest Go release.

Mistaking loop termination for successful completion

A bare for scanner.Scan() { ... } loop does not tell you why iteration stopped. The official Go Scan and Err documentation explains that both EOF and errors can end iteration, and Err() returns errors other than EOF.

The reproduction input consists of 70KiB of ASCII x followed by one LF. In this environment, the default scanner stops with bufio.Scanner: token too long before returning a token. Formats that put JSON, long paths, or synthesis request text on one line can have a large individual line independently of total file size.

The default MaxScanTokenSize is 64 × 1024. Do not interpret that as a contract accepting exactly 65,536 bytes of line content: space for delimiters can matter too. Separate parser headroom from the application's allowed content size instead of relying on an exact boundary. Official constant and Buffer documentation

Separate the reading buffer from application policy

The example keeps two policies independent.

Call Buffer before the first Scan(). The example's 128KiB is a comparison value, not a recommended default. Choose it based on the data contract, concurrency, and memory budget.

Do not treat the max argument of Buffer(buf, max) as a strict input policy by itself. The final scenario below reads a 96KiB line despite setting the second argument to 32KiB, because the initial buffer is 128KiB. It demonstrates that a small second argument does not constrain an already-large initial buffer.

Runnable code: long-token errors and a separate length check

Save this as scanner_demo.go and run go run scanner_demo.go. No external modules are required. maxLine == 0 disables only the application limit for comparison; it does not remove the scanner's own limit.

The example collects lengths rather than line contents. It returns no result slice on error, but that does not roll back database writes or output already performed in a real processing loop. Work that must succeed as a whole needs separate staging or transaction policies.

Actual output and nine checked scenarios

All nine scenarios were executed. The last one illustrates the different roles of scanner configuration and application validation. Lengths are bytes rather than character counts, and this code validates neither JSON syntax nor UTF-8 validity.

Before retrying or switching to Reader

The official Go Scanner source commentary states that when scanning stops, the underlying reader may have advanced beyond the last token. Do not assume that attaching a new scanner to the same failed stream is a recovery procedure. Retry from retained input or a reopened file, and account for duplicate processing of earlier lines.

• For line-based input with a known limit, a configured Scanner with explicit error handling is straightforward.

• Consider bufio.Reader when large lines need fragment processing or you need direct control of reading position after errors. With ReadSlice, handle fragments and ErrBufferFull when the buffer fills. Switching to ReadString does not automatically impose resource limits. Official Reader documentation

• Copy s.Bytes() if retaining it for later use, because a subsequent scan may overwrite it. This example reads its length immediately and does not retain the slice.

• A scanner buffer limit is not a limit on total input, line count, or processing time. Add those limits separately for HTTP requests or long-lived streams.

Only default and enlarged buffers and line-length checks were executed for this article. Real network I/O errors, throughput or memory benchmarks, and production load tests were not performed.

한국어 버전: Go bufio.Scanner가 긴 줄에서 멈출 때: token too long과 Buffer 한도 처리