Unicode NFC vs NFKC: Fixing TTS Dictionary Matching

If a TTS user dictionary misses Korean text that looks identical to a stored entry, check the Unicode normalization forms of the input and dictionary keys first. Consistent NFC normalization is a useful starting point for general text matching. Choose NFKC, which also folds fullwidth characters and compatibility symbols, according to domain policy. This article reproduces the mismatch in Python and explains the implications for pronunciation dictionaries.

Concept illustration of matching separated pieces and a combined shape to a common form. This is not an exact character-conversion diagram.

Identical-looking Korean text can have different keys

The Korean syllable 가 can be represented as U+AC00 or as the sequence U+1100 followed by U+1161. They are canonically equivalent, but their unnormalized code point sequences differ. Visual rendering alone is not a reliable way to compare string keys. Unicode normalization FAQ

If a dictionary stores only U+AC00 while input contains the two-code-point form, a direct comparison or key lookup can fail. Storing both strings as UTF-8 does not automatically remove that distinction.

Choose NFC and NFKC deliberately

NFC performs canonical decomposition followed by composition where possible. NFKC additionally performs compatibility decomposition, removing more distinctions between representations. See Unicode UAX #15 for the normalization definitions and constraints.

These are results observed in the execution environment used for this article:

If your dictionary distinguishes ① from 1, NFKC merges them into the same key. It also converts ㎏ to kg, but that does not decide that the text should be spoken as “kilograms.” Unicode normalization and context-dependent verbalization of numbers, units, and text are separate operations.

Reproduce the fix: normalize input and dictionary keys consistently

Save the following UTF-8 code as normalize_demo.py and run it. No external packages are required. The Python unicodedata documentation describes the supported forms and behavior of normalize.

The first two assertions verify that the original strings differ and that normalizing the input to NFC makes the dictionary lookup succeed. The last assertion demonstrates that NFKC removes the distinction between the keys ① and 1. On success, the conversion results are followed by lookup and compatibility-key collision: PASS.

Apply the policy inside a TTS pipeline

The following is an engineering recommendation based on the reproduction. It is not a performance benchmark or an audio-quality evaluation of a specific TTS engine.

1. Preserve the original input and create a separate representation for dictionary lookup.

2. Apply the same policy when loading user and domain dictionaries, not only when processing incoming text.

3. Detect entries that acquire the same normalized key. If their pronunciations differ, report a collision rather than silently overwriting one.

4. Apply compatibility normalization only where folding forms such as fullwidth characters is appropriate. Analyze symbols first if their original distinctions matter.

5. Handle number and unit verbalization, language selection, and context separately.

Normalization can change text length. If subtitles or error messages must point back into the original input, account for position mapping. Concatenating individually normalized strings also does not always produce a normalized result. For streaming input, use a normalization implementation that preserves boundary state rather than normalizing arbitrary chunks independently. The Unicode FAQ discusses concatenation.

Scope and validation environment

The reproduction ran on Python 3.12.14 with Unicode database 15.0.0. Checks covered Korean text, combining accents, fullwidth characters, compatibility symbols, NFC dictionary lookup, and NFKC key collisions. Additional sample checks covered repeated NFC and NFKC application. This was not a complete Unicode conformance test or a pronunciation/audio-quality test of a real synthesis engine.

NFC does not merge every visually similar character, and NFKC is not a general visual-confusables detector either. Normalization also does not detect byte encodings or repair incorrectly decoded strings. An earlier Korean-language article discusses character sets and encodings:

한글 인코딩

References

• Unicode UAX #15: Normalization Forms

• Unicode normalization FAQ

• Python 3.12 unicodedata

한국어 버전

TTS 사전에서 같은 한글이 매칭되지 않을 때: Unicode NFC와 NFKC