Unicode normalization
From Wikipedia, the free encyclopedia
Unicode normalization is a form of text normalization that transforms equivalent characters or sequences of characters into a consistent underlying representation so that they may be easily compared.
Contents |
[edit] Composition and Decomposition
Underlying Unicode's normalization methods is the concept of character composition and decomposition. Character composition is the process of combining simpler characters into fewer precomposed characters, such as the n character and the combining ~ character into the single ñ character. Decomposition is the opposite process, breaking precomposed characters back into their component pieces.
Unicode composes and decomposes based on characters and sequences it deems equivalent. It has two standards for this: canonical, which distinguishes between functionally equivalent but visually distinct characters, and compatibility, which does not. See the article on Unicode equivalence for more information.
[edit] Standards
Unicode defines four normalization standards.
| NFD Normalization Form Canonical Decomposition |
Characters are decomposed by canonical equivalence. |
| NFC Normalization Form Canonical Composition |
Characters are decomposed and then recomposed by canonical equivalence. It is possible for the result to be a different sequence of characters than the original. |
| NFKD Normalization Form Compatibility Decomposition |
Characters are decomposed by compatibility equivalence. |
| NFKC Normalization Form Compatibility Composition |
Characters are decomposed by compatibility equivalence, then recomposed by canonical equivalence. |
All the above methods will standardize the order in which decomposed characters appear, even sequences that were already decomposed prior to normalization. They may also replace characters or sequences with equivalent characters or sequences even if it doesn't result in the number of characters changing. These are done to achieve the consistency in encoding required for normalization.
