Unicode character property

Unicode assigns character properties to each code point.^[1] These properties can be used to handle "characters" (code points) in processes, like in line-breaking, script direction right-to-left or applying controls. Slightly inconsequently, some "character properties" are also defined for code points that have no character assigned, and code points that are labeled like "<not a character>". The character properties are described in Standard Annex #44.^[2]

Properties have levels of forcefulness: normative, informative, contributory, or provisional. For practical reasons, a character property can be assigned by specifying a continuous range of code points that have the same property.

Character property

Name

Unicode characters are assigned a unique Name (na).^[1] The name, in English, is composed of uppercase letters A-Z, digits 0-9, - (hyphen-minus) and <space>. Some sequences are excluded: names beginning with a space or hyphen, names ending with a space or hyphen, repeated spaces or hyphens, and space after hyphen are not allowed. The name is guaranteed to be unique within Unicode, and can be used to identify a code point and its character. Ideographic characters, of which there are tens of thousands, are named in the pattern "cjk unified ideograph-hhhh". For example, U+4E00 一 cjk unified ideograph-4e00. Formatting characters are named too: U+00A0   no-break space.

Starting from Unicode version 2.0, the published name for a code point will never change. In the event of a misspelling in a publication, a correct name will later be assigned to the code point as an Character Name Alias. Within the whole range of names, an alias is unique too.

Apart from these normative names, informal names can be assigned. These are usually other commonly used names for a character, used for illustration, but these informal names are not guaranteed to be unique.

These code points do not have a Name (na=""): Controls (General Category: Cc), Private use (Co), Surrogate (Cs), Non-characters (Cn) and Reserved (Cn). They may be referenced, informally, by a generic or specific meta-name, called "Code Point Labels": <control>, <control-0088>, <reserved>, <noncharacter-hhhh>, <private-use-hhhh>, <surrogate>. Since these labels contain <>-brackets, they can never appear as a Name, which prevents confusion.

Version 1.0 names

In version 2.0 of Unicode, many names were changed. From then on the rule "a name will never change" came into effect, including the strict (normative) use of alias names. Disused version 1.0-names were moved to the property Alias, to provide some backward compatibility.

General Category

Each code point is assigned a value for General Category. This is one of the character properties that are also defined for unassigned code points, and code points that are defined "not a character".

Punctuation

Characters have separate properties to denote they are a punctuation character. The properties all have a Yes/No values: Dash, Diacritic, Quotation_Mark, Space, Terminal_Punctuation, Whitespace.

Whitespace

Whitespace is a commonly used concept for a typographic effect. Basically it covers invisible characters that have a spacing effect in rendered text. It includes spaces, tabs, and new line formatting controls. In Unicode, such a character has the property set "WSpace=yes". In version 6.3, there are 25 whitespace characters.

Other general characteristics

Ideographic, alphabetic, noncharacter.

Display-related properties

Shaping, width.

Bidirectional writing

Four character properties pertain to bi-directional writing: Bidirectional Character Type, (formally Bidi_Class); Bidi_Control, Bidi_Mirrored and Bidi_Mirroring_Glyph.

One of Unicode's major features is support of bi-directional (Bidi) text display R-to-L and L-to-R. The Unicode Bidirectional Algorithm UAX9^[7] describes the process of presenting text with altering script directions. For example, it enables a Hebrew quote in an English text. The Bidi_Character_Type marks a characters behaviour in directional writing. To override a direction, Unicode has defined seven special Bidi_controls, formatting control characters (LRM, LRE, LRO, RLM, RLE, RLO, PDF). These characters can enforce a direction, and by definition only affect bi-directional writing.

Each code point has a property called Bidirectional Character Type, formally Bidi_Class. It defines its behaviour in a bidirectional text as interpreted by the algorithm. There are 19 possible types.

In normal situations, the algorithm can determine the direction of a text by this character property. To control more complex Bidi situations, e.g. when an English text has a Hebrew quote, extra options are added to Unicode. Seven characters have the property Bidi_Control=Yes: LRM, RLM, LRE, RLE, PDF, LRO, RLO as named in the table. These are invisible formatting control characters, only used by the algorithm and with no effect outside of bidirectional formatting.^[7] Despite the name, they are formatting characters, not control characters, and have General category "Other, format (Cf)" in the Unicode definition.

Basically, the algorithm determines a sequence of characters with the same strong direction type (R-to-L or L-to-R), taking in account an overruling by the special Bidi-controls. Number strings (Weak types) are assigned a direction according to their strong environment, as are Neutral characters. Finally, the characters are displayed per string's direction.

Two other character properties are relevant to the bidirectional text: Bidi_Mirrored=Yes indicates that the glyph should be mirrored when written R-to-L. The property Bidi_Mirroring_Glyph=U+hhhh can then point to the mirrored character. For example, brackets "()" are mirrored this way. Shaping cursive scripts such as Arabic, and mirroring glyphs that have a direction, is not part of the algorithm.

Casing

The Case value is Normative in Unicode. It pertains to those scripts with uppercase (aka capital, majuscule) and the lowercase (aka small, minuscule) letter. Case-difference occurs in the scripts Latin, Greek, Coptic, Cyrillic, Glagolitic, Armenian, Deseret, and archaic Georgian.

(upper, lower, title, folding—both simple and full)

Numeric values and types

Decimal

Characters are classified with a Numeric type.^[1] Numeric are all characters such as fractions, subscripts, superscripts, Roman numerals, currency numerators, encircled numbers, and script-specific digits. All these have a numeric value that can be decimal, including zero and negatives, but also a vulgar fraction. If there is not such a value, as with most of the scripts, the numeric type is "None".

The characters that do have a numeric value are separated in three groups: Decimal (De), Digit (Di) and Numeric (Nu, i.e. all other). "Decimal" means the character is a straight decimal digit. Only characters that are part of a contiguous encoded range 0..9 have numeric type Decimal. Other digits, like superscripts, have numeric type Digit. All numeric characters like fractions and roman numerals end up with the type "Numeric". The intended effect is that an even more simple parser can use these decimal numeric values, without being distracted by say a numeric superscript or a fraction. Some 41 CJK Ideographs that represent a number, including those used for accounting, are typed Numeric.

On the other hand, characters that could have a numeric value as a second meaning are still marked Numeric type "None", and have no numeric value (""). E.g. Latin letters can be used in paragraph numbering like (II.A.1.b), but the letters "I", "A" and "b" are not numeric (type "None") and have no numeric value.

Hexadecimal digits

Hexadecimal characters are those in the series with hexadecimal values 0...9ABCDEF (sixteen characters, decimal value 0-15). The character property Hex_Digit is set to Yes when a character is in such a series. The series are:

Leaving out repetition of the decimals 0-9 (twice), 44 characters marked as such. The property ASCII_Hex_Digit marks only those hexadecimal characters that are in ASCII, i.e. the top two row from the table.

So Unicode has no separate characters for hexadecimal values. A consequence is, that when using regular characters it is impossible to determine whether hexadecimal value is intended, or even whether a value is intended at all. That should be determined at a higher level, e.g. by prepending "0x" to a hexadecimal number or by context. The only feature is that Unicode can note that a sequence can or can not be a hexadecimal value.

Block

A block is a named, continuous range of code points. It is identified by its first and last code point. It may contain code points that are reserved, not-assigned etc. Each character that is assigned, has a single "block name" value from the currently 209 names. Unassigned code points outside of an existing block, have the default value "No_block".

Script

Each assigned character can have a single value for its "Script" property, signifing to which script it belongs.^[16] The value is a four-letter code in the range Aaaa-Zzzz, as available in ISO 15924, which is mapped to a writing system. Apart from when describing the background and usage of a script, Unicode does not use a connection between a script and languages that use that script. So "Hebrew" refers to the Hebrew script, not to the Hebrew language.

The special code Zyyy for "Common" allows a single value for a character that is used in multiple scripts. The code Zinh "Inherited script", used for combining characters and certain other special-purpose code points, indicates that a character "inherits" its script identity from the character with which it is combined. (Unicode formerly used the private code Qaai for this purpose.) The code Zzzz "Unknown" is used for all characters that do not belong to a script (i.e. the default value), such as symbols and formatting characters. Overall, characters of a single script can be scattered over multiple blocks, like Latin characters. And the other way around too: multiple scripts can be present is a single block, even when the block name suggests different: e.g. block Letterlike Symbols contains characters from the Latin, Greek and Common scripts.

When the Script is "" (blank), according to Unicode the character does not belong to a script. This pertains to symbols, because the existing ISO script codes "Zmth" (Mathematical notation) and "Zsym" (Symbol) are not used in Unicode. The "Script" property is also blank for code points that are not a typographic character like controls, substitutes, and private use code points.

If there is a specific script alias name in ISO 15924, is used in the character name: U+0041 A latin capital letter a, and U+05D0 א hebrew letter alef.

Normalization properties

Decompositions, decomposition type, canonical combining class, composition exclusions, and more.

Age

Age is the version of the Standard in which the code point was first designated. The version number is shortened to the numbering major.minor, although there more detailed version numbers are used: versions 4.0.0 and 4.0.1 both are named 4.0 as Age. Given the releases, Age can be from the range: 1.0, 1.1, 2.0, 2.1, 3.0, 3.1, 3.2, 4.0, 4.1, 5.0, 5.1, 5.2, 6.0 and 6.1.^[17][18] Code points that are not assigned, have Age=Unassigned.

Deprecated

Once a character has been defined, it will not be withdrawn or changed in defining properties (code point, name). But it can be declared deprecated: A coded character whose use is strongly discouraged.^[19] As of version 6.1, 111 characters are deprecated. A deprecation is noted in the code chart, and usually an alternative is available.

Boundaries

(grapheme cluster, word, line, and sentence)

References

1. ↑ 1.0 1.1 1.2 Unicode 6.0 chapter 4
2. ↑ "Unicode Standard Annex #44: Unicode Character Database". The Unicode Standard. 2012-01-23version 6.1.0 
3. ↑ Unicode 6.0, Chapter 4, table 4-9
4. ↑ Unicode 6.0, Chapter 2, table 2-3: Types of code points
5. ↑ Stability policy: Property Value Stability and table. Stability policy: Some gc groups will never change. gc=Nd corresponds with Numeric Type=De (decimal).
6. ↑ Unicode 6.0, Chapter 4, table 4-12 Name=""; a Code Point Label may be used to identify a nameless code point. E.g. <control-hhhh>, <control-0088>. The Name remains blank, which can prevent inadvertently replacing, in documentation, a Control Name with a true Control code. Unicode also uses <not a character> for <noncharacter>.
7. ↑ 7.0 7.1 UAX 9, Standard Annex "Unicode Bidirectional Algorithm"
8. ↑ Unicode Blocks data file. As of Unicode version 6.3
9. ↑ UAX 24: Unicode Script Property (4alpha code)
10. ↑ UAX 24: Script data file
11. ↑ Including unassigned code points: non-character, reserved
12. ↑ The script has one or multiple characters in the block, as defined by the Script Property. This is independent of the block name
13. ↑ "Common" (Zyyy) and "Inherited" (Zinh or Qaai) refer to Scripts in ISO 15924
14. ↑ Called "C0 Controls and Basic Latin" in ISO/IEC 10646
15. ↑ Called "C1 Controls and Latin-1 Supplement" in ISO/IEC 10646
16. ↑ Unicode Standard Annex #24: Unicode Script Property
17. ↑ Pre version 4
18. ↑ Versions 4.0 and later
19. ↑ "3.4 Characters and Encoding: rule D13".