260 likes | 470 Views
From UCS-2 to UTF-16. Discussion and practical example for the transition of a Unicode library from UCS-2 to UTF-16. Why is this an issue?. The concept of the Unicode standard changed during its first few years Unicode 2.0 (1996) expanded the code point range from 64k to 1.1M
E N D
From UCS-2 to UTF-16 Discussion and practical example for the transition of a Unicode library from UCS-2 to UTF-16
Why is this an issue? • The concept of the Unicode standard changed during its first few years • Unicode 2.0 (1996) expanded the code point range from 64k to 1.1M • APIs and libraries need to follow this change and support the full range • Upcoming character assignments (Unicode 3.1, 2001) fall into the added range
“Unicode is a 16-bit character set” • Concept: 16-bit, fixed-width character set • Saving space by not including precomposed, rarely-used, obsolete, … characters • Compatibility, transition strategies, and acceptance forced loosening of these principles • Unicode 3.1: >90k assigned characters
16-bit APIs • APIs developed for Unicode 1.1 used 16-bit characters and strings: UCS-2 • Assuming 1:1 character:code unit • Examples: Win32, Java, COM, ICU, Qt/KDE • Byte-based UTF-8 (1993) mostly for MBCS compatibility and transfer protocols
Extending the range • Set aside two blocks of 1k 16-bit values, “surrogates”, for extension • 1k x 1k = 1M = 10000016 additional code points using a pair of code units • 16-bit form now variable-width UTF-16 • “Unicode scalar values” 0..10ffff16 • Proposed: 1994; part of Unicode 2.0 (1996)
Parallel with ISO-10646 • ISO-10646 uses 31-bit codes: UCS-4 • UCS-2: 16-bit codes for subset 0..ffff16 • UTF-16: transformation of subset 0..10ffff16 • UTF-8 covers all 31 bits • Private Use areas above 10ffff16 slated for removal from ISO-10646 for UTF interoperability and synchronization with Unicode
21-bit code points • Code points (“Unicode scalar values”) up to 10ffff16 use 21 bits • 16-bit code units still good for strings: variable-width like MBCS • Default string unit size not big enough for code points • Dual types for programming?
C: char/wchar_t dual types • C/C++ standards: dual types • Strings mostly with char units (8 bits) • Code points: wchar_t, 8..32 bits • Typical use in I18N-ed programs: (8-bit) char strings but (16/32-bit) wchar_t (or 32-bit int) characters; code point type is implementation-dependent
Unicode: dual types, too? • Strings could continue with 16-bit units • Single code points could get 32-bit data type • Dual-type model like C/C++ MBCS
Alternatives to dual 16/32 types • UTF-32: all types 32 bits wide, fixed-width • UTF-8: same complexity after range extension beyond just the BMP, closer to C/C++ model – byte-based • Use pairs of 16-bit units • Use strings for everything • Make string unit size flexible 8/16/32 bits
UCS-2 to UTF-32 • Fixed-width, single base type for strings and code points • UCS-2 programming assumptions mostly intact • Wastes at least 33% space, typically 50% • Performance bottleneck CPU - memory
UCS-2 to UTF-8 • UCS-2 programming assumes many characters in single code units • Breaks a lot of code • Same question of type for code points; follow C model, 32-bit wchar_t? – More difficult transition than other choices
Surrogate pairs for single chars • Caller avoids code point calculation • But: caller and callee need to detect and handle pairs: caller choosing argument values, callee checking for errors • Harder to use with code point constants because they are published as scalar values • Significant change for caller from using scalars
Strings for single chars • Always pass in string (and offset) • Most general, handles graphemes in addition to code points • Harder to use with code point constants because they are published as scalar values • Significant change for caller from using scalars
UTF-flexible • In principle, if the implementation can handle variable-width, MBCS-style strings, could it handle any UTF-size as a compile-time choice? • Adds interoperability with UTF-8/32 APIs • Almost no assumptions possible • Complexity of transition even higher than of transition to pure UTF-8, performance?
Interoperability • Break existing API users no more than necessary • Interoperability with other APIs: Win32, Java, COM, now also XML DOM • UTF-16 is Unicode default: good compromise (speed/ease/space) • String units should stay 16 bits wide
Does everything need to change? • String operations: search, substring, concatenation, … work with any UTF without change • Character property lookup and similar: need to support the extended range • Formatting: should handle more code points or even graphemes • Careful evaluation of all public APIs
ICU: some of all • Strings: UTF-16, UChar type remains 16-bit • New UChar32 for code points • Provide macros for C to deal with all UTFs: iteration, random access, … • C++ CharacterIterator: many new functions • Property lookup/low-level: UChar32 • Formatting: strings for graphemes
Scalar code points:property lookup • Old, 16-bit:UChar u_tolower(UChar c){ u[v[c15..7]+c6..0];} • New, 21-bit:UChar32 u_tolower(UChar32 c){ u[v[w[c20..10]+c9..4]+c3..0];}
Formatting: grapheme strings • Old:void setDecimalSymbol(UChar c); • New:void setDecimalSymbol(const UnicodeString &s);
Codepage conversion • To Unicode: results are one or two UTF-16 code units, surrogates stored directly in the conversion table • From Unicode: triple-stage compact array access from 21-bit code points like property lookup • Single-character-conversion to Unicode now returns UChar32 values
API first… • Tools and basic functions and classes are in place (property lookup, conversion, iterators, BiDi) • Public APIs reviewed and changed (“luxury” of early project stage) or deprecated and superseded by new versions • Higher-level implementations to follow before Unicode 3.1 published
More implementations follow… • Collation: need to prepare for >64k primary keys • Normalization and Transliteration • Word/Sentence break iteration • Etc. • No non-BMP data before Unicode 3.1 is stable
Other libraries • Java: planning stage for transition • Win32: rendering and UniScribe API largely UTF-16-ready • Linux: standardizing on 32-bit Unicode wchar_t, has UTF-8 locales like other Unixes for char* APIs • W3C: standards assume full UTF-16 range
Summary • Transition from UCS-2 to UTF-16 gains importance after four years of standard • APIs for single characters need change or new versions • String APIs: no change • Implementations need to handle 21-bit code points • Range of options
Resources • Unicode FAQ: http://www.unicode.org/unicode/faq/ • Unicode on IBM developerWorks: http://www.ibm.com/developer/unicode/ • ICU: http://oss.software.ibm.com/icu/