Character encoding

From Wikipedia for FEVERv2
Jump to navigation Jump to search

In computing, data storage, and data transmission, character encoding is used to represent a repertoire of characters by some kind of encoding system that assigns a number to each character for digital representation. Character encoding_sentence_0

Depending on the abstraction level and context, corresponding code points and the resulting code space may be regarded as bit patterns, octets, natural numbers, electrical pulses, etc. A character encoding is used in computation, data storage, and transmission of textual data. Character encoding_sentence_1

"Character set", "character map", "codeset" and "code page" are related, but not identical, terms. Character encoding_sentence_2

Early character codes associated with the optical or electrical telegraph could only represent a subset of the characters used in written languages, sometimes restricted to upper case letters, numerals and some punctuation only. Character encoding_sentence_3

The low cost of digital representation of data in modern computer systems allows more elaborate character codes (such as Unicode) which represent most of the characters used in many written languages. Character encoding_sentence_4

Character encoding using internationally accepted standards permits worldwide interchange of text in electronic form. Character encoding_sentence_5

History Character encoding_section_0

The history of character codes illustrates the evolving need for machine-mediated character-based symbolic information over a distance, using once-novel electrical means. Character encoding_sentence_6

The earliest codes were based upon manual and hand-written encoding and cyphering systems, such as Bacon's cipher, Braille, International maritime signal flags, and the 4-digit encoding of Chinese characters for a Chinese telegraph code (Hans Schjellerup, 1869). Character encoding_sentence_7

With the adoption of electrical and electro-mechanical techniques these earliest codes were adapted to the new capabilities and limitations of the early machines. Character encoding_sentence_8

The earliest well-known electrically-transmitted character code, Morse code, introduced in the 1840s, used a system of four "symbols" (short signal, long signal, short space, long space) to generate codes of variable length. Character encoding_sentence_9

Though most commercial use of Morse code was via machinery, it was also used as a manual code, generatable by hand on a telegraph key and decipherable by ear, and persists in amateur radio use. Character encoding_sentence_10

Most codes are of fixed per-character length or variable-length sequences of fixed-length codes (e.g. Unicode). Character encoding_sentence_11

Common examples of character encoding systems include Morse code, the Baudot code, the American Standard Code for Information Interchange (ASCII) and Unicode. Character encoding_sentence_12

Unicode, a well defined and extensible encoding system, has supplanted most earlier character encodings, but the path of code development to the present is fairly well known. Character encoding_sentence_13

The Baudot code, a five-bit encoding, was created by Émile Baudot in 1870, patented in 1874, modified by Donald Murray in 1901, and standardized by CCITT as International Telegraph Alphabet No. Character encoding_sentence_14

2 (ITA2) in 1930. Character encoding_sentence_15

The name "baudot" has been erroneously applied to ITA2 and its many variants. Character encoding_sentence_16

ITA2 suffered from many shortcomings and was often "improved" by many equipment manufacturers, sometimes creating compatibility issues. Character encoding_sentence_17

In 1959 the U.S. military defined its Fieldata code, a six-or seven-bit code, introduced by the U.S. Army Signal Corps. Character encoding_sentence_18

While Fieldata addressed many of the then-modern issues (e.g. letter and digit codes arranged for machine collation), Fieldata fell short of its goals and was short-lived. Character encoding_sentence_19

In 1963 the first ASCII (American Standard Code for Information Interchange) code was released (X3.4-1963) by the ASCII committee (which contained at least one member of the Fieldata committee, W. F. Leubbert) which addressed most of the shortcomings of Fieldata, using a simpler code. Character encoding_sentence_20

Many of the changes were subtle, such as collatable character sets within certain numeric ranges. Character encoding_sentence_21

ASCII63 was a success, widely adopted by industry, and with the follow-up issue of the 1967 ASCII code (which added lower-case letters and fixed some "control code" issues) ASCII67 was adopted fairly widely. Character encoding_sentence_22

ASCII67's American-centric nature was somewhat addressed in the European ECMA-6 standard. Character encoding_sentence_23

Somewhat historically isolated, IBM's Binary Coded Decimal (BCD) was a six-bit encoding scheme used by IBM in as early as 1959 in its 1401 and 1620 computers, and in its 7000 Series (for example, 704, 7040, 709 and 7090 computers), as well as in associated peripherals. Character encoding_sentence_24

BCD extended existing simple four-bit numeric encoding to include alphabetic and special characters, mapping it easily to punch-card encoding which was already in widespread use. Character encoding_sentence_25

It was the precursor to EBCDIC. Character encoding_sentence_26

For the most part, IBMs codes were used primarily with IBM equipment, which was more or less a closed ecosystem, and did not see much adoption outside of IBM "circles". Character encoding_sentence_27

IBM's Extended Binary Coded Decimal Interchange Code (usually abbreviated as EBCDIC) is an eight-bit encoding scheme developed in 1963. Character encoding_sentence_28

The limitations of such sets soon became apparent, and a number of ad hoc methods were developed to extend them. Character encoding_sentence_29

The need to support more writing systems for different languages, including the CJK family of East Asian scripts, required support for a far larger number of characters and demanded a systematic approach to character encoding rather than the previous ad hoc approaches. Character encoding_sentence_30

In trying to develop universally interchangeable character encodings, researchers in the 1980s faced the dilemma that on the one hand, it seemed necessary to add more bits to accommodate additional characters, but on the other hand, for the users of the relatively small character set of the Latin alphabet (who still constituted the majority of computer users), those additional bits were a colossal waste of then-scarce and expensive computing resources (as they would always be zeroed out for such users). Character encoding_sentence_31

In 1985, the average user's hard disk drive could store only about 10 megabytes, so it was very important at the time to make every bit count. Character encoding_sentence_32

The compromise solution that was eventually found and developed into Unicode was to break the assumption (dating back to telegraph codes) that each character should always directly correspond to a particular sequence of bits. Character encoding_sentence_33

Instead, characters would first be mapped to a universal intermediate representation in the form of abstract numbers called code points. Character encoding_sentence_34

Code points would then be represented in a variety of ways and with various default numbers of bits per character (code units) depending on context. Character encoding_sentence_35

To encode code points higher than the length of the code unit, such as above 256 for 8-bit units, the solution was to implement variable-width encodings where an escape sequence would signal that subsequent bits should be parsed as a higher code point. Character encoding_sentence_36

Terminology Character encoding_section_1

Character encoding_description_list_0

Character encoding_unordered_list_1

  • A character is a minimal unit of text that has semantic value.Character encoding_item_1_0
  • A character set is a collection of characters that might be used by multiple languages. Example: The Latin character set is used by English and most European languages, though the Greek character set is used only by the Greek language.Character encoding_item_1_1
  • A coded character set is a character set in which each character corresponds to a unique number.Character encoding_item_1_2
  • A code point of a coded character set is any allowed value in the character set or code space.Character encoding_item_1_3
  • A code space is a range of integers whose values are code pointsCharacter encoding_item_1_4
  • A code unit is a bit sequence used to encode each character of a repertoire within a given encoding form. This is referred to as a code value in some documents.Character encoding_item_1_5

Character encoding_description_list_2

The character repertoire is an abstract set of more than one million characters found in a wide variety of scripts including Latin, Cyrillic, Chinese, Korean, Japanese, Hebrew, and Aramaic. Character encoding_sentence_37

Other symbols such as musical notation are also included in the character repertoire. Character encoding_sentence_38

Both the Unicode and GB18030 standards have a character repertoire. Character encoding_sentence_39

As new characters are added to one standard, the other standard also adds those characters, to maintain parity. Character encoding_sentence_40

The code unit size is equivalent to the bit measurement for the particular encoding: Character encoding_sentence_41

Character encoding_unordered_list_3

  • A code unit in US-ASCII consists of 7 bits;Character encoding_item_3_6
  • A code unit in UTF-8, EBCDIC and GB18030 consists of 8 bits;Character encoding_item_3_7
  • A code unit in UTF-16 consists of 16 bits;Character encoding_item_3_8
  • A code unit in UTF-32 consists of 32 bits.Character encoding_item_3_9

Example of a code unit: Consider a string of the letters "abc" followed by U+10400 𐐀 DESERET CAPITAL LETTER LONG I (represented with 1 char32_t, 2 char16_t or 4 char8_t). Character encoding_sentence_42

That string contains: Character encoding_sentence_43

Character encoding_unordered_list_4

  • four characters;Character encoding_item_4_10
  • four code pointsCharacter encoding_item_4_11
  • either:Character encoding_item_4_12
    • four code units in UTF-32 (00000061, 00000062, 00000063, 00010400)Character encoding_item_4_13
    • five code units in UTF-16 (0061, 0062, 0063, d801, dc00), orCharacter encoding_item_4_14
    • seven code units in UTF-8 (61, 62, 63, f0, 90, 90, 80).Character encoding_item_4_15

The convention to refer to a character in Unicode is to start with 'U+' followed by the codepoint value in hexadecimal. Character encoding_sentence_44

The range of valid code points for the Unicode standard is U+0000 to U+10FFFF, inclusive, divided in 17 planes, identified by the numbers 0 to 16. Character encoding_sentence_45

Characters in the range U+0000 to U+FFFF are in plane 0, called the Basic Multilingual Plane (BMP). Character encoding_sentence_46

This plane contains most commonly-used characters. Character encoding_sentence_47

Characters in the range U+10000 to U+10FFFF in the other planes are called supplementary characters. Character encoding_sentence_48

The following table shows examples of code point values: Character encoding_sentence_49

Character encoding_table_general_0

CharacterCharacter encoding_header_cell_0_0_0 Unicode code pointCharacter encoding_header_cell_0_0_1 GlyphCharacter encoding_header_cell_0_0_2
Latin ACharacter encoding_cell_0_1_0 U+0041Character encoding_cell_0_1_1 ΑCharacter encoding_cell_0_1_2
Latin sharp SCharacter encoding_cell_0_2_0 U+00DFCharacter encoding_cell_0_2_1 ßCharacter encoding_cell_0_2_2
Han for EastCharacter encoding_cell_0_3_0 U+6771Character encoding_cell_0_3_1 Character encoding_cell_0_3_2
AmpersandCharacter encoding_cell_0_4_0 U+0026Character encoding_cell_0_4_1 &Character encoding_cell_0_4_2
Inverted exclamation markCharacter encoding_cell_0_5_0 U+00A1Character encoding_cell_0_5_1 ¡Character encoding_cell_0_5_2
Section signCharacter encoding_cell_0_6_0 U+00A7Character encoding_cell_0_6_1 §Character encoding_cell_0_6_2

A code point is represented by a sequence of code units. Character encoding_sentence_50

The mapping is defined by the encoding. Character encoding_sentence_51

Thus, the number of code units required to represent a code point depends on the encoding: Character encoding_sentence_52

Character encoding_unordered_list_5

  • UTF-8: code points map to a sequence of one, two, three or four code units.Character encoding_item_5_16
  • UTF-16: code units are twice as long as 8-bit code units. Therefore, any code point with a scalar value less than U+10000 is encoded with a single code unit. Code points with a value U+10000 or higher require two code units each. These pairs of code units have a unique term in UTF-16:Character encoding_item_5_17
  • UTF-32: the 32-bit code unit is large enough that every code point is represented as a single code unit.Character encoding_item_5_18
  • GB18030: multiple code units per code point are common, because of the small code units. Code points are mapped to one, two, or four code units.Character encoding_item_5_19

Unicode encoding model Character encoding_section_2

Unicode and its parallel standard, the ISO/IEC 10646 Universal Character Set, together constitute a modern, unified character encoding. Character encoding_sentence_53

Rather than mapping characters directly to octets (bytes), they separately define what characters are available, corresponding natural numbers (code points), how those numbers are encoded as a series of fixed-size natural numbers (code units), and finally how those units are encoded as a stream of octets. Character encoding_sentence_54

The purpose of this decomposition is to establish a universal set of characters that can be encoded in a variety of ways. Character encoding_sentence_55

To describe this model correctly requires more precise terms than "character set" and "character encoding." Character encoding_sentence_56

The terms used in the modern model follow: Character encoding_sentence_57

A character repertoire is the full set of abstract characters that a system supports. Character encoding_sentence_58

The repertoire may be closed, i.e. no additions are allowed without creating a new standard (as is the case with ASCII and most of the ISO-8859 series), or it may be open, allowing additions (as is the case with Unicode and to a limited extent the Windows code pages). Character encoding_sentence_59

The characters in a given repertoire reflect decisions that have been made about how to divide writing systems into basic information units. Character encoding_sentence_60

The basic variants of the Latin, Greek and Cyrillic alphabets can be broken down into letters, digits, punctuation, and a few special characters such as the space, which can all be arranged in simple linear sequences that are displayed in the same order they are read. Character encoding_sentence_61

But even with these alphabets, diacritics pose a complication: they can be regarded either as part of a single character containing a letter and diacritic (known as a precomposed character), or as separate characters. Character encoding_sentence_62

The former allows a far simpler text handling system but the latter allows any letter/diacritic combination to be used in text. Character encoding_sentence_63

Ligatures pose similar problems. Character encoding_sentence_64

Other writing systems, such as Arabic and Hebrew, are represented with more complex character repertoires due to the need to accommodate things like bidirectional text and glyphs that are joined together in different ways for different situations. Character encoding_sentence_65

A coded character set (CCS) is a function that maps characters to code points (each code point represents one character). Character encoding_sentence_66

For example, in a given repertoire, the capital letter "A" in the Latin alphabet might be represented by the code point 65, the character "B" to 66, and so on. Character encoding_sentence_67

Multiple coded character sets may share the same repertoire; for example ISO/IEC 8859-1 and IBM code pages 037 and 500 all cover the same repertoire but map them to different code points. Character encoding_sentence_68

A character encoding form (CEF) is the mapping of code points to code units to facilitate storage in a system that represents numbers as bit sequences of fixed length (i.e. practically any computer system). Character encoding_sentence_69

For example, a system that stores numeric information in 16-bit units can only directly represent code points 0 to 65,535 in each unit, but larger code points (say, 65,536 to 1.4 million) could be represented by using multiple 16-bit units. Character encoding_sentence_70

This correspondence is defined by a CEF. Character encoding_sentence_71

Next, a character encoding scheme (CES) is the mapping of code units to a sequence of octets to facilitate storage on an octet-based file system or transmission over an octet-based network. Character encoding_sentence_72

Simple character encoding schemes include UTF-8, UTF-16BE, UTF-32BE, UTF-16LE or UTF-32LE; compound character encoding schemes, such as UTF-16, UTF-32 and ISO/IEC 2022, switch between several simple schemes by using byte order marks or escape sequences; compressing schemes try to minimise the number of bytes used per code unit (such as SCSU, BOCU, and Punycode). Character encoding_sentence_73

Although UTF-32BE is a simpler CES, most systems working with Unicode use either UTF-8, which is backward compatible with fixed-width ASCII and maps Unicode code points to variable-width sequences of octets, or UTF-16BE, which is backward compatible with fixed-width UCS-2BE and maps Unicode code points to variable-width sequences of 16-bit words. Character encoding_sentence_74

See comparison of Unicode encodings for a detailed discussion. Character encoding_sentence_75

Finally, there may be a higher level protocol which supplies additional information to select the particular variant of a Unicode character, particularly where there are regional variants that have been 'unified' in Unicode as the same character. Character encoding_sentence_76

An example is the XML attribute xml:lang. Character encoding_sentence_77

The Unicode model uses the term character map for historical systems which directly assign a sequence of characters to a sequence of bytes, covering all of CCS, CEF and CES layers. Character encoding_sentence_78

Character sets, character maps and code pages Character encoding_section_3

Historically, the terms "character encoding", "character map", "character set" and "code page" were synonymous in computer science, as the same standard would specify a repertoire of characters and how they were to be encoded into a stream of code units – usually with a single character per code unit. Character encoding_sentence_79

But now the terms have related but distinct meanings, due to efforts by standards bodies to use precise terminology when writing about and unifying many different encoding systems. Character encoding_sentence_80

Regardless, the terms are still used interchangeably, with character set being nearly ubiquitous. Character encoding_sentence_81

A "code page" usually means a byte-oriented encoding, but with regard to some suite of encodings (covering different scripts), where many characters share the same codes in most or all those code pages. Character encoding_sentence_82

Well-known code page suites are "Windows" (based on Windows-1252) and "IBM"/"DOS" (based on code page 437), see Windows code page for details. Character encoding_sentence_83

Most, but not all, encodings referred to as code pages are single-byte encodings (but see octet on byte size.) Character encoding_sentence_84

IBM's Character Data Representation Architecture (CDRA) designates with coded character set identifiers (CCSIDs) and each of which is variously called a "charset", "character set", "code page", or "CHARMAP". Character encoding_sentence_85

The term "code page" does not occur in Unix or Linux where "charmap" is preferred, usually in the larger context of locales. Character encoding_sentence_86

In contrast to a "coded character set", a "character encoding" is a map from abstract characters to code words. Character encoding_sentence_87

A "character set" in HTTP (and MIME) parlance is the same as a character encoding (but not the same as CCS). Character encoding_sentence_88

"Legacy encoding" is a term sometimes used to characterize old character encodings, but with an ambiguity of sense. Character encoding_sentence_89

Most of its use is in the context of Unicodification, where it refers to encodings that fail to cover all Unicode code points, or, more generally, using a somewhat different character repertoire: several code points representing one Unicode character, or versa (see e.g. code page 437). Character encoding_sentence_90

Some sources refer to an encoding as legacy only because it preceded Unicode. Character encoding_sentence_91

All Windows code pages are usually referred to as legacy, both because they antedate Unicode and because they are unable to represent all 2 possible Unicode code points. Character encoding_sentence_92

Character encoding translation Character encoding_section_4

As a result of having many character encoding methods in use (and the need for backward compatibility with archived data), many computer programs have been developed to translate data between encoding schemes as a form of data transcoding. Character encoding_sentence_93

Some of these are cited below. Character encoding_sentence_94

Cross-platform: Character encoding_sentence_95

Character encoding_unordered_list_6

  • Web browsers – most modern web browsers feature automatic character encoding detection. On Firefox 3, for example, see the View/Character Encoding submenu.Character encoding_item_6_20
  • iconv – program and standardized API to convert encodingsCharacter encoding_item_6_21
  • luit – program that converts encoding of input and output to programs running interactivelyCharacter encoding_item_6_22
  • convert_encoding.py – Python based utility to convert text files between arbitrary encodings and line endings.Character encoding_item_6_23
  • decodeh.py – algorithm and module to heuristically guess the encoding of a string.Character encoding_item_6_24
  • International Components for Unicode – A set of C and Java libraries to perform charset conversion. uconv can be used from ICU4C.Character encoding_item_6_25
  • – This is a translation of the Mozilla automatic-encoding-detection code into the Python computer language.Character encoding_item_6_26
  • The newer versions of the Unix command attempt to do a basic detection of character encoding (also available on Cygwin).Character encoding_item_6_27
  • C++ template library with simple interface to convert between C++/user-defined streams. charset defined many character-sets and allows you to use Unicode formats with support of endianness.Character encoding_item_6_28

Unix-like: Character encoding_sentence_96

Character encoding_unordered_list_7

  • cmv – simple tool for transcoding filenames.Character encoding_item_7_29
  • convmv – convert a filename from one encoding to another.Character encoding_item_7_30
  • cstocs – convert file contents from one encoding to another for the Czech and Slovak languages.Character encoding_item_7_31
  • enca – analyzes encodings for given text files.Character encoding_item_7_32
  • recode – convert file contents from one encoding to anotherCharacter encoding_item_7_33
  • utrac – convert file contents from one encoding to another.Character encoding_item_7_34

Windows: Character encoding_sentence_97

Character encoding_unordered_list_8

  • Encoding.Convert – .NET APICharacter encoding_item_8_35
  • MultiByteToWideChar/WideCharToMultiByte – Convert from ANSI to Unicode & Unicode to ANSICharacter encoding_item_8_36
  • cscvt – character set conversion toolCharacter encoding_item_8_37
  • enca – analyzes encodings for given text files.Character encoding_item_8_38

See also Character encoding_section_5

Character encoding_unordered_list_9

  • Percent encodingCharacter encoding_item_9_39
  • Alt codeCharacter encoding_item_9_40
  • Character encodings in HTMLCharacter encoding_item_9_41
  • :Category:Character encoding – articles related to character encoding in generalCharacter encoding_item_9_42
  • :Category:Character sets – articles detailing specific character encodingsCharacter encoding_item_9_43
  • Hexadecimal representationsCharacter encoding_item_9_44
  • Mojibake – character set mismap.Character encoding_item_9_45
  • Mojikyo – a system ("glyph set") that includes over 100,000 Chinese character drawings, modern and ancient, popular and obscure.Character encoding_item_9_46
  • TRON, part of the TRON project, is an encoding system that does not use Han Unification; instead, it uses "control codes" to switch between 16-bit "planes" of characters.Character encoding_item_9_47
  • Universal Character Set charactersCharacter encoding_item_9_48
  • Charset sniffing – used in some applications when character encoding metadata is not availableCharacter encoding_item_9_49

Common character encodings Character encoding_section_6

Credits to the contents of this page go to the authors of the corresponding Wikipedia page: en.wikipedia.org/wiki/Character encoding.