What is Unicode?

Many of the writing systems used today are thousands of years old while digital text is still relatively new. In the early days of computer technology, digital representation of texts was mainly oriented around English. But nowadays much of global human interaction takes place online. People exchange information across linguistic and national borders. This change called for the development of a standardized structure for the exchange of texts in different alphabets and systems of writing. At the same time, technological advancements opened up new possibilities for displaying characters.

A good example are the emojis on your smartphone. These fun icons can be inserted with a special keyboard and used just like letters, almost as if they were a natural part of the alphabet. But how does that work? The Unicode standard forms the basis of it.

Unicode stands for universal character encoding. It is a standard for the binary coding of letters, numbers, and other characters and enables texts to be saved and processed in digital systems.

What makes Unicode special (and innovative at the time it came out) is that it’s not bound by the formats and encodings of any single human language. It was created to serve as a uniform standard for the representation of all human languages and writing systems.

Since Unicode 1.0 was released in 1991, the standard has successfully served its purpose. It is used in browsers and operating systems as a uniform format. Version 13.0, released in 2020 by the Unicode Consortium, boasts a repertoire of 143,859 characters.

The Unicode Consortium, a nonprofit organization based in California, is responsible for the continued development of the standard. Leading tech companies like Adobe, Apple, Facebook, Google, IBM, Microsoft, Netflix, and SAP are members of the consortium. The Unicode character set is fully congruent with the Universal Coded Character Set (UCS), defined by the International Standard ISO/IEC 10646.

Text and writing are present everywhere in modern life. Reading and writing are among the first things we learn at school. So, it doesn’t come as a surprise that many people take the presence of digital text for granted. But how exactly does the technical representation of writing work? Let’s take a trip into the world of digital character coding.

Before we move on, it’s important to understand that on a deeper level all information in digital systems consists of sequences of zeros and ones. This is called binary representation. Binary code can be compared to an alphabet, in which there are only two “letters” (zero and one). Each place in a series of zeros and ones is referred to as a bit.

The basic idea is to represent the characters of various alphabets as sequences of zeros and ones. This is how letters and numbers are encoded, as well as any other distinguishable states. All these characters are referred to as “symbols”. The longer a sequence of zeros and ones is, the more symbols can be represented. The number of symbols that can be represented doubles with every bit you add to the sequence.

A concrete example: If we have binary “words” that are two bits long, we can encode four numbers.

If we insert another bit at the beginning of the sequence, the number of possible bit words doubles. The new “words” will consist of the bit sequences from above, with either a zero or one placed before them. So now we can encode eight numbers:

For the sake of simplicity, we’ve demonstrated the encoding of numbers here. But the same principle comes into play for encoding letters or any other kind of character or state. See the following simplified example of the binary encoding of letters:

Keep in mind that our explanation up to this point has had nothing to do with writing systems. We’re just talking about the internal model that’s used for the digital representation of characters. The graphic representation of a character is called a glyph. There are various glyphs for the same character, depending on the font used. Even within one font there can be several variants for a glyph - e.g., for bold and italics. The following table illustrates this encoding:

Digitally encoding characters touches on several specific concepts. To provide a precise definition of Unicode, we’ve summed up some of the most important terms here:

Before the arrival of Unicode, there were a large variety of specific encodings. The norm was to establish a separate encoding for each language or language family. This frequently led to display errors or data inconsistencies. To prevent this, character encodings were often modelled as a downward-compatible superset of an already-existing standard. In this way, the modern Unicode standard builds on the earlier encoding ISO Latin-1, which in turn is based on the ASCII encoding.

The Unicode standard defines characters and corresponding code points for letters, syllable characters, ideograms, punctuation marks, special characters, and numerals. It supports not only the Latin alphabet but also the Greek, Cyrillic, Arabic, Hebrew, and Thai alphabets, as well as Japanese (Katakana, Hiragana), Chinese, and Korean (Hangul) writing systems. In addition, there are also mathematical, commercial, and technical characters, and historical control characters for teleprinters.

The characters are summarized in a series of character tables. We’ll give you an overview of the most common character tables.

The Unicode standard serves as the universal foundation for processing, saving, and sharing text in every language. Most modern software components that work with text, such as libraries, protocols, and databases, are based on Unicode. In this section, we’ll illustrate the wide range of use cases for Unicode.

Unicode is the internal standard for representing text in most operating systems. Some operating systems, such as Apple’s macOS, allow the use of Unicode characters in file names.

The Unicode variant UTF-8 has become the standard for encoding HTML documents. In 2016, more than 80 percent of the most visited websites in the world used UTF-8 for saving and representing their HTML documents. The Punycode is now established for the use of domain names with Unicode letters not from ASCII.

Many modern programming languages use Unicode as a basis for processing text. A newer development is the option to use Unicode characters to name variables and functions. This is possible in, for example, ECMAScript/JavaScript. The following example shows how Unicode characters are used in code:

The popular and widely used database MySQL supports the complete Unicode character set with the character encoding “utf8mb4”. When, on the other hand, the character encoding “utf8” is used, Unicode letters whose code points are more than 3 bytes are lost.

Fonts contain the glyphs for the graphic representation of text. Due to the large number of Unicode characters, there is no single font that contains every character. In fact, there are only a few fonts that even cover the Basic Multilingual Plane in its entirety. Here are a few examples:

In many cases, people use Unicode without ever being aware of it. Digital text usually appears in Unicode and can be copied, pasted, and edited by users. Sometimes a user needs to insert a specific Unicode letter into a text. There are various ways to do this, as we explain below.

Special onscreen keyboards (also called virtual keyboards or soft keyboards) are probably the most common way to insert Unicode characters into text. On keyboards on mobile devices, you can easily switch between different languages and alphabets. In each case, the keyboard layout will change, but all characters will come from the Unicode repertoire. The characters can be mixed and combined.

Good examples for this are emojis: Emojis are perfectly normal Unicode characters, just like numbers and letters. The display of emojis is independent of their internal modelling. Each operating system displays emojis slightly differently.

Onscreen keyboards are also used on desktop computers. In Windows, macOS, and many Linux distributions, virtual keyboards can be opened and display various Unicode characters, depending on the language you’ve chosen. Since the number of keys is limited, not all Unicode letters will be displayed. Rather, you’ll see a language-specific collection of the most common characters.

Aside from onscreen keyboards, Unicode character tables are probably one of the most convenient ways to access Unicode letters. A quick review: A coded character set is the set of all the characters and their corresponding unique code points. This kind of structure can easily be displayed as a table, which in the case of Unicode are called code charts. These tables are used to copy specific characters, which can then be pasted elsewhere. They can also be consulted by the end user to find out a code point for use as a numeric character reference - more on this in the next section.

Many desktop operating systems also have a Unicode character table, which provides an overview of all the available Unicode characters, including code point, description, and glyphs. A character can be copied and pasted in one click. You can also make a character table yourself with just a few lines of code. Later in this article, you’ll find an example of this using Python.

The assignment of characters to code points is a crucial part of the Unicode standard. If you know the code point for a character, you’ll be able to use the character in various contexts. In Windows, Unicode symbols are inserted using a special key combination on the normal hard keyboard. Please note that the code point number normally has to be entered in hexadecimal notation.

Programmers usually require the numeric character reference. The hexadecimal notation of a code point allows the Unicode character to be displayed in characters from the ASCII character set. We’ll show you how that process works in HTML; the process is generally the same for Python, C++, and other languages.

The general formula for inserting a character using its numeric reference includes the reference itself as well as an opening and closing term. In HTML documents, the numeric reference is opened with “&#x” and closed with “;”. The two to four digit hexadecimal code point should be inserted in between these two terms, resulting in the pattern “&#xNNNN;”.

For example, to insert the copyright symbol “©” into an HTML document, you should follow these steps:

1. Look for the character in the Unicode table.
2. Check the code point for the character. In this example, the code point is “U+00A9” in hexadecimal notation.
3. Create the character reference and insert it into the HTML source text or a markdown document.

In our case, we’ll insert “©”, which will give us the rendered character “©”.

There is also a less common method, which uses code points in decimal rather than hexadecimal notation. In this case, the numerical reference will begin with “&#” (without the “x”) and end as before with “;”. The code point in decimal notation comes in between. For the copyright symbol, the result would be “©”.

Since writing Unicode characters as numerical references isn’t intuitive for humans, there is also another method - named character entities. They’re defined for frequently used characters and assign each character a short, memorable name. A named character entity begins with an ampersand “&” and ends with a semicolon “;”. The name is placed in between the two symbols without any spaces. So, for example, to make the copyright sign “©” in HTML, you can simply write “©”.

Most programming languages contain fundamental functions that can be used to transform characters and code points. These functions are often called “ord(Character)” and “chr(Code point)”. The functions work together as follows:

‘chr(ord(Character)) == Character’

Note that it’s always possible to ascertain which code point corresponds to a character. The other way around will only work for numbers that are defined as code points in the coded character set. In the following example, we show how these functions generally work in Python.

You can easily create a character table for code points from the Unicode character set using these functions. Just iterate the code points and output the corresponding character. This can be done with just a few lines of code in Python:

The International Components for Unicode (ICU) are collected in a program library made available by the Unicode Consortium. The library is published with an open source license and can be used on many different operating systems. The software facilitates programmatic internationalization (often abbreviated as “i18n”). Its applications include:

• Processing Unicode texts
• Supporting regular expressions in Unicode
• Parsing and formatting calendar data, date and time information, numbers, currencies, and messages

There are two versions of the ICU library:

• “icu4c” is written in C/C++ and provides an API for these languages.
• “icu4j” is written in Java and provides an API for this language.

The implementation of the components delivers consistent results independent of the platform being used.

Most HTML documents use the character encoding UTF-8. To ensure that a page is displayed without character errors, you should place a “charset” meta specification in the head of the HTML document. This instructs the browser that the requested document should be interpreted as UTF-8. See the following example:

The popular social media platform Twitter doesn’t allow any text formatting in its tweets, profiles, or usernames, limiting the creative possibilities for users. Resourceful developers have found a way around this: Twitter uses Unicode, which means that it’s possible to use special characters to compose a text that looks formatted. Especially useful are characters that are similar to letters from the Latin alphabet. The easiest way to do this is using a Twitter Fonts Generator.