Popularity of text encodings

Use of the main encodings on the web from 2001 to 2012 as recorded by Google,[1] with UTF-8 overtaking all others in 2008 and over 60% of the web in 2012 (since then approaching 100%). The ASCII-only figure includes all web pages that only contain ASCII characters, regardless of the declared header.

Declared character set for the 10 million most popular websites since 2010

UTF-8 has been the most common encoding for the World Wide Web since 2008.[2] As of October 2023, UTF-8 accounts for 98.0% of all web pages (and 99.0% of top 10,000 pages and 98.5% of the top 1,000 highest ranked web pages, the next most popular encoding, ISO-8859-1, is used by 13 of those sites).[3] Although many pages only use ASCII characters to display content, few websites now declare their encoding to only be ASCII instead of UTF-8.[4]

Virtually all countries and over 97% all of the tracked languages have 95% or more use of UTF-8 encodings on the web. See below for the major alternative encodings:

The second-most popular encoding varies depending on locale, and is typically more efficient for the associated language. One such encoding is the Chinese GB 18030 standard, which is a full Unicode Transformation Format, still 95.2% of websites in China and territories use UTF-8[5][6][7] with it (effectively[8]) the next popular encoding. Big5 is another popular Chinese (for traditional characters) encoding and is next-most popular in Taiwan after UTF-8 at 96.2%, and it's also second-most used in Hong Kong, while there as elsewhere, UTF-8 is even more dominant.[9] The single-byte Windows-1251 is twice as efficient for the Cyrillic script and still 94.3% of Russian websites use UTF-8[10] (however e.g. Greek and Hebrew encodings are also twice as efficient, and UTF-8 has over 99% use for those languages).[11][12] Japanese and Korean language websites also have relatively high non-UTF-8 use compared to most other countries, with Japanese UTF-8 use at 94.5% followed by the legacy Shift JIS and then EUC-JP encoding.[13][14][1] South Korea has 95.0% UTF-8 use, with the rest of websites mainly using EUC-KR which is more efficient for Korean text.

With the exception of GB 18030 (and UTF-16 and UTF-8), other (legacy) encodings do not do not support all Unicode characters, since they were designed for specific languages.

As of October 2023, Breton has the lowest UTF-8 use on the web of any tracked language, with 89.6% use.[15] Well over a third of the languages tracked have 100.0% use of UTF-8 on the web, such as Vietnamese, Marathi, Telugu, Tamil, Javanese, Pañjābī/Punjabi, Gujarati, Farsi/Persian, Hausa, Pashto, Kannada, Lao, Kurdish languages, Tagalog, Somali, Khmer/Cambodian, isiZulu/Zulu, Turkmen, Tajik (has 4 different scripts), and a lot of the languages with the fewest speakers (often with their own scripts) such as, Armenian, Mongolian (which has a top-to-bottom script[16]), Maldivian (Thaana), Greenlandic (Kalaallisut) and also sign languages.[17]

Local storage on computers has considerably more use of "legacy" single-byte encodings than on the web. Attempts to update to UTF-8 have been blocked by editors that do not display or write UTF-8 unless the first character in a file is a byte order mark, making it impossible for other software to use UTF-8 without being rewritten to ignore the byte order mark on input and add it on output. UTF-16 files are also fairly common on Windows, but not in other systems.[18][19]

In the memory of a computer program, usage of UTF-16 is very common, particularly in Windows but and also in JavaScript, Qt, and many other cross-platform software libraries. Compatibility with the Windows API is a major reason for this.

At one time it was believed by many (and is still believed today by some) that having fixed-size code units offers computational advantages, which led many systems, in particular Windows, to use the fixed-size UCS-2 with two bytes per character. This is false: strings are almost never randomly accessed, and sequential access is the same speed. In addition, even UCS-2 was not "fixed size" if combining characters are considered, and when Unicode exceeded 65536 code points it had to be replaced with the non-fixed-sized UTF-16 anyway.

Recently it has become clear that the overhead of translating from/to UTF-8 on input and output, and dealing with potential encoding errors in the input UTF-8, vastly overwhelms any savings UTF-16 could offer. So newer software systems are starting to use UTF-8. International Components for Unicode (ICU) has historically only used UTF-16, and still does only for Java, while it now supports UTF-8 (for C/C++ and other languages indirectly), e.g. used that way by Microsoft; supported as the "Default Charset"[20] including the correct handling of "illegal UTF-8".[21] The default string primitive used in newer programing languages, such as Go,[22] Julia, Rust and Swift 5,[23] assume UTF-8 encoding. PyPy is also using UTF-8 for its strings,[24] and Python is looking into storing all strings with UTF-8.[25] Microsoft now recommends the use of UTF-8 for applications using the Windows API, while continuing to maintain a legacy "Unicode" (meaning UTF-16) interface.[26]