Arabic diacritics

The literal meaning of تَشْكِيل tashkīl is 'forming'. As the normal Arabic text does not provide enough information about the correct pronunciation, the main purpose of tashkīl (and ḥarakāt) is to provide a phonetic guide or a phonetic aid; i.e. show the correct pronunciation for children who are learning to read or foreign learners.

The bulk of Arabic script is written without ḥarakāt (or short vowels). However, they are commonly used in texts that demand strict adherence to exact pronunciation. This is true, primarily, of the Qur'an ⟨ٱلْقُرْآن⟩ (al-Qurʾān) and poetry. It is also quite common to add ḥarakāt to hadiths ⟨ٱلْحَدِيث⟩ (al-ḥadīth; plural: al-ḥādīth) and the Bible. Another use is in children's literature. Moreover, ḥarakāt are used in ordinary texts in individual words when an ambiguity of pronunciation cannot easily be resolved from context alone. Arabic dictionaries with vowel marks provide information about the correct pronunciation to both native and foreign Arabic speakers. In art and calligraphy, ḥarakāt might be used simply because their writing is considered aesthetically pleasing.

An example of a fully vocalised (vowelised or vowelled) Arabic from the Bismillah:

بِسْمِ ٱللَّٰهِ ٱلرَّحْمَٰنِ ٱلرَّحِيمِ
bismi -llāhi r-raḥmāni r-raḥīmⁱ
In the name of God, the All-Merciful, the Especially-Merciful.

Some Arabic textbooks for foreigners now use ḥarakāt as a phonetic guide to make learning reading Arabic easier. The other method used in textbooks is phonetic romanisation of unvocalised texts. Fully vocalised Arabic texts (i.e. Arabic texts with ḥarakāt/diacritics) are sought after by learners of Arabic. Some online bilingual dictionaries also provide ḥarakāt as a phonetic guide similarly to English dictionaries providing transcription.

7th-century kufic script without any ḥarakāt or i‘jām.

The i‘jām ⟨إِعْجَام⟩ (sometimes also called nuqaṭ)[6] are the diacritic points that distinguish various consonants that have the same form (rasm), such as ⟨ـبـ⟩ /b/ ب, ⟨ـتـ⟩ /t/ ت, ⟨ـثـ⟩ /θ/ ث, ⟨ـنـ⟩ /n/ ن, and ⟨ـيـ⟩ /j/ ي. Typically i‘jām are not considered diacritics but part of the letter.

Early manuscripts of the Qur’ān did not use diacritics either for vowels or to distinguish the different values of the rasm. Vowel pointing was introduced first, as a red dot placed above, below, or beside the rasm, and later consonant pointing was introduced, as thin, short black single or multiple dashes placed above or below the rasm (image). These i‘jām became black dots about the same time as the ḥarakāt became small black letters or strokes.

Typically, Egyptians do not use dots under final yā’ ⟨ي⟩, which looks exactly like alif maqṣūrah ⟨ى⟩ in handwriting and in print. This practice is also used in copies of the muṣḥaf (Qurʾān) scribed by ‘Uthman Ṭāhā. The same unification of yā and alif maqṣūrā has happened in Persian, resulting in what the Unicode Standard calls "Arabic Letter Farsi Yeh", that looks exactly the same as yā in initial and medial forms, but exactly the same as alif maqṣūrah in final and isolated forms ⟨یـ  ـیـ  ـی⟩.

Isolated kāf with ‘alāmātu-l-ihmāl and without top stroke next to initial kāf with top stroke.

At the time when the i‘jām was optional, unpointed letters were ambiguous. To clarify that a letter would lack i‘jām in pointed text (i.e. ⟨ح⟩ /ħ/, ⟨د⟩ /d/, ⟨ر⟩ /r/, ⟨س⟩ /s/, ⟨ص⟩ /sˤ/, ⟨ط⟩ /tˤ/, ⟨ع⟩ /ʕ/, ⟨ل⟩ /l/, ⟨ه⟩ /h/), the letter could be marked with a small v- or seagull-shaped diacritic above, also a superscript semicircle (crescent), a subscript dot (except in the case of ⟨ح⟩; three dots were used with ⟨س⟩), or a subscript miniature of the letter itself. A superscript stroke known as jarrah, resembling a long fatħah, was used for a contracted (assimilated) sin. Thus ⟨ڛ سۣ سۡ سٚ⟩ were all used to indicate that the letter in question was truly ⟨س⟩ and not ⟨ش⟩.[7] These signs, collectively known as ‘alāmātu-l-ihmāl, are still occasionally used in modern Arabic calligraphy, either for their original purpose (i.e. marking letters without i‘jām), or often as purely decorative space-fillers. The small ک above the kāf in its final and isolated forms ⟨ك  ـك⟩ was originally an ‘alāmatu-l-ihmāl that became a permanent part of the letter. Previously this sign could also appear above the medial form of kāf, when that letter was written without the stroke on its ascender. When kaf was written without that stroke, it could be mistaken for lam, thus kaf was distinguished with a superscript kaf or a small superscript hamza (nabrah), and lam with a superscript l-a-m (lam-alif-mim).[8]

Although normally a diacritic is not considered a letter of the alphabet, the hamza هَمْزة (hamzah, glottal stop), often stands as a separate letter in writing, is written in unpointed texts and is not considered a tashkīl. It may appear as a letter by itself or as a diacritic over or under an alif, wāw, or yā.

Which letter is to be used to support the hamzah depends on the quality of the adjacent vowels;

• If the glottal stop occurs at the beginning of the word, it is always indicated by hamza on an alif: above if the following vowel is /a/ or /u/ and below if it is /i/.
• If the glottal stop occurs in the middle of the word, hamzah above alif is used only if it is not preceded or followed by /i/ or /u/:
  • If /i/ is before or after the glottal stop, a yāʼ with a hamzah is used (the two dots which are usually beneath the yāʾ disappear in this case): ⟨ئ⟩.
  • Otherwise, if /u/ is before or after the glottal stop, a wāw with a hamzah is used: ⟨ؤ⟩.
• If the glottal stop occurs at the end of the word (ignoring any grammatical suffixes), if it follows a short vowel it is written above alif, wāw, or yā the same as for a medial case; otherwise on the line (i.e. if it follows a long vowel, diphthong or consonant).
• Two alifs in succession are never allowed: /ʔaː/ is written with alif maddah ⟨آ⟩ and /aːʔ/ is written with a free hamzah on the line ⟨اء⟩.

Consider the following words: ⟨أَخ⟩ /ʔax/ ("brother"), ⟨إسْماعِيل⟩ /ʔismaːʕiːl/ ("Ismael"), ⟨أُمّ⟩ /ʔumm/ ("mother"). All three of above words "begin" with a vowel opening the syllable, and in each case, alif is used to designate the initial glottal stop (the actual beginning). But if we consider middle syllables "beginning" with a vowel: ⟨نَشْأة⟩ /naʃʔa/ ("origin"), ⟨أَفْئِدة⟩ /ʔafʔida/ ("hearts"—notice the /ʔi/ syllable; singular ⟨فُؤاد⟩ /fuʔaːd/), ⟨رُؤُوس⟩ /ruʔuːs/ ("heads", singular ⟨رَأْس⟩ /raʔs/), the situation is different, as noted above. See the comprehensive article on hamzah for more details.

Historically Arabic script has been adopted and used by many tonal languages, examples include Xiao'erjing for Mandarin Chinese as well as Ajami script adopted for writing various languages of Western Africa. However, one of the shortcomings of Arabic, especially in comparison to Latin-derived scripts or other indigenous writing systems was that Arabic did not have a way of indicating tones.

However, in the adoption of the Arabic Script for Rohingya language, known as Rohingya Fonna, 3 tone markers have been developed and used in manuscripts. These tone markers form part of the standardized and accepted orthographic convention of Rohingya. This is the only known instance of tone markers within the Arabic script.[9][10]

Tone markers act as "modifiers" of vowel diacritics. In simpler words, they are "diacritics for the diacritics". They are written "outside" of the word, meaning that they are written above the vowel diacritic if the diacritic is written above the word, and they are written below the diacritic if the diacritic is written below the word. They are only ever written where there are vowel diacritics. This is important to note, as without the diacritic present, there is no way to distinguish between tone markers and I‘jām i.e. dots that are used for purpose of phonetic distinctions of consonants.

The Hārbāy as it is called in Rohingya, is a single dot that's placed on top of Fatḥah and Ḍammah, or curly Fatḥah and curly Ḍammah (vowel diacritics unique to Rohinghya), or their respective Fatḥatan and Ḍammatan versions, and it's placed underneath Kasrah or curly Kasrah, or their respective Kasratan version. (e.g. دً࣪ / دٌ࣪ / دࣨ࣪ / دٍ࣭‎) This tone marker indicates a short high tone (/˥/).[9][10]

The Ṭelā as it is called in Rohingya, is two dots that are placed on top of Fatḥah and Ḍammah, or curly Fatḥah and curly Ḍammah, or their respective Fatḥatan and Ḍammatan versions, and it's placed underneath Kasrah or curly Kasrah, or their respective Kasratan version. (e.g. دَ࣫ / دُ࣫ / دِ࣮‎) This tone marker indicates a long falling tone (/˥˩/).[9][10]

The Ṭāna as it is called in Rohingya, is a fish-like looping line that is placed on top of Fatḥah and Ḍammah, or curly Fatḥah and curly Ḍammah, or their respective Fatḥatan and Ḍammatan versions, and it's placed underneath Kasrah or curly Kasrah, or their respective Kasratan version. (e.g. دࣤ࣬ / دࣥ࣬ / دࣦ࣯‎) This tone marker indicates a long rising tone (/˨˦/).[9][10]

Evolution of early Arabic calligraphy (9th–11th century). The Basmala was taken as an example, from kufic Qur’ān manuscripts. (1) Early 9th century, script with no dots or diacritic marks (see image of early Basmala Kufic);
(2) and (3) 9th–10th century under Abbasid dynasty, Abu al-Aswad's system established red dots with each arrangement or position indicating a different short vowel; later, a second black-dot system was used to differentiate between letters like fā’ and qāf (see image of middle Kufic);
(4) 11th century, in al-Farāhídi's system (system we know today) dots were changed into shapes resembling the letters to transcribe the corresponding long vowels (see image of modern Kufic in Qur'an).

According to tradition, the first to commission a system of harakat was Ali who appointed Abu al-Aswad al-Du'ali for the task. Abu al-Aswad devised a system of dots to signal the three short vowels (along with their respective allophones) of Arabic. This system of dots predates the i‘jām, dots used to distinguish between different consonants.

• 
  Early Basmala Kufic
• 
  Middle Kufic
• 
  Modern Kufic in Qur'an

The process of automatically restoring diacritical marks is called diacritization or diacritic restoration. It is useful to avoid ambiguity in applications such as Arabic machine translation, text-to-speech, and information retrieval. Automatic diacritization algorithms have been developed.[12][13] For Modern Standard Arabic, the state-of-the-art algorithm has a word error rate (WER) of 4.79%. The most common mistakes are proper nouns and case endings.[14] Similar algorithms exist for other varieties of Arabic.[15]

• Arabic alphabet:
  • I‘rāb (إِعْرَاب), the case system of Arabic
  • Rasm (رَسْم), the basic system of Arabic consonants
  • Tajwīd (تَجْوِيد), the phonetic rules of recitation of Qur'an in Arabic
• Hebrew:
  • Hebrew diacritics, the Hebrew equivalent
  • Niqqud, the Hebrew equivalent of ḥarakāt
  • Dagesh, the Hebrew diacritic similar to Arabic i‘jām and shaddah

• Alexis Neme and Sébastien Paumier (2019), "Restoring Arabic vowels through omission-tolerant dictionary lookup", Lang Resources & Evaluation, Vol. 53, pp. 1-65