Ambiguity is a common phenomenon in natural languages. It takes many forms, including orthographic ambiguity. In Arabic, orthographic ambiguity is manifested by the absence of diacritics in most everyday texts. These diacritics indicate the correct vowelization of words. Hence, we refer to diacritical ambiguity in Arabic. To quantify and analyze diacritical ambiguity in Arabic, we conducted a statistical study at two levels: at the word level with a lexicon of 1.19 million unique diacritized tokens and at the text level with a dataset of 100,000 lines. According to the results, 4 out of 5 letters receive diacritics in Arabic. At the text level, more than 97% of words are ambiguous, meaning they accept more than one possible diacritization, and the average number of diacritization choices is rather high, up to 19.63. These findings demonstrate how significant diacritical ambiguity is in Arabic and how substantial diacritics are in Arabic script.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Statistical Investigation of Diacritical Ambiguity in Arabic

  • Maimouna Lapointe,
  • Ayoub Kadim,
  • Azzedine Dliou

摘要

Ambiguity is a common phenomenon in natural languages. It takes many forms, including orthographic ambiguity. In Arabic, orthographic ambiguity is manifested by the absence of diacritics in most everyday texts. These diacritics indicate the correct vowelization of words. Hence, we refer to diacritical ambiguity in Arabic. To quantify and analyze diacritical ambiguity in Arabic, we conducted a statistical study at two levels: at the word level with a lexicon of 1.19 million unique diacritized tokens and at the text level with a dataset of 100,000 lines. According to the results, 4 out of 5 letters receive diacritics in Arabic. At the text level, more than 97% of words are ambiguous, meaning they accept more than one possible diacritization, and the average number of diacritization choices is rather high, up to 19.63. These findings demonstrate how significant diacritical ambiguity is in Arabic and how substantial diacritics are in Arabic script.