In a groundbreaking research achievement, scholars at Mohamed bin Zayed University of Artificial Intelligence have developed the first benchmark for assessing the ability of artificial intelligence models to understand and interact with Arab culture across 13 local dialects. The study highlights a significant gap between AI models’ capacity to comprehend what Arabs express and their ability to communicate in ways that reflect daily conversational practices.
The Challenge of Cultural Nuance
When an AI model is tasked with describing an Emirati wedding, it generally provides accurate details regarding customs, traditions, and attire, alongside appropriate congratulatory phrases for the families of the newlyweds. However, when asked to express these concepts in the manner typical to Emiratis, the fluency, spontaneity, and cultural familiarity demonstrated earlier tend to diminish.
This paradox forms the core of the recent study from the university, revealing that while large language models possess extensive knowledge of Arab culture, they still struggle to articulate this knowledge in the authentic voices and precise dialects of their users.
Bridging the Gap in Understanding
The study’s lead researchers, Professor Fajri Kotto, an assistant professor in the Department of Natural Language Processing, and researcher Mohamed Dihaan shared insights with the Emirates News Agency (WAM). They assert that addressing this gap paves the way for the next phase in the development of Arabic-speaking AI, not just to grasp the language but also to appreciate its nuanced use across different Arab societies.
Dihaan noted, “The Arabic language unites over 400 million speakers worldwide. Yet a prominent challenge in developing AI lies in the fact that models have been predominantly trained on Modern Standard Arabic, which may make them appear fluent in tests, despite insufficient exposure to natural conversations in various Arabic dialects, including the precise details of local cultures.”
Introducing the ‘ArabCulture-Dialogue’ Benchmark
To tackle this issue, researchers have established the “ArabCulture-Dialogue” standard, the first reference framework for testing cultural inference in Arabic through multi-turn dialogues, incorporating both Modern Standard Arabic and 13 local dialects from diverse Arab nations.
To ensure accuracy and relevance to real-world language use, the team engaged 26 native Arabic speakers, with two speakers from each of the 13 Arab countries, each having lived in their respective countries for at least a decade.
The participants based their dialogues on scenarios derived from their cultural environments, evolving them into brief exchanges, which they then articulated in their local dialects. The dialogue set encompassed 12 topics from daily life, including weddings, food, child-rearing, agriculture, arts, and games.
Performance Analysis and Findings
Kotto explained that the researchers assessed the models through three main tasks: selecting the most culturally appropriate response from several options, translating between Modern Standard Arabic and a specific dialect in both directions, and continuing the dialogue in the required dialect.
The results indicated that the stronger models were capable of identifying culturally suitable responses, achieving scores close to 95% even when transitioning from Modern Standard Arabic to a specific dialect. However, their performance sharply declined when the tasks shifted from understanding dialects to producing them—whether translating a sentence into Emirati dialect or maintaining a dialogue in it.
These findings reveal a subtle but significant gap in AI development; the issue is not solely about understanding meaning but about the ability to retain the dialect’s identity and cultural specificity.
Dialect Distinctions and Challenges
The study noted that the models excelled in addressing shared customs throughout the Arab world, yet encountered greater difficulties grasping the unique traditions of each country. Conversations in North African dialects proved to be among the most challenging, while Emirati dialogues ranked as particularly demanding for the models.
Models successfully generated the target dialect in less than half of the instances, and some specialised Arabic models only managed this in a few cases. Conversely, the models fared significantly better in translating from dialects to Modern Standard Arabic than in the opposite direction, as reverting the text to standard Arabic was easier, while imbuing it with an authentic local tone presented the greatest challenge.
The Path Forward for AI in the Arab World
Dihaan highlighted a paradox: while cultural knowledge resides within the models, it often requires minimal direction. The identification of a specific country and region for the dialogue led to enhanced accuracy in outcomes.
This insight suggests that advancing Arabic AI necessitates not only larger datasets but also more diverse and precise data that genuinely reflect the social and cultural contexts in which the language is used.
These findings carry strategic importance for the UAE, which has prioritised AI as a national agenda, from the National AI Strategy 2031 to the development of local AI models like “Jais,” a bilingual large language model supporting both Arabic and English, in which Mohamed bin Zayed University has played a pivotal role.
Expanding the Horizon of AI Research
This study underscores the university’s leading role in elevating Arabic AI research from general language support to a more advanced understanding of linguistic and cultural diversity within the Arab world.
The next step is not just for AI to speak Arabic but to comprehend which variant it is speaking, in what context, and with whom, while expressing local culture without losing its uniqueness.
Kotto concluded, “The broader takeaway from the study sends a clarion call to all who develop Arabic language AI models: the system that supports Arabic does not inherently understand the language across its various dialects and the cultures and identities these dialects embody.”
The results pave the way for a new era of scientific exploration, enabling the expansion of the dialects covered by the benchmark and the development of more accurate methods to evaluate models’ abilities to understand local cultural contexts and produce language authentically.
The research paper titled “Cultural Metric Evaluation of Large Language Models in Modern Standard Arabic and Arabic Dialects” was presented at the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2026), serving as a foundation for future research aimed at developing AI models with greater capacities for understanding the linguistic and cultural diversity of the Arab world, while simultaneously bolstering efforts to preserve local cultures and identities.