In a significant development, Mohamed bin Zayed University for Artificial Intelligence is striving to foster a deeper understanding of Arabic culture through the lens of artificial intelligence (AI). This initiative aims not only to recognise various dialects but also to express the nuances embedded within them. Researchers at the university have pioneered a benchmark to assess AI models’ ability to comprehend and interact with Arabic culture across 13 local dialects.
Bridging the Cultural Gap
The research highlights a notable gap between AI models’ ability to understand Arabic speakers and their proficiency in communicating in a way that reflects the way those speakers converse in their everyday lives. For instance, while an AI model may accurately depict the customs and traditions of an Emirati wedding—including detailed descriptions of attire and appropriate congratulatory phrases—it struggles to articulate these elements in the local dialect typical of daily interactions.
A Significant Study
This paradox formed the crux of a recent study from the university, revealing that while large language models have extensive knowledge of Arabic culture, they face significant challenges when it comes to expressing this knowledge in the authentic voices and specific dialects of their speakers. Professors Fajri Koto and researcher Mohammed Dehan, the primary investigators of the study, remarked that closing this gap is crucial for advancing AI that can not only understand Arabic but also appreciate the intricacies of its use in diverse Arab communities.
The Challenge of Dialects
Dehan highlighted that Arabic unites over 400 million speakers globally, yet one of the key challenges in AI development is the almost exclusive training of models on Modern Standard Arabic. As a consequence, these models may perform well on testing but lack exposure to the rich array of natural conversations found in various dialects, missing out on local cultural details.
Introducing ArabCulture-Dialogue
To address this issue, researchers developed the ArabCulture-Dialogue benchmark, described as the first standard for testing cultural inference in Arabic language dialogues. This benchmark encompasses Modern Standard Arabic and 13 local dialects from different Arab nations. The research team involved 26 native Arabic speakers, with two from each of the 13 countries, each having spent a minimum of ten years in their homeland.
Practical Applications of the Research
Participants created dialogues based on culturally relevant scenarios, facilitating short exchanges expressed in their local dialects. A wide range of topics was covered, from weddings and cuisine to child-rearing, agriculture, arts, and games. Koto indicated that researchers assessed models through three main tasks: selecting culturally appropriate responses among several options, translating both ways between Modern Standard Arabic and specific dialects, and continuing dialogues in the desired dialect.
Notable Findings
Results indicated that the more robust AI models excelled at identifying culturally appropriate responses, achieving nearly 95% success in transitions from Modern Standard Arabic to specific dialects. However, their performance decreased markedly when it came to producing these dialects—whether translating a sentence into Emirati dialect or continuing a dialogue therein.
These findings underline a nuanced challenge in AI development; the issue is not solely one of understanding meaning but also of preserving the cultural identity and specificity of dialects. The study revealed that models performed well with shared cultural practices across the Arab world but struggled with unique customs of individual countries, notably facing difficulties with North African dialects and finding Emirati dialect particularly challenging.
Implications for Future AI Development
The study’s results carry significant implications, especially in the UAE, where AI is a national priority as outlined in the UAE National Artificial Intelligence Strategy 2031. The research contributes to the creation of local AI models, including ‘Jais’, a bilingual large language model that supports both Arabic and English, developed with the assistance of the university.
A Call for Diverse Data
Koto emphasised a broader lesson from this study: merely supporting Arabic is not sufficient if models do not grasp its diverse dialects and the cultural identities they embody. The results open avenues for future research, potentially expanding the dialects examined by the benchmark and enhancing the evaluation of AI capabilities to understand local cultural contexts and produce language authentically.
A Step Forward
In conclusion, this research represents a milestone in the advancement of AI within the Arabic linguistic landscape, moving from general language support to an advanced understanding of linguistic and cultural diversity. The next phase focuses on ensuring AI not only communicates in Arabic but also knows which Arabic to use, the context for its usage, and can express local culture without losing its uniqueness.