- In “BALSAM: A Platform for Benchmarking Arabic Large Language Models” (ACL Anthology), Rawan Al-Matham (King Salman Global Academy For Arabic Language) and others “introduce BALSAM, a comprehensive, community-driven benchmark aimed at advancing Arabic LLM development and evaluation. It includes 78 NLP tasks from 14 broad categories, with 52K examples divided into 37K test and 15K development, and a centralized, transparent platform for blind evaluation. [They} envision BALSAM as a unifying platform that sets standards and promotes collaborative research to advance Arabic LLM capabilities.”
- In “Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps” (arXiv), Ahmed Alzubaidi (Technology Innovation Institute, Abu Dhabi) and others “the first systematic review of Arabic LLM benchmarks, analyzing 40+ evaluation benchmarks across NLP tasks, knowledge domains, cultural understanding, and specialized capabilities. We propose a taxonomy organizing benchmarks….[Their] analysis reveals significant progress in benchmark diversity while identifying critical gaps: limited temporal evaluation, insufficient multi-turn dialogue assessment, and cultural misalignment in translated datasets. We examine three primary approaches: native collection, translation, and synthetic generation discussing their trade-offs regarding authenticity, scale, and cost.”