Beyond Accuracy in Low-Resource Llm Evaluation: A Structured Critical Review and Reliability-Centered Framework
DOI:
https://doi.org/10.61166/interkoneksi.v4i2.96Keywords:
large language models, low-resource languages, multilingual evaluation, benchmark validity, cultural validityAbstract
Large language models (LLMs) are commonly compared through aggregate accuracy on benchmarks translated from, or culturally anchored to, high-resource languages. In low-resource settings, the resulting score can conflate model capability with translation artifacts, cultural mismatch, dialect exclusion, tokenization inefficiency, contamination, metric failure, and unreliable automated judging. This structured critical review synthesizes 27 peer-reviewed works published from 2018 to 2026 and develops the LRL-Eval Card, a reliability-centered reporting framework. Multi-label coding found that metric validity was the most frequent theme (20/27), whereas benchmark origin and tokenization appeared in only three papers each. The evidence shows that culturally sensitive subsets can change model rankings, answer-option order can alter performance substantially, multilingual LLM judges have weak cross-language agreement, contamination can inflate scores, and tokenization affects both accuracy and access cost. The LRL-Eval Card records ten non-compensatory evidence dimensions rather than producing another composite score. Reliable low-resource evaluation therefore requires triangulation across native-authored and translated items, language varieties, native-speaker judgments, contamination controls, tokenizer diagnostics, validated metrics, and uncertainty estimates.
Downloads
References
Ahia, O., Kumar, S., Gonen, H., Kasai, J., Mortensen, D., Smith, N., & Tsvetkov, Y. (2023). Do all languages cost the same? Tokenization in the era of commercial language models. In H. Bouamor, J. Pino, & K. Bali (Eds.), Proceedings of the 2023 conference on empirical methods in natural language processing (pp. 9904–9923). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.614
Ahuja, K., Diddee, H., Hada, R., Ochieng, M., Ramesh, K., Jain, P., Nambi, A., Ganu, T., Segal, S., Ahmed, M., Bali, K., & Sitaram, S. (2023). MEGA: Multilingual evaluation of generative AI. In H. Bouamor, J. Pino, & K. Bali (Eds.), Proceedings of the 2023 conference on empirical methods in natural language processing (pp. 4232–4267). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.258
Arnob, N. M. K., & Mahi, A. B. S. (2026). One language, three of its voices: Evaluating multilingual LLMs across Persian, Dari, and Tajiki on translation and understanding tasks. In R. Merchant & K. Megerdoomian (Eds.), The proceedings of the first workshop on NLP and LLMs for the Iranian language family (pp. 98–104). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.silkroadnlp-1.10
Azime, I. A., Tonja, A. L., Belay, T. D., Chanie, Y., Balcha, B. F., Abadi, N. H., Ademtew, H. B., Nerea, M. A., Yadeta, D. D., Geremew, D. D., Tesfu, A. A., Slusallek, P., Solorio, T., & Klakow, D. (2025). ProverbEval: Exploring LLM evaluation challenges for low-resource language understanding. In L. Chiruzzo, A. Ritter, & L. Wang (Eds.), Findings of the association for computational linguistics: NAACL 2025 (pp. 6265–6281). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.findings-naacl.350
Bandarkar, L., Liang, D., Muller, B., Artetxe, M., Shukla, S. N., Husa, D., Goyal, N., Krishnan, A., Zettlemoyer, L., & Khabsa, M. (2024). The belebele benchmark: A parallel reading comprehension dataset in 122 language variants. In L.-W. Ku, A. Martins, & V. Srikumar (Eds.), Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 749–775). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.acl-long.44
Bestgen, Y. (2022). Please, don’t forget the difference and the confidence interval when seeking for the state-of-the-art status. In N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, J. Odijk, & S. Piperidis (Eds.), Proceedings of the thirteenth language resources and evaluation conference (pp. 5956–5962). European Language Resources Association. https://aclanthology.org/2022.lrec-1.640/
Bokaei, Z., Webber, B., & Magdy, W. (2026). Benchmarking offensive language detection in Persian and Pashto. In R. Merchant & K. Megerdoomian (Eds.), The proceedings of the first workshop on NLP and LLMs for the Iranian language family (pp. 13–23). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.silkroadnlp-1.2
Faisal, F., & Anastasopoulos, A. (2025). Testing the boundaries of LLMs: Dialectal and language-variety tasks. In Y. Scherrer, T. Jauhiainen, N. Ljubešić, P. Nakov, J. Tiedemann, & M. Zampieri (Eds.), Proceedings of the 12th workshop on NLP for similar languages, varieties and dialects (pp. 68–92). Association for Computational Linguistics. https://aclanthology.org/2025.vardial-1.6/
Fu, X., & Liu, W. (2025). How reliable is multilingual LLM-as-a-judge? In C. Christodoulopoulos, T. Chakraborty, C. Rose, & V. Peng (Eds.), Findings of the association for computational linguistics: EMNLP 2025 (pp. 11040–11053). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.findings-emnlp.587
Gevers, I., De Marez, V., Van Nooten, J., Lemmens, J., Kosar, A., Lotfi, E., Banar, N., Fivez, P., De Bruyne, L., & Daelemans, W. (2025). In benchmarks we trust ... Or not? In C. Christodoulopoulos, T. Chakraborty, C. Rose, & V. Peng (Eds.), Proceedings of the 2025 conference on empirical methods in natural language processing (pp. 23662–23676). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.emnlp-main.1208
Goyal, N., Gao, C., Chaudhary, V., Chen, P.-J., Wenzek, G., Ju, D., Krishnan, S., Ranzato, M., Guzmán, F., & Fan, A. (2022). The flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10, 522–538. https://doi.org/10.1162/tacl_a_00474
Han, W., Zhang, Y., Chen, Z., Binbinliu, Pechenizkiy, M., Fang, M., & Zheng, Y. (2026). MuBench: Assessment of multilingual capabilities of large language models across 61 languages. In M. Liakata, V. P. Moreira, J. Zhang, & D. Jurgens (Eds.), Findings of the Association for Computational Linguistics: ACL 2026 (pp. 16163–16192). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.findings-acl.794
Hasan, Md. A., Hasanain, M., Ahmad, F., Laskar, S. R., Upadhyay, S., Sukhadia, V. N., Kutlu, M., Chowdhury, S. A., & Alam, F. (2025). NativQA: Multilingual culturally-aligned natural query for LLMs. In W. Che, J. Nabende, E. Shutova, & M. T. Pilehvar (Eds.), Findings of the association for computational linguistics: ACL 2025 (pp. 14886–14909). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.findings-acl.770
Joshi, P., Santy, S., Budhiraja, A., Bali, K., & Choudhury, M. (2020). The state and fate of linguistic diversity and inclusion in the NLP world. In D. Jurafsky, J. Chai, N. Schluter, & J. Tetreault (Eds.), Proceedings of the 58th annual meeting of the association for computational linguistics (pp. 6282–6293). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.560
Khashabi, D., Cohan, A., Shakeri, S., Hosseini, P., Pezeshkpour, P., Alikhani, M., Aminnaseri, M., Bitaab, M., Brahman, F., Ghazarian, S., Gheini, M., Kabiri, A., Mahabagdi, R. K., Memarrast, O., Mosallanezhad, A., Noury, E., Raji, S., Rasooli, M. S., Sadeghi, S., … Yaghoobzadeh, Y. (2021). ParsiNLU: A suite of language understanding challenges for persian. Transactions of the Association for Computational Linguistics, 9, 1147–1162. https://doi.org/10.1162/tacl_a_00419
Laskar, M. T. R., Alqahtani, S., Bari, M. S., Rahman, M., Khan, M. A. M., Khan, H., Jahan, I., Bhuiyan, A., Tan, C. W., Parvez, M. R., Hoque, E., Joty, S., & Huang, J. X. (2024). A systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations. In Y. Al-Onaizan, M. Bansal, & Y.-N. Chen (Eds.), Proceedings of the 2024 conference on empirical methods in natural language processing (pp. 13785–13816). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.764
Li, Y., Guo, Y., Guerin, F., & Lin, C. (2024). An open-source data contamination report for large language models. In Y. Al-Onaizan, M. Bansal, & Y.-N. Chen (Eds.), Findings of the association for computational linguistics: EMNLP 2024 (pp. 528–541). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-emnlp.30
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C., Manning, C. D., Ré, C., Acosta-Navas, D., Hudson, D. A., … Koreeda, Y. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research. https://openreview.net/forum?id=iO4LZibEqW
Lundin, J. M., Zhang, A., Karim, N., Louzan, H., Wei, G., Adelani, D. I., & Carroll, C. (2026). The token tax: Systematic bias in multilingual tokenization. In E. A. Chimoto, C. Lignos, S. Muhammad, I. Abdulmumin, C. Siro, & D. I. Adelani (Eds.), Proceedings of the 7th workshop on African natural language processing (AfricaNLP 2026) (pp. 103–112). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.africanlp-main.10
Panahi, S., Kelleher, J., & Nedumpozhimana, V. (2026). When LLMs annotate: Reliability challenges in low-resource NLI. In H. Hettiarachchi, T. Ranasinghe, A. Plum, P. Rayson, R. Mitkov, M. Gaber, D. Premasiri, F. A. Tan, & L. Uyangodage (Eds.), Proceedings of the second workshop on language models for low-resource languages (LoResLM 2026) (pp. 178–188). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.loreslm-1.17
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020). Beyond accuracy: Behavioral testing of NLP models with CheckList. In D. Jurafsky, J. Chai, N. Schluter, & J. Tetreault (Eds.), Proceedings of the 58th annual meeting of the association for computational linguistics (pp. 4902–4912). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.442
Singh, S., Romanou, A., Fourrier, C., Adelani, D. I., Ngui, J. G., Vila-Suero, D., Limkonchotiwat, P., Marchisio, K., Leong, W. Q., Susanto, Y., Ng, R., Longpre, S., Ruder, S., Ko, W.-Y., Bosselut, A., Oh, A., Martins, A., Choshen, L., Ippolito, D., … Hooker, S. (2025). Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evaluation. In W. Che, J. Nabende, E. Shutova, & M. T. Pilehvar (Eds.), Proceedings of the 63rd annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 18761–18799). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.acl-long.919
Tricco, A. C., Lillie, E., Zarin, W., O’Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., Hempel, S., Akl, E. A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M. G., Garritty, C., … Straus, S. E. (2018). PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Annals of Internal Medicine, 169(7), 467–473. https://doi.org/10.7326/m18-0850
Van Doren, M., Ford, C., Barajas, J., VanMeter, R., & Holland, C. (2026). “Be my cheese?”: Cultural nuance benchmarking for machine translation in multilingual LLMs. In S. Mille, S. Gehrmann, P. Schmidtová, O. Dušek, M. Fadaee, K. Lo, E. Santus, & G. Stanovsky (Eds.), Proceedings of the fifth workshop on generation, evaluation and metrics (GEM) (pp. 52–76). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.gem-main.6
Variš, D., Stephen, A., & Libovický, J. (2026). TokCollate: A comprehensive tool for tokenizer evaluation and visualization across languages. In G. Durrett & P. Jian (Eds.), Proceedings of the 64th annual meeting of the Association for Computational Linguistics (volume 3: System demonstrations) (pp. 418–427). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.acl-demo.41
Yao, F., Zhuang, Y., Sun, Z., Xu, S., Kumar, A., & Shang, J. (2024). Data contamination can cross language barriers. In Y. Al-Onaizan, M. Bansal, & Y.-N. Chen (Eds.), Proceedings of the 2024 conference on empirical methods in natural language processing (pp. 17864–17875). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.990
Zaghouani, W. (2026). Lost in dialect: The annotation gap in multilingual LLM safety. In K. Huang, F. Mo, P. Chen, & M. Jiang (Eds.), Proceedings of the 1st workshop on multilinguality in the era of large language models (MeLLM 2026) (pp. 1–13). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.mellm-1.1
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Noorullah Zamindar
This work is licensed under a Creative Commons Attribution 4.0 International License.
