{"id":"seaexam-seabench","title":"SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia","authors":[{"given":"Chaoqun","family":"Liu"},{"given":"Wenxuan","family":"Zhang"},{"given":"Jiahao","family":"Ying"},{"given":"Mahani","family":"Aljunied"},{"given":"Anh Tuan","family":"Luu"},{"given":"Lidong","family":"Bing"}],"first_public":"2025-02-10","date_note":"First arXiv version.","date_source":"https://arxiv.org/abs/2502.06298","year":2025,"venue":"NAACL 2025 Findings","paper_url":"https://aclanthology.org/2025.findings-naacl.341/","doi":"10.18653/v1/2025.findings-naacl.341","arxiv":"2502.06298","relationship":"uses","bibtex_key":"seaexamseabench2025","record_url":"https://yuntiandeng.com/wildchat/papers/seaexam-seabench/","citation_title":"SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia","citation_type":"paper-conference","bibtex_type":"inproceedings","container_title":"Findings of the Association for Computational Linguistics: NAACL 2025","citation_url":"https://aclanthology.org/2025.findings-naacl.341/","citation_year":2025,"citation_note":"NAACL 2025 Findings proceedings paper.","pages":"6134–6151","publisher":"Association for Computational Linguistics","code_url":"https://github.com/DAMO-NLP-SG/SeaBench","category":"evaluation","relationship_label":"Uses WildChat","dataset_version":"Wild Queries combines filtered WildChat-1M and LMSYS-Chat-1M inputs in Indonesian, Thai, and Vietnamese; the paper reports 4,658 queries across the two sources.","summary":"Builds Southeast Asian evaluation sets from local sources and uses real conversations to test how closely they reflect everyday requests. WildChat and LMSYS-Chat-1M supply the Wild Queries comparison corpus after language checks, length limits, and filtering.","finding":"In embedding-based comparisons, the locally constructed SeaExam and SeaBench are closer to the observed query distribution than translated MMLU and MT-Bench.","evidence_url":"https://aclanthology.org/2025.findings-naacl.341.pdf","evidence_note":"Section 3.1: Wild Queries construction and distribution comparisons. Appendix B and Table 10: language breakdown."}