SEA-SpeechBench A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia

Overview

Audio and multimodal LLMs are measured almost entirely in English. Southeast Asia — over 650 million speakers, many of its languages tonal and many of them low-resource — is absent from existing audio benchmarks, and the regional speech models now appearing have no common yardstick. SEA-SpeechBench is a benchmark for speech understanding across the region.

• 11 languages. The official languages of nine Southeast Asian countries: Burmese, English, Filipino, Indonesian, Khmer, Lao, Malay, Mandarin Chinese, Tamil, Thai, Vietnamese.
• 9 tasks in 3 categories. Speech processing (ASR, ST, SQA), paralinguistics (ER, GR, AgeR, SpkR), and temporal reasoning (TCQ, TLoc).
• 97,194 samples. Across 99 evaluation sets and 597 hours of audio, curated from 23 public and community corpora.
• Bilingual prompting. Every task is evaluated with both English and native-language instructions, matching how people in the region address these systems.
• Temporal reasoning. Two new tasks treat audio as a searchable temporal space, probing both directions of the time↔content mapping over recordings up to 3 minutes.
• 16 systems evaluated. Open models from 2B to 30B parameters, a cascaded baseline evaluated on translation, and the commercial Gemini 2.5 Flash and GPT-4o.

Task suite

Speech processing and paralinguistic tasks use clips of 30 seconds or less, matching the input limits of most current models; the two temporal reasoning tasks extend to 3 minutes. All sources are unified into one format, resampled to 16 kHz mono, and sampled under a fixed seed with speaker-disjoint splits.

Composition

Sample distribution across the benchmark. Low-resource languages such as Khmer, Lao and Burmese are deliberately included to cover the region's full linguistic spectrum. Short clips concentrate around sentence length; the temporal tasks add stratified long-form bins of 30–60s, 60–120s and 120–180s.

Results

All nine tasks are evaluated under both English and native-language prompts. Tables that break results down by language or by audio duration show the English-prompt condition, to keep them readable; the remaining tables report both. Per-dataset results under both conditions are in the paper's appendix.

Key findings

• Temporal reasoning degrades sharply with audio duration. The best temporal localization score achieved by any system falls from 35.74 F1 on 0–30s clips to 5.38 on 120–180s clips. Predicted spans also skew wide: recall exceeds precision at every duration and in nearly every system, though both remain low.
• Prompt language produces systematic disparities. Aggregated across tasks, instructing a model in native language rather than in English costs almost nothing in performance for Chinese and Indonesian, a moderate amount for Filipino, Vietnamese, Malay, Lao and Thai, and a large amount for Burmese, Tamil and Khmer. A text-only control shows that MERaLiON-2-3B translates tested native prompts correctly, yet still scores up to 15.23 points higher under English prompts on age recognition. The deficit is therefore in cross-modal instruction following rather than language understanding.
• Emotion recognition and speech translation remain below usable accuracy. The highest emotion recognition score is 42.55 (Kimi-Audio, native prompts); no other system exceeds 25 under either prompt condition. Speech translation peaks at 21.39 BLEU (GPT-4o) and 53.94 chrF (Gemini 2.5 Flash), with the strongest open-weight result a Whisper + Qwen3.6 cascade at 20.96 BLEU.
• Refusal is a distinct failure mode from error. GPT-4o scores 14.49 macro-F1 on speaker recognition, but the breakdown shows abstentions rather than wrong predictions: it declines 89.5% of queries and is correct on 100% of the 10.5% it answers. Qwen2.5-Omni-7B scores comparably low (15.55 under English prompts, 18.61 under native prompts) and also refuses frequently, but unlike GPT-4o it makes substantive errors on the queries it does accept.

Citation

@inproceedings{liao2026seaspeechbench,
  title     = {{SEA-SpeechBench}: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia},
  author    = {Liao, Jingyi and Zhang, Wenyu and Liu, Zhuohan and He, Yingxu and Lin, Geyu and
               Zou, Xunlong and Sun, Shuo and Alsagoff, Syed Ali Redha and Aw, Ai Ti},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026}
}