Introduction and Motivation
Question answering has advanced rapidly with large language models, but predominantly for high-resource languages, in both text and spoken settings. Spoken question answering (SQA) benchmark for Telugu remains unexplored, and the reliability of automatic evaluation in this setting remains unquantified.
Benchmark Dataset: VākQA
We introduce VākQA, a Telugu SQA benchmark of 2,001 factoid question-answer pairs across six domains, with 2.53 hours of speech audio, bilingual transcriptions, and human-verified reference answers.
Evaluation Methodology
We first validate evaluation methods against human judgements:
- Gemini-as-a-judge best approximates human ratings but is non-uniformly strict.
- Open-weight judges systematically penalize correct Telugu answers that differ in surface form from the reference.
Experimental Findings
Using this validated setup, we benchmark proprietary and open-weight models across input modality, language, and domain. We observe that:
- Telugu phrasing retains cultural specificity that is lost in translation.
- Speech input introduces phonetic confusions that alter question meaning.
- Cascaded ASR-MT errors compound progressively.
Release
VākQA is publicly released.