GPT · California · Apple Machine Learning
DeepAmbigQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness
Compiled by KHAO Editorial — aggregated from 1 source + 1 reference discovered via search. See llms.txt for citation guidance.
★ Tier-1 Source
DeepAmbigQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness.
Key facts
- Experiments reveal that, even state-of-the-art GPT-5 show incomplete answers, achieving only 0.13 exact match on ambiguous questions and 0.21 on non-ambiguous questions
- Authors Jiabao Ji†**, Min Li, Priyanshu Kumar, Shiyu Chang†, Saloni Potdar
- Based on this, they build DEEPAMBIGQA, a dataset of 3,600 questions requiring multi-hop reasoning and half of them explicit name ambiguity resolving
- DeepAmbigQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness
Summary
Authors Jiabao Ji†**, Min Li, Priyanshu Kumar, Shiyu Chang†, Saloni Potdar. Large language models (LLMs) with integrated search tools show strong promise in open-domain question answering (QA), yet they often struggle to produce complete answer set to complex questions such as “Which actor from the film Heat won at least one Academy Award?”, which requires (1) distinguishing between multiple films sharing the same title and (2) reasoning across a large set of actors to gather and integrate evidence. Experiments reveal that, even state-of-the-art GPT-5 show incomplete answers, achieving only 0.13 exact match on ambiguous questions and 0.21 on non-ambiguous questions. † University of California, Santa Barbara.