Ilja Kuzborskij

Research Scientist, Google DeepMind, UK

Quantifying and Estimating Epistemic Uncertainty in LLMs through Iterative Prompting

Abstract

This talk explores the challenge of understanding and quantifying uncertainty in large language models (LLMs), with a focus on identifying when their outputs are unreliable. We will examine the two key types of uncertainty at play: epistemic uncertainty, which arises from a lack of knowledge (e.g., about facts or language), and aleatoric uncertainty, which reflects inherent randomness (e.g., multiple valid answers).

In this talk, we propose an information-theoretic metric that quantifies epistemic uncertainty in any autoregressive prediction, such as LLM generation (the gap between the LLM’s outputs and the ‘ground truth’) and introduce a practical method to estimate it by leveraging iterative prompting based on the model’s own outputs. We will demonstrate that this metric is effective in detecting high epistemic uncertainty and pinpointing outputs where the model is likely to produce unreliable or “hallucinated” responses.

(joint work with Yasin Abbasi-Yadkori, András György, and Csaba Szepesvári)

---

About Ilja Kuzborskij

Ilja Kuzborskij is a research scientist at the Foundations team at Google DeepMind in London. He obtained his Ph.D. from the Swiss Federal Institute of Technology Lausanne (EPFL) in 2018 and did postdoctoral research at the University of Milan with Nicolò Cesa-Bianchi. His work mainly focuses on problem-dependent theories of generalization, uncertainty estimation, and related topics such as concentration inequalities.