Generalization for Diffusion Models: An Algorithmic-Dependent Framework Based on Stability

Speaker: Patrick Rebeschini (Professor of Statistics and Machine Learning) Date: 28-01-2025, 2pm-3pm (BST) Location: Department of Computer Science, CS1.04, University of Warwick, Coventry, UK

Patrick Rebeschini

Abstract

The success of diffusion models highlights the need for theoretical guarantees to explain their generalization capabilities, particularly given the high-dimensional nature of the training data and the sampling procedures they employ. Prior efforts to account for the empirical aspects of score-matching mechanisms either rely on uniform learning strategies, which produce algorithm-agnostic error bounds and often suffer from exponential dependence to problem dimensionality, or employ methods tailored to specific and unrealistic settings.

In this work, we depart from uniform learning approaches and propose a general theory of algorithm-dependent generalization error bounds, leveraging the framework of algorithmic stability from supervised learning. We introduce notions of score stability, which quantify the sensitivity of a score-matching algorithm to changes in the dataset. Using this setting, we derive generalization error bounds for the score-matching loss, as well as for the sampling procedure, with bounds in terms of log-likelihood, KL divergence, Fisher inception distance, and maximum-mean discrepancy.

(Joint work with Tyler Farghly, George Deligiannidis, and Arnaud Doucet)

About Patrick Rebeschini

Patrick Rebeschini serves as a Professor of Statistics and Machine Learning in the Department of Statistics at the University of Oxford. He earned his Ph.D. in Operations Research and Financial Engineering from Princeton University. Before joining Oxford in 2017, he was a postdoctoral researcher in the Departments of Electrical Engineering and Computer Science at Yale University. His research focuses on uncovering and leveraging fundamental principles in high-dimensional probability, statistics, and optimization to develop computationally efficient and statistically optimal algorithms for machine learning and artificial intelligence.