Statistics Seminar by John Cherian

John Cherian

(Stanford University)

Conformal Prediction with Conditional Guarantees

Abstract :
We consider the problem of constructing distribution-free prediction sets with finite-sample conditional guarantees. Prior work has shown that it is impossible to provide exact conditional coverage universally in finite samples. Thus, most popular methods only provide marginal coverage over the covariates. We bridge this gap by defining a spectrum of problems that interpolate between marginal and conditional validity. After the reformulation of conditional coverage as coverage over a class of covariate shifts, we show how to simultaneously obtain exact finite sample coverage over all possible shifts when the target class of shifts is finite dimensional. For example, our algorithm outputs intervals with exact coverage over each group from a collection of protected groups. For more flexible, infinite dimensional classes where exact coverage is impossible, we provide a simple procedure for quantifying the gap between the coverage of our algorithm and the target level. Moreover, by tuning a single hyperparameter, we allow the practitioner to control the size of this gap across shifts of interest. Our methods can be easily incorporated into existing split conformal inference pipelines, and thus can be used to quantify the uncertainty of modern black-box algorithms without distributional assumptions. This is joint work with Isaac Gibbs (Stanford) and Emmanuel Candes (Stanford).