Fixed Confidence (FC)¶

Problem

Given a set of models \(N\) can we find the best model \(N^*\) with a confidence greater than or equal to \((1 - p)\) where \(p\) is our p-value. For example if we want to find the model that is the best model with 95% confidence we would set \(p=0.05\).

Compared to the Fixed Budget approach here we find the best model and know that it is the best model with \((1 - p)\) confidence.

Caveat

All of the Fixed Confidence methods assume that the evaluations scores produced by the models follow a Gaussian (normal) distribution. For a great guide on knowing what distribution your evaluation metric would produce see Dror and Reichart, 2018 guide.

Methods