fiesta.util.belief_calc¶
-
fiesta.util.belief_calc(est_means, est_variances, eval_counts, samples=100000)[source] - Parameters
est_means (
ndarray) – A vector of each models mean scores shape: (Number of models,)est_means – A vector of each models variance of there scores: shape: (Number of models,)
eval_counts (
ndarray) – A vector stating the number of times each model has been evaluated/ran: shape: (Number of models,)samples (
int) – Number of samples to generate from our belief distribution for each model. This should be large e.g. minimum 10000
- Return type
- Returns
The number of times a model performed best for each sample based on each model belief distribution, normliased by the number of samples (this is in affect the confidence for each model of whether that model is the best model, p-value can be calculated by 1 - the confidence value for each model). shape (Number of models,)
- Raises
ValueError – If the
eval_countscontains values less than 3. As it is required that each model has been evaluated a minimum of 3 times, this is due to our prior beleif in our algorthim.