fiesta.util.fc_func_stats

Given a Fixed Confidence (FC) method like TTTS() it will run that function with the given keyword arguments N times and will report the summary statistics: min, mean, max evaluations and percentage of times the FC function correctly found the best model based on the correct_model_index argument.

fiesta.util.fc_func_stats(N, correct_model_index, fc_func_name, **fc_kwargs)[source]
Parameters

  • N (int) – Number of times to run the Fixed Confidence (FC) function

  • correct_model_index (int) – The index of the best model

  • model_funcs – Functions that will generated model evaluation scores

  • fc_func_name (str) – The name of the FC function being evaluated e.g. non_adaptive_fc or TTTS

  • fc_kwargs – Keyword arguments to give to the FC function.

Return type

Tuple[int, int, int, float]

Returns

Tuple containing:

  1. min

  2. mean

  3. max

  4. percentage correct

Where the first 3 relate to the number of evaluations the FC function required to get to the confidence level required. The last is the percentage of those runs where the most confident model was the correct model determined by the correct_model_index argument.

Raises

ValueError – If the fc_func_name is not in the list of acceptable FC function names.