fiesta.util.fb_func_stats

Given a Fixed Budget (FB) method like sequential_halving() it will run that function with the given keyword arguments N times and will report the the probability that the FB function chose the best model from all N runs.

fiesta.util.fb_func_stats(N, correct_model_index, fb_func_name, **fb_kwargs)[source]
Parameters

  • N (int) – Number of times to run the Fixed Budget (FB) function.

  • correct_model_index (int) – The index of the best model.

  • fb_func_name (str) – The name of the FB function being evaluated e.g. sequential_halving or non_adaptive_fb

  • fb_kwargs – Keyword arguments to give to the FB function.

Return type

float

Returns

The probability the best model was correctly identified by the FB function across the N runs.

Raises

ValueError – If the fb_func_name is not in the list of acceptable FB function names.