fiesta package

Submodules

fiesta.fiesta module

fiesta.fiesta.TTTS(data, model_functions, split_function, p_value, logit_transform=False, samples=100000)[source]
Parameters

  • data (List[Dict[str, Any]]) – A list of dictionaries, that as a whole represents the entire dataset. Each dictionary within the list represents one sample from the dataset.

  • model_functions (List[Callable[[List[Dict[str, Any]], List[Dict[str, Any]]], float]]) – A list of functions that represent different models e.g. pytorch model. Which take a train and test dataset as input and returns a metric score e.g. Accuracy. The model functions should not have random seeds set else it defeats the point of finding the best model independent of the random seed and data split.

  • split_function (Callable[[List[Dict[str, Any]]], Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]]) – A function that can be used to split the data into train and test splits. This should produce random splits each time it is called. If you would like to use a fixed split each time, you can hard code this function to produce a fixed split each time.

  • p_value (float) – The significance value for the best model to be truely the best model e.g. 0.05 if you want to be at least 95% confident.

  • logit_transform (bool) – Whether to transform the model function’s returned metric score by the logit function.

  • samples (int) – Number of samples to generate from our belief distribution for each model. This argument is passed directly to fiesta.util.belief_calc() within this function. This should be large e.g. minimum 10000.

Return type

Tuple[List[float], List[float], int, List[List[float]]]

Returns

Tuple containing 4 values:

  1. The confidence socres for each model, the best model should have the highest confidence

  2. The number of times each model was evaluated as a proportion of the number of evaluations

  3. The total number of model evaluations

  4. The scores that each model generated when evaluated.

NOTE

That if the logit transform is True then the last item in the tuple would be scores that have been transformed by the logit function.

Raises

ValueError – If the p_value is not between 0 and 1.

fiesta.fiesta.logger = <Logger fiesta.fiesta (WARNING)>
fiesta.fiesta.non_adaptive_fb(data, model_functions, split_function, budget, logit_transform=False)[source]
Parameters

  • data (List[Dict[str, Any]]) – A list of dictionaries, that as a whole represents the entire dataset. Each dictionary within the list represents one sample from the dataset.

  • model_functions (List[Callable[[List[Dict[str, Any]], List[Dict[str, Any]]], float]]) – A list of functions that represent different models e.g. pytorch model. Which take a train and test dataset as input and returns a metric score e.g. Accuracy. The model functions should not have random seeds set else it defeats the point of finding the best model independent of the random seed and data split.

  • split_function (Callable[[List[Dict[str, Any]]], Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]]) – A function that can be used to split the data into train and test splits. This should produce random splits each time it is called. If you would like to use a fixed split each time, you can hard code this function to produce a fixed split each time.

  • budget (int) – The total number of evaluations

  • logit_transform (bool) – Whether to transform the model function’s returned metric score by the logit function.

Return type

Tuple[int, List[List[float]]]

Returns

Tuple containing 2 values:

  1. The best performing model function index given the budget

  2. The scores that each model generated when evaluated.

NOTE

That if the logit transform is True then the last item in the tuple would be scores that have been transformed by the logit function.

Raises

ValueError – Given budget \(T\) and the models \(N\) this will be raised if: \(T < |N|\)

fiesta.fiesta.non_adaptive_fc(data, model_functions, split_function, p_value, logit_transform=False, samples=100000)[source]
Parameters

  • data (List[Dict[str, Any]]) – A list of dictionaries, that as a whole represents the entire dataset. Each dictionary within the list represents one sample from the dataset.

  • model_functions (List[Callable[[List[Dict[str, Any]], List[Dict[str, Any]]], float]]) – A list of functions that represent different models e.g. pytorch model. Which take a train and test dataset as input and returns a metric score e.g. Accuracy. The model functions should not have random seeds set else it defeats the point of finding the best model independent of the random seed and data split.

  • split_function (Callable[[List[Dict[str, Any]]], Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]]) – A function that can be used to split the data into train and test splits. This should produce random splits each time it is called. If you would like to use a fixed split each time, you can hard code this function to produce a fixed split each time.

  • p_value (float) – The significance value for the best model to be truely the best model e.g. 0.05 if you want to be at least 95% confident.

  • logit_transform (bool) – Whether to transform the model function’s returned metric score by the logit function.

  • samples (int) – Number of samples to generate from our belief distribution for each model. This argument is passed directly to fiesta.util.belief_calc() within this function. This should be large e.g. minimum 10000.

Return type

Tuple[List[float], List[float], int, List[List[float]]]

Returns

Tuple containing 4 values:

  1. The confidence socres for each model, the best model should have the highest confidence

  2. The number of times each model was evaluated as a proportion of the number of evaluations

  3. The total number of model evaluations

  4. The scores that each model generated when evaluated.

NOTE

That if the logit transform is True then the last item in the tuple would be scores that have been transformed by the logit function.

Raises

ValueError – If the p_value is not between 0 and 1.

fiesta.fiesta.sequential_halving(data, model_functions, split_function, budget, logit_transform=False)[source]
Parameters

  • data (List[Dict[str, Any]]) – A list of dictionaries, that as a whole represents the entire dataset. Each dictionary within the list represents one sample from the dataset.

  • model_functions (List[Callable[[List[Dict[str, Any]], List[Dict[str, Any]]], float]]) – A list of functions that represent different models e.g. pytorch model. Which take a train and test dataset as input and returns a metric score e.g. Accuracy. The model functions should not have random seeds set else it defeats the point of finding the best model independent of the random seed and data split.

  • split_function (Callable[[List[Dict[str, Any]]], Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]]) – A function that can be used to split the data into train and test splits. This should produce random splits each time it is called. If you would like to use a fixed split each time, you can hard code this function to produce a fixed split each time.

  • budget (int) – The total number of evaluations

  • logit_transform (bool) – Whether to transform the model function’s returned metric score by the logit function.

Return type

Tuple[int, List[float], List[List[float]]]

Returns

Tuple containing 3 values:

  1. The best performing model function index given the budget

  2. The number of times each model was evaluated as a proportion of the total number of evaluations

  3. The scores that each model generated when evaluated.

NOTE

That if the logit transform is True then the last item in the tuple would be scores that have been transformed by the logit function.

Raises

ValueError – Given budget \(T\) and the models \(N\) this will be raised if: \(T < (|N| * \log_2|N|)\)

fiesta.util module

fiesta.util.belief_calc(est_means, est_variances, eval_counts, samples=100000)[source]
Parameters

  • est_means (ndarray) – A vector of each models mean scores shape: (Number of models,)

  • est_means – A vector of each models variance of there scores: shape: (Number of models,)

  • eval_counts (ndarray) – A vector stating the number of times each model has been evaluated/ran: shape: (Number of models,)

  • samples (int) – Number of samples to generate from our belief distribution for each model. This should be large e.g. minimum 10000

Return type

List[float]

Returns

The number of times a model performed best for each sample based on each model belief distribution, normliased by the number of samples (this is in affect the confidence for each model of whether that model is the best model, p-value can be calculated by 1 - the confidence value for each model). shape (Number of models,)

Raises

ValueError – If the eval_counts contains values less than 3. As it is required that each model has been evaluated a minimum of 3 times, this is due to our prior beleif in our algorthim.

fiesta.util.fb_func_stats(N, correct_model_index, fb_func_name, **fb_kwargs)[source]
Parameters

  • N (int) – Number of times to run the Fixed Budget (FB) function.

  • correct_model_index (int) – The index of the best model.

  • fb_func_name (str) – The name of the FB function being evaluated e.g. sequential_halving or non_adaptive_fb

  • fb_kwargs – Keyword arguments to give to the FB function.

Return type

float

Returns

The probability the best model was correctly identified by the FB function across the N runs.

Raises

ValueError – If the fb_func_name is not in the list of acceptable FB function names.

fiesta.util.fc_func_stats(N, correct_model_index, fc_func_name, **fc_kwargs)[source]
Parameters

  • N (int) – Number of times to run the Fixed Confidence (FC) function

  • correct_model_index (int) – The index of the best model

  • model_funcs – Functions that will generated model evaluation scores

  • fc_func_name (str) – The name of the FC function being evaluated e.g. non_adaptive_fc or TTTS

  • fc_kwargs – Keyword arguments to give to the FC function.

Return type

Tuple[int, int, int, float]

Returns

Tuple containing:

  1. min

  2. mean

  3. max

  4. percentage correct

Where the first 3 relate to the number of evaluations the FC function required to get to the confidence level required. The last is the percentage of those runs where the most confident model was the correct model determined by the correct_model_index argument.

Raises

ValueError – If the fc_func_name is not in the list of acceptable FC function names.

fiesta.util.lists_same_size(*lists)[source]

Checks if the lists given as arguments are of the same size, if not it raises a ValueError.

Raises

ValueError – If any of the lists are not the same size

Return type

None

fiesta.util.pull_arm(mean, sd)[source]
Parameters

  • mean (float) – The mean of the normal distribution.

  • sd (float) – The standard deviation of the normal distribution.

Return type

float

Returns

A value that has been sampled from a normal distribution that has a mean and standard deviation of those given as arguments.

Module contents