fiesta.non_adaptive_fb

This is the standard approach where the budget \(T\) is spent fairly across all of the \(N\) models. The best model \(N^*\) would be the model that has the highest mean score across all of the evaluations.

Example

Given \(T=10\) and \(N=5\) we would give each of the \(N\) models \(2\) evaluations each.

fiesta.fiesta.non_adaptive_fb(data, model_functions, split_function, budget, logit_transform=False)[source]
Parameters

  • data (List[Dict[str, Any]]) – A list of dictionaries, that as a whole represents the entire dataset. Each dictionary within the list represents one sample from the dataset.

  • model_functions (List[Callable[[List[Dict[str, Any]], List[Dict[str, Any]]], float]]) – A list of functions that represent different models e.g. pytorch model. Which take a train and test dataset as input and returns a metric score e.g. Accuracy. The model functions should not have random seeds set else it defeats the point of finding the best model independent of the random seed and data split.

  • split_function (Callable[[List[Dict[str, Any]]], Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]]) – A function that can be used to split the data into train and test splits. This should produce random splits each time it is called. If you would like to use a fixed split each time, you can hard code this function to produce a fixed split each time.

  • budget (int) – The total number of evaluations

  • logit_transform (bool) – Whether to transform the model function’s returned metric score by the logit function.

Return type

Tuple[int, List[List[float]]]

Returns

Tuple containing 2 values:

  1. The best performing model function index given the budget

  2. The scores that each model generated when evaluated.

NOTE

That if the logit transform is True then the last item in the tuple would be scores that have been transformed by the logit function.

Raises

ValueError – Given budget \(T\) and the models \(N\) this will be raised if: \(T < |N|\)