Documentation

Welcome to the multiscorer wiki!

How To Use

This is the complete API documentation. For examples of use see Examples Wiki.

multiscorer.MultiScorer class:

Use this class to encapsulate and/or aggregate multiple scoring functions so that it can be passed as an argument for scoring in scikit's cross_val_score function.
Instances of this class are also callables, with signature as needed by cross_val_score.

Methods:

• MultiScorer(metrics)

  Get a new instance of MultiScorer.

  • Parameters

    • metrics: dict
      The metrics to be used by the scorer.
      The dictionary must have as key a name (str) for the metric and as value a tuple containing the metric function itself and a dict literal of the additional named arguments to be passed to the function.
      The metric function should be one of the sklearn.metrics function or any other callable with the same signature: metric(y_real, y, **kwargs).

• __call__(estimator, X, y)

  To be called by for evaluation from sklearn's GridSearchCV or cross_val_score.

  • Parameters

    • ...as defined in the respective documentation.

  • Returns

    • dummy : int (0.5)
      A dummy value of 0.5 just for compatibility reasons.

• get_metric_names()

  Get all the metric names as given when initialized.

  • Parameters: None

  • Returns:

    • metric_names : list
      A list containing the given names of the metrics.

• get_results(metric=None, fold='all')

  Get the results of a specific or all the metrics. This method should be called after the object itself has been called so that the metrics are applied.

  • Parameters

    • metric: str or None (default)
      The given name of a metric to return its result(s). If omitted the results of all metrics will be returned.
    • fold : int in range [1, number_of_folds] or 'all' (Default)
      Get the metric(s) results for the specific fold.
      The number of folds corresponds to the number of times the instance is called.
      If its value is a number, either the score of a single metric for that fold or a dictionary of the (single) scores for that fold will be returned, depending on the value of metric parameter.
      If its value is 'all', either a list of a single metric or a dictionary containing the lists of scores for all folds will be returned, depending on the value of metric parameter.

  • Returns

    • metric_result_for_one_fold
      The result of the designated metric function for the specific fold, if metric parameter was not omitted and an integer value was given to fold parameter.
      If the value of metric does not correspond to a metric name, None will be returned.
    • all_metric_results_for_one_fold: dict
      A dict having as keys the names of the metrics and as values their results for the specific fold.
      This will be returned only if metric parameter was omitted and an integer value was given to fold parameter.
    • metric_results_for_all_folds: list
      A list of length number_of_folds containing the results of all folds for the specific metric, if metric parameter was not omitted and value 'all' was given to fold.
      If the value of metric does not correspond to a metric name, None will be returned.
    • all_metric_results_for_all_folds: dict with lists as values
      A dict having as keys the names of the metrics and as values lists (of length number_of_folds) of their results for all folds.
      This will be returned only if metric parameter was omitted and 'all' value was given to fold parameter.

  • Raises

    • UserWarning
      If this method is called before the instance is called for evaluation.
    • ValueError
      If the value for fold parameter is not appropriate.

Note: number_of_folds is not an actual variable / constant or parameter, it just indicates how many times the instance is called (possibly by scikit's cross_val_score), thus creating lists of that length.