feat: memoization at the set level during an operation when the number of sets can be huge

[danielelotito]

I’m working on benchmarking different sampling strategies and I could use your input on how to set this up efficiently with an ado custom experiment.

Here’s the situation:
• For each combination of points in the training set, I need to train a separate model, but the order of points doesn’t matter.

• I want to make sure we don’t train the same model twice on the same set of points.

• My idea is to use the orchestrator, so it feels natural that the fundamental unit should be a unique identifier for each set of points. But setting up a space in this way sounds impractical

• Each set should then be linked to a set of performance metrics (measured properties).

Basically, I would like to:

1. Generate the identifier for training sets
2. Associate each identifier with its performance metric.
3. Avoid duplicate training runs.

A solution that I would like:

• Generate the identifier for training set at runtime (In a consistent way), and use this identifier in such a way that if I am about to run the training related to this identifier again, I use the memoized result.

Is this already possible within ado? What would be the recommended practice?

Additional detail
The size of the benchmarking sampling strategy must be suited for the caching approach. I.e. sampling O(10) on a total of O(10) and with constraints on the sampling.

[danielelotito]

The procedure will be

• Canonicalize the Training Set: Sort the IDs of the points in your training set to ensure the order doesn't matter.
• Hash the Set
• Use the Hash as a Memoization Key: Before running the expensive training, check your cache for this hash. If it's a miss, run the job, and then store the results with the hash as the key.

I need to address the combinatorial expansion, for a POC consider the following

from math import lgamma, log, exp

# These variables are used for the heatmap generation
n_features = 10
n_bits_per_feature = 16
hash_bool = True
hash_size = 32  # using a hash of 32 bit
reduction_factor = 0.2  # possibilities * reduction factor gives you the number of configs that deserves a trial


def compute_log_size_in_bit(
    dataset_size,
    training_size,
    n_features,
    n_bits_per_feature,
    hash_bool,
    hash_size,
    reduction_factor,
):
    """
    Computes the natural logarithm of the size in bits of the configuration space
    to handle very large numbers in a stable way.
    Returns the log of the total bits, or -1 if inputs are invalid.
    """
    if training_size >= dataset_size:
        return -1

    log_comb = (
        lgamma(dataset_size + 1)
        - lgamma(training_size + 1)
        - lgamma(dataset_size - training_size + 1)
    )

    log_n_possibilities = log_comb + log(reduction_factor)

    if hash_bool:
        log_n_bit_total = log_n_possibilities + log(hash_size)
    else:
        log_n_bit_total = (
            log_n_possibilities + log(n_features) + log(n_bits_per_feature)
        )

    return log_n_bit_total