[question] Estimation of the result on test sample

[945fc41467]

Enter the chapter number

3.3

Enter the page number

184 to 202

What is the cell's number in the notebook

No response

Enter the environment you are using to run the notebook

None

Question

Hello

Second question :

It seems there is two schools in statistics about validating a model. The kind of “traditional” approach I was taught at school : making hypothesis about data (most often normality), deduce the law of the predictor, and deduce the 9x% confidence interval. Often, this approach not really convince me, as the hypothesis about normality is often made without serious justification, but because this is the only way to compute the confidence interval.

The "machine learning" approach, described in the book is, at the opposite, completely empirical : you said « my model work, because I checked it work on a test data ». So you don’t need any discussable hypothesis on the distribution of the data. At first glance it looked rather more rigorous than the first one. But actually, when you are doing this, you are estimating the unknown performance score your model would give on you entire univers (in the statistical sens) by computing a the score on you test sample. So, you are still doing a basic statistical inference. And who say “statistical inference” say “confidence interval” : you should be sure that you test sample is representative of your univers.

That means, when you compute a roc_auc or a precision/recall curve, theses curves should have a confidence interval around them. When you say “with this threshold, the precision is p%" you should give a CI around p%.

How do you deal with that ? Is there a way to compute this CI ?

[ankitgpt18]

can i do this?

[ageron]

That's such a great question! 👍

I love the comparison between the classical analysis and ML's empirical approach. I studied biology before computer science, and this led me to have more trust in empirical observations than theoretical models (when it comes to complex systems at least), as the theoretical models often make simplifying assumptions that don't actually hold and give you a false sense of rigor.

That said, you are totally right: when you compute a metric like accuracy, AUC, precision, recall, F1, etc., on a test set, you’re really estimating the expected performance of your model over the true data distribution D, so there should be error bars. In fact, I often see ML papers that claim to have beaten the state of the art, but their accuracy is 93.2% while the other models reach 93.1%: I'm pretty sure this difference falls within the confidence interval.

A few methods are mentionned in the book:

• Use k-fold cross-validation: you get k metrics instead of one, which gives you get an idea of the variance of your estimate.
• For metrics that are just the mean of a per-sample metric (e.g., accuracy, MSE, MAE...), the central limit theorem applies (assuming the dataset is large enough, which is generally the case): we can look at the sample variance of the metric, s², and the 95% confidence interval is the mean metric ±1.96 s / √_n_ (where n is the dataset size).
• If the metric is more complicated than a simple mean, for example the RMSE, we can use boostrapping, as shown in the Chapter 2 notebook:

When the dataset has tens of millions of samples, the term √n grows large, so the confidence interval shrinks. I guess that's one reason why error bars might be omitted: they are no longer meaningful at this scale. Indeed, if the confidence interval is from 93.2244% to 93.2245%, then it's no use reporting it because it's almost certainly wrong: several assumptions are probably not that accurate, for example the test set is frozen in time but real world data continues to evolve, and it's hard to estimate how much this matters (not impossible, but complicated and not necessarily worth the effort). Moreover, even if it was perfectly accurate, it would still be an average, and user experience might vary greatly (e.g., perhaps the model only has 10% accuracy on average for some categories of users).

I hope this helps!

[945fc41467]

Thank you very much.

Yes I think in answer my question. I will have to experiment this next time I have the occasion.