- A parameter is associated with a population.
- A statistic is associated with a sample.
- A statistic is biased if it systematically overestimates or underestimates a parameter.
- A statistic is unbiased if it does systematically overestimate or underestimate a parameter.
Everything we care about is an estimator; the sample mean estimates the population mean, the sample variance estimates the population variance, the statistical learning model built on the sample estimates the statistical model that generates the population.
$$ \begin{align}
E[\hat{\theta}] & = \theta \
\text{For example:} \
E[\bar{X}] & = \mu
\end{align} $$