The problem of estimating the probability density function for a given set of observations.
Empirical distribution function¶
Compute the empirical CDF and numerically differentiate it.
Take the range of the sample and split it up into n bins, where n is a hyperparameter. Then assign a probability to each bin according to the proportion of the sample that fell within its bounds.
Kernel Density Estimation¶
The predicted density function given an a sample is:
Where is the kernel and is a smoothing parameter.
A variety of kernels can be used. A common one is the Gaussian, defined as:
The complexity at inference time is linear in the size of the sample.
Estimates the density as a weighted sum of parametric distributions. The predicted density function for a sample is:
Where is the number of distributions and each distribution, , is parameterised by . It is also weighted by a single scalar where .
The Gaussian is a common choice for the distribution. In this case the estimator is known as a Gaussian Mixture Model.
All of the parameters can be learnt using Expectation-Maximization, except for which is a hyperparameter.