The Living Thing / Notebooks : Density estimation

A statistical estimation problem where you are not trying to estimate a function of a distribution of random observations, but the distribution itself. In a sense, all of statistics implicitly does density estimation, but this is usually purely instrumental in the course of discovering the real parameter of interest.

So, estimating distributions nonparametrically we do as with the usual function approximation. We might set ourselves different loss functions than in regression also; Instead of, e.g. expected \(L_p\) prediction error we might use a traditional function approximation \(L_p\) loss, or a probability divergence measure.

Unlike general functional approximation, the problem is constrained; Our density has to integrate to unity by definition; it may not be negative.

The most common one, that we use implicity all the time, is to simply take the empirical distribution as a distribution estimate; that is, taking the data as a model for itself. This has various non-useful features such as being rough and rather hard to visualise, unless your spectacle prescription can resolve Dirac deltas.

Two common smoother methods are kernel density estimates and mixture models (aka radial basis functions), but you might also like to try splines or arbitrary functional bases such as dictionary/wavelet bases or product kernel methods or log/non-negative regression, or, relatedly, gaussian process methods? (I am told they are related; but I know little of them)


  1. When would I actually want to estimate, specifically, a density?
  1. What are appropriate purpose for each of probability metrics?
  2. What about non-parametric conditional density estimation? Are there any general ways to do this?
  3. What does the kernel trick get me here?
  4. Can I use spectral methods to do this outside of kernel estimation?

Divergence measures/contrasts

There are many choices for loss functions between densities here; any of the probability metrics will do. For reasons of tradition or convenience, when the object of interest is the density itself, certain choices dominate:

But there are others.

Risk Bounds

But having chosen the divergence you wish to minimise, you now have to choose with respect to which criterion you wish to minimse it? Minimax? in probability? In expectation? …? Every combination is a different publication. Hmf.

Minimising Expected (or whatever) MISE

Surprsingly complex. this works fine for KDEs where it turns out just be a Wiener filter where you have to choose a bandwidth. How do you do this for parametric estimators, though?

Connection to point processes

There is a connection between spatial point process intensity estimation and density estimation. TBD.

k-NN estimates.

Filed here because too small to do elsewhere

Bruce E. Hansen notes:

To use nearest neighbor methods, the integer k must be selected. This is similar to bandwidth selection, although here k is discrete, not continuous. K.C. Li (Annals of Statistics, 1987) showed that for the knn regression estimator under conditional homoskedasticity, it is asymptotically optimal to pick k by Mallows, Generalized CV, or CV. Andrews (Journal of Econometrics, 1991) generalized this result to the case of heteroskedasticity, and showed that CV is asymptotically optimal.


Necessarily scattershot; you probably want to see the refs for a particular density estimation method.

Battey, H., & Liu, H. (2013) Smooth projected density estimation. arXiv:1308.3968 [Stat].
Berman, M., & Diggle, P. (1989) Estimating Weighted Integrals of the Second-Order Intensity of a Spatial Point Process. Journal of the Royal Statistical Society. Series B (Methodological), 51(1), 81–92.
Birgé, L. (2008) Model selection for density estimation with L2-loss. arXiv:0808.1416 [Math, Stat].
Bosq, D. (1998) Nonparametric statistics for stochastic processes: estimation and prediction. (2nd ed.). New York: Springer
Cox, D. R.(1965) On the Estimation of the Intensity Function of a Stationary Point Process. Journal of the Royal Statistical Society. Series B (Methodological), 27(2), 332–337.
Cunningham, J. P., Shenoy, K. V., & Sahani, M. (2008) Fast Gaussian process methods for point process intensity estimation. (pp. 192–199). ACM Press DOI.
Efromovich, S. (2007) Conditional density estimation in a regression setting. The Annals of Statistics, 35(6), 2504–2535. DOI.
Eilers, P. H. C., & Marx, B. D.(1996) Flexible smoothing with B-splines and penalties. Statistical Science, 11(2), 89–121. DOI.
Ellis, S. P.(1991) Density estimation for point processes. Stochastic Processes and Their Applications, 39(2), 345–358. DOI.
Giesecke, K., Kakavand, H., & Mousavi, M. (2008) Simulating point processes by intensity projection. In Simulation Conference, 2008. WSC 2008. Winter (pp. 560–568). DOI.
Gu, C. (1993) Smoothing Spline Density Estimation: A Dimensionless Automatic Algorithm. Journal of the American Statistical Association, 88(422), 495–504. DOI.
Hall, P. (1987) On Kullback-Leibler Loss and Density Estimation. The Annals of Statistics, 15(4), 1491–1519. DOI.
Hasminskii, R., & Ibragimov, I. (1990) On Density Estimation in the View of Kolmogorov’s Ideas in Approximation Theory. The Annals of Statistics, 18(3), 999–1010. DOI.
Ibragimov, I. (2001) Estimation of analytic functions. In Institute of Mathematical Statistics Lecture Notes - Monograph Series (pp. 359–383). Beachwood, OH: Institute of Mathematical Statistics
Koenker, R., & Mizera, I. (2006) Density estimation by total variation regularization. Advances in Statistical Modeling and Inference, 613–634.
Norets, A. (2010) Approximation of conditional densities by smooth mixtures of regressions. The Annals of Statistics, 38(3), 1733–1766. DOI.
Panaretos, V. M., & Zemel, Y. (2016) Separation of Amplitude and Phase Variation in Point Processes. The Annals of Statistics, 44(2), 771–812. DOI.
Papangelou, F. (1974) The conditional intensity of general point processes and an application to line processes. Zeitschrift Für Wahrscheinlichkeitstheorie Und Verwandte Gebiete, 28(3), 207–226. DOI.
Reynaud-Bouret, P., Rivoirard, V., & Tuleau-Malot, C. (2011) Adaptive density estimation: A curse of support?. Journal of Statistical Planning and Inference, 141(1), 115–139. DOI.
Schoenberg, F. P.(2005) Consistent parametric estimation of the intensity of a spatial–temporal point process. Journal of Statistical Planning and Inference, 128(1), 79–93. DOI.
Shimazaki, H., & Shinomoto, S. (2010) Kernel bandwidth optimization in spike rate estimation. Journal of Computational Neuroscience, 29(1–2), 171–182. DOI.
Sugiyama, M., Takeuchi, I., Suzuki, T., Kanamori, T., Hachiya, H., & Okanohara, D. (2010) Conditional density estimation via least-squares density ratio estimation. In International Conference on Artificial Intelligence and Statistics (pp. 781–788).
van Lieshout, M.-C. N. M.(2011) On Estimation of the Intensity Function of a Point Process. Methodology and Computing in Applied Probability, 14(3), 567–578. DOI.
Zeevi, A. J., & Meir, R. (1997) Density Estimation Through Convex Combinations of Densities: Approximation and Estimation Bounds. Neural Networks: The Official Journal of the International Neural Network Society, 10(1), 99–109. DOI.