# Density estimation

A statistical estimation problem where you are not trying to estimate a function of a distribution of random observations, but the distribution itself. In a sense, all of statistics implicitly does density estimation, but this is usually purely instrumental in the course of discovering the real parameter of interest.

So, estimating distributions nonparametrically we do as with the usual function approximation. We might set ourselves different loss functions than in regression also; Instead of, e.g. expected $L_p$ prediction error we might use a traditional function approximation $L_p$ loss, or a probability divergence measure.

Unlike general functional approximation, the problem is constrained; Our density has to integrate to unity by definition; it may not be negative.

The most common one, that we use implicity all the time, is to simply take the empirical distribution as a distribution estimate; that is, taking the data as a model for itself. This has various non-useful features such as being rough and rather hard to visualise, unless your spectacle prescription can resolve Dirac deltas.

Two common smoother methods are kernel density estimates and mixture models (aka radial basis functions), but you might also like to try splines or arbitrary functional bases such as dictionary/wavelet bases or product kernel methods or log/non-negative regression, or, relatedly, gaussian process methods? (I am told they are related; but I know little of them)

Questions:

1. When would I actually want to estimate, specifically, a density?
• Visualisation
• nonparametric regression without any better ideas
• …?
1. What are appropriate purpose for each of probability metrics?
2. What about non-parametric conditional density estimation? Are there any general ways to do this?
3. What does the kernel trick get me here?
4. Can I use spectral methods to do this outside of kernel estimation?

## Divergence measures/contrasts

There are many choices for loss functions between densities here; any of the probability metrics will do. For reasons of tradition or convenience, when the object of interest is the density itself, certain choices dominate:

• $L_2$ with respect to the density over Lebesgue measure on the state space, which we call the MISE.
• KL-divergence. (May not do what you want if you care about performance near 0. See Hall87.)
• Hellinger distance

But there are others.

## Risk Bounds

But having chosen the divergence you wish to minimise, you now have to choose with respect to which criterion you wish to minimse it? Minimax? in probability? In expectation? …? Every combination is a different publication. Hmf.

## Minimising Expected (or whatever) MISE

Surprsingly complex. This works fine for Kernel Density Estimators where it turns out just be a Wiener filter where you have to choose a bandwidth. How do you do this for parametric estimators, though?

## Connection to point processes

There is a connection between spatial point process intensity estimation and density estimation. See Densities and intensities.

## k-NN estimates.

Filed here because too small to do elsewhere

To use nearest neighbor methods, the integer k must be selected. This is similar to bandwidth selection, although here k is discrete, not continuous. K.C. Li (Annals of Statistics, 1987) showed that for the knn regression estimator under conditional homoskedasticity, it is asymptotically optimal to pick k by Mallows, Generalized CV, or CV. Andrews (Journal of Econometrics, 1991) generalized this result to the case of heteroskedasticity, and showed that CV is asymptotically optimal.

## kernel density estimators

### Fancy ones

HT Gery Geenens for a lecture he just gave on convolution kernel density estimation, where he drew a nice parallel between additive noise in kde estimation and multiplicative noise in non-negative-valued variables.

## Refs

Necessarily scattershot; you probably want to see the refs for a particular density estimation method.

BaLi13
Battey, H., & Liu, H. (2013) Smooth projected density estimation. arXiv:1308.3968 [Stat].
BeDi89
Berman, M., & Diggle, P. (1989) Estimating Weighted Integrals of the Second-Order Intensity of a Spatial Point Process. Journal of the Royal Statistical Society. Series B (Methodological), 51(1), 81–92.
Birg08
Birgé, L. (2008) Model selection for density estimation with L2-loss. arXiv:0808.1416 [Math, Stat].
Bosq98
Bosq, D. (1998) Nonparametric statistics for stochastic processes: estimation and prediction. (2nd ed.). New York: Springer
Cox65
Cox, D. R.(1965) On the Estimation of the Intensity Function of a Stationary Point Process. Journal of the Royal Statistical Society. Series B (Methodological), 27(2), 332–337.
CuSS08
Cunningham, J. P., Shenoy, K. V., & Sahani, M. (2008) Fast Gaussian process methods for point process intensity estimation. (pp. 192–199). ACM Press DOI.
Efro07
Efromovich, S. (2007) Conditional density estimation in a regression setting. The Annals of Statistics, 35(6), 2504–2535. DOI.
EiMa96
Eilers, P. H. C., & Marx, B. D.(1996) Flexible smoothing with B-splines and penalties. Statistical Science, 11(2), 89–121. DOI.
Elli91
Ellis, S. P.(1991) Density estimation for point processes. Stochastic Processes and Their Applications, 39(2), 345–358. DOI.
GiKM08
Giesecke, K., Kakavand, H., & Mousavi, M. (2008) Simulating point processes by intensity projection. In Simulation Conference, 2008. WSC 2008. Winter (pp. 560–568). DOI.
Gu93
Gu, C. (1993) Smoothing Spline Density Estimation: A Dimensionless Automatic Algorithm. Journal of the American Statistical Association, 88(422), 495–504. DOI.
Hall87
Hall, P. (1987) On Kullback-Leibler Loss and Density Estimation. The Annals of Statistics, 15(4), 1491–1519. DOI.
HaIb90
Hasminskii, R., & Ibragimov, I. (1990) On Density Estimation in the View of Kolmogorov’s Ideas in Approximation Theory. The Annals of Statistics, 18(3), 999–1010. DOI.
Ibra01
Ibragimov, I. (2001) Estimation of analytic functions. In Institute of Mathematical Statistics Lecture Notes - Monograph Series (pp. 359–383). Beachwood, OH: Institute of Mathematical Statistics
KoMi06
Koenker, R., & Mizera, I. (2006) Density estimation by total variation regularization. Advances in Statistical Modeling and Inference, 613–634.
Nore10
Norets, A. (2010) Approximation of conditional densities by smooth mixtures of regressions. The Annals of Statistics, 38(3), 1733–1766. DOI.
PaZe16
Panaretos, V. M., & Zemel, Y. (2016) Separation of Amplitude and Phase Variation in Point Processes. The Annals of Statistics, 44(2), 771–812. DOI.
Papa74
Papangelou, F. (1974) The conditional intensity of general point processes and an application to line processes. Zeitschrift Für Wahrscheinlichkeitstheorie Und Verwandte Gebiete, 28(3), 207–226. DOI.
ReRT11
Reynaud-Bouret, P., Rivoirard, V., & Tuleau-Malot, C. (2011) Adaptive density estimation: A curse of support?. Journal of Statistical Planning and Inference, 141(1), 115–139. DOI.
Scho05
Schoenberg, F. P.(2005) Consistent parametric estimation of the intensity of a spatial–temporal point process. Journal of Statistical Planning and Inference, 128(1), 79–93. DOI.
ShSh10
Shimazaki, H., & Shinomoto, S. (2010) Kernel bandwidth optimization in spike rate estimation. Journal of Computational Neuroscience, 29(1–2), 171–182. DOI.
STSK10
Sugiyama, M., Takeuchi, I., Suzuki, T., Kanamori, T., Hachiya, H., & Okanohara, D. (2010) Conditional density estimation via least-squares density ratio estimation. In International Conference on Artificial Intelligence and Statistics (pp. 781–788).
Lies11
van Lieshout, M.-C. N. M.(2011) On Estimation of the Intensity Function of a Point Process. Methodology and Computing in Applied Probability, 14(3), 567–578. DOI.
ZeMe97
Zeevi, A. J., & Meir, R. (1997) Density Estimation Through Convex Combinations of Densities: Approximation and Estimation Bounds. Neural Networks: The Official Journal of the International Neural Network Society, 10(1), 99–109. DOI.