Inverse problems

March 30, 2016 — June 30, 2022

functional analysis
linear algebra
probability
sparser than thou
statistics
Figure 1: X-ray crystallography is a classic artisanal inverse problem, so having a picture related to that makes me look clever.

Robert Ackroyd introduces some handy phrasing for the connections between statistical estimation theory and inverse problem solving.

Photogrammetry, MRIs, SLAM, volumetric reconstruction and X-ray crystallography are all examples of inverse problems. any of these can be constructed as classical belief propagaation, especially Gaussian BP, or in a basic case least squares.

I happen to think that this is a case where it is much easier to explain in terms of Bayesian inference, so my attempt at an actual explanation is under Bayesian inverse problems.

We can do it in terms of frequentist methods, but it does not add much in the way of explanatory value; we end up considering regularizers instead of priors, but the working in between is pretty much the same. (IMO) However, in doing so we focus on point estimates rather than entire densities, which encourages us to solve the problem by optimisation rather than integration, which is a useful insight, for example, when we consider Laplace approximations.

1 Domain-specific model inversion

PEST, PEST++, and pyemu are some integrated systems for uncertainty quantification that use some weird terminology, such a FOSM (First-order-second-moment) models. They use various linear-algebra tricks to find plausible subspaces and samples.

2 Interesting specific techniques

Figure 2

Leaning to reconstruct introduces partly-learned, partly designed reconstruction operator trick. 🏗️

3 Radiance fields

A fun way of reconstructing objects from photos; differentiable photogrammetry.

4 References

Adler, and Öktem. 2018. Learned Primal-Dual Reconstruction.” IEEE Transactions on Medical Imaging.
Alberti, De Vito, Lassas, et al. 2021. Learning the Optimal Regularizer for Inverse Problems.” arXiv:2106.06513 [Cs, Math, Stat].
Aster, Borchers, and Thurber. 2019. Parameter Estimation and Inverse Problems.
Basir, and Senocak. 2022. Physics and Equality Constrained Artificial Neural Networks: Application to Forward and Inverse Problems with Multi-Fidelity Data Fusion.” Journal of Computational Physics.
Bissantz, Hohage, and Munk. 2004. Consistency and Rates of Convergence of Nonlinear Tikhonov Regularization with Random Noise.” Inverse Problems.
Borcea, Druskin, and Knizhnerman. 2005. On the Continuum Limit of a Discrete Inverse Spectral Problem on Optimal Finite Difference Grids.” Communications on Pure and Applied Mathematics.
Borgerding, and Schniter. 2016. Onsager-Corrected Deep Networks for Sparse Linear Inverse Problems.” arXiv:1612.01183 [Cs, Math].
Brehmer, Louppe, Pavez, et al. 2020. Mining Gold from Implicit Models to Improve Likelihood-Free Inference.” Proceedings of the National Academy of Sciences.
Bui-Thanh. 2012. A Gentle Tutorial on Statistical Inversion Using the Bayesian Paradigm.”
Chen, and Oliver. 2013. Levenberg–Marquardt Forms of the Iterative Ensemble Smoother for Efficient History Matching and Uncertainty Quantification.” Computational Geosciences.
Cranmer, Brehmer, and Louppe. 2020. The Frontier of Simulation-Based Inference.” Proceedings of the National Academy of Sciences.
Daubechies, Defrise, and De Mol. 2004. An Iterative Thresholding Algorithm for Linear Inverse Problems with a Sparsity Constraint.” Communications on Pure and Applied Mathematics.
Engl, Hofinger, and Kindermann. 2005. Convergence Rates in the Prokhorov Metric for Assessing Uncertainty in Ill-Posed Problems.” Inverse Problems.
Engl, and Nashed. 1981. Generalized Inverses of Random Linear Operators in Banach Spaces.” Journal of Mathematical Analysis and Applications.
Fernández-Martínez, Fernández-Muñiz, Pallero, et al. 2013. From Bayes to Tarantola: New Insights to Understand Uncertainty in Inverse Problems.” Journal of Applied Geophysics.
Grigorievskiy, Lawrence, and Särkkä. 2017. Parallelizable Sparse Inverse Formulation Gaussian Processes (SpInGP).” In arXiv:1610.08035 [Stat].
Holl, Koltun, and Thuerey. 2022. Scale-Invariant Learning by Physics Inversion.” In.
Kaipio, and Somersalo. 2005. Statistical and Computational Inverse Problems. Applied Mathematical Sciences.
Kaipio, and Somersalo. 2007. Statistical Inverse Problems: Discretization, Model Reduction and Inverse Crimes.” Journal of Computational and Applied Mathematics.
Lehtinen, Paivarinta, and Somersalo. 1989. Linear Inverse Problems for Generalised Random Variables.” Inverse Problems.
Mandelbaum. 1984. Linear Estimators and Measurable Linear Transformations on a Hilbert Space.” Zeitschrift Für Wahrscheinlichkeitstheorie Und Verwandte Gebiete.
Mosegaard, and Tarantola. 1995. Monte Carlo Sampling of Solutions to Inverse Problems.” Journal of Geophysical Research: Solid Earth.
———. 2002. Probabilistic Approach to Inverse Problems.” In International Geophysics.
Murray-Smith, and Pearlmutter. 2005. Transformations of Gaussian Process Priors.” In Deterministic and Statistical Methods in Machine Learning. Lecture Notes in Computer Science.
O’Callaghan, and Ramos. 2011. Continuous Occupancy Mapping with Integral Kernels.” In Twenty-Fifth AAAI Conference on Artificial Intelligence.
O’Sullivan. 1986. A Statistical Perspective on Ill-Posed Inverse Problems.” Statistical Science.
Oliver. 2022. Hybrid Iterative Ensemble Smoother for History Matching of Hierarchical Models.” Mathematical Geosciences.
Pikkarainen. 2006. State Estimation Approach to Nonstationary Inverse Problems: Discretization Error and Filtering Problem.” Inverse Problems.
Plumlee. 2017. Bayesian Calibration of Inexact Computer Models.” Journal of the American Statistical Association.
Putzky, and Welling. 2017. Recurrent Inference Machines for Solving Inverse Problems.” arXiv:1706.04008 [Cs].
Qian. 2023. Xinvert: A Python Package for Inversion Problems in Geophysical Fluid Dynamics.” Journal of Open Source Software.
Schnell, Holl, and Thuerey. 2022. Half-Inverse Gradients for Physical Deep Learning.” arXiv:2203.10131 [Physics].
Schwab, and Stuart. 2012. Sparse Deterministic Approximation of Bayesian Inverse Problems.” Inverse Problems.
Stuart. 2010. Inverse Problems: A Bayesian Perspective.” Acta Numerica.
Sun, Scanlon, Save, et al. 2021. Reconstruction of GRACE Total Water Storage Through Automated Machine Learning.” Water Resources Research.
Tait, and Damoulas. 2020. Variational Autoencoding of PDE Inverse Problems.” arXiv:2006.15641 [Cs, Stat].
Tarantola. 2005. Inverse Problem Theory and Methods for Model Parameter Estimation.
Tonolini, Radford, Turpin, et al. 2020. Variational Inference for Computational Imaging Inverse Problems.” Journal of Machine Learning Research.
Tropp, and Wright. 2010. Computational Methods for Sparse Solution of Linear Inverse Problems.” Proceedings of the IEEE.
Wei, Fan, Carin, et al. 2017. An Inner-Loop Free Solution to Inverse Problems Using Deep Neural Networks.” arXiv:1709.01841 [Cs].
Welter, Doherty, Hunt, et al. 2012. Approaches in Highly Parameterized Inversion: PEST++, a Parameter Estimation Code Optimized for Large Environmental Models.”
Welter, White, Hunt, et al. 2015. Approaches in Highly Parameterized Inversion—PEST++ Version 3, a Parameter ESTimation and Uncertainty Analysis Software Suite Optimized for Large Environmental Models.” USGS Numbered Series 7-C12. Techniques and Methods.
White. 2018. A Model-Independent Iterative Ensemble Smoother for Efficient History-Matching and Uncertainty Quantification in Very High Dimensions.” Environmental Modelling & Software.
White, Fienen, Barlow, et al. 2018. A Tool for Efficient, Model-Independent Management Optimization Under Uncertainty.” Environmental Modelling & Software.
White, Fienen, and Doherty. 2016a. pyEMU: A Python Framework for Environmental Model Uncertainty Analysis Version .01.”
———. 2016b. A Python Framework for Environmental Model Uncertainty Analysis.” Environmental Modelling & Software.
White, Hunt, Fienen, et al. 2020. Approaches to Highly Parameterized Inversion: PEST++ Version 5, a Software Suite for Parameter Estimation, Uncertainty Analysis, Management Optimization and Sensitivity Analysis.” USGS Numbered Series 7-C26. Techniques and Methods.
Zammit-Mangion, Bertolacci, Fisher, et al. 2021. WOMBAT v1.0: A fully Bayesian global flux-inversion framework.” Geoscientific Model Development Discussions.
Zhang, Lu, Guo, et al. 2019. Quantifying Total Uncertainty in Physics-Informed Neural Networks for Solving Forward and Inverse Stochastic Problems.” Journal of Computational Physics.