Closely related is AutoML.
Problem statement
According to Gilles Louppe and Manoj Kumar:
We are interested in solving
\[x^* = \arg \min_x f(x)\]
under the constraints that
\(f\) is a black box for which no closed form is known (nor its gradients);
\(f\) is expensive to evaluate;
evaluations of \(y=f(x)\) may be noisy.
This is similar to the typical framing of reinforcement learning problems where there is a similar explore/exploit tradeoff, although I do not know the precise disciplinary boundaries that may transect these areas. They both might be thought of as stochastic optimal control problems.
The most common method seems to the âBayesian optimisationâ, which is typically based on Gaussian Process regressions. However, this is not a requirement, and many possible wacky regression models can give you the optimisation surrogate.
Of renewed interest for its use in hyperparameter/model selection, in e.g. regularising complex models, which is compactly referred to these days as automl.
You could also obviously use it in industrial process control, which is where I originally saw this kind of thing, in the form of sequential ANOVA design, which is an incredible idea itself, although that is now years old so is not nearly so hip. Since this effectively an attempt at optimal, nonlinear, heteroskedastic sequential ANOVA, I am led to wonder if we can dispense with ANOVA now. Does this stuff actually work well enough? Or is it the same thing, repackaged?
Implementation

Surrogate Optimization Toolbox (pySOT) for global deterministic optimization problems. pySOT is hosted on GitHub: https://github.com/dme65/pySOT.
The main purpose of the toolbox is for optimization of computationally expensive blackbox objective functions with continuous and/or integer variables. All variables are assumed to have bound constraints in some form where none of the bounds are infinity. The tighter the bounds, the more efficient are the algorithms since it reduces the search region and increases the quality of the constructed surrogate. This toolbox may not be very efficient for problems with computationally cheap function evaluations. Surrogate models are intended to be used when function evaluations take from several minutes to several hours or more.
skopt (aka
scikitoptimize
)[âŠ]is a simple and efficient library to minimize (very) expensive and noisy blackbox functions. It implements several methods for sequential modelbased optimization.

Spearmint is a package to perform Bayesian optimization according to the algorithms outlined in the paper (SnLA12).
The code consists of several parts. It is designed to be modular to allow swapping out various âdriverâ and âchooserâ modules. The âchooserâ modules are implementations of acquisition functions such as expected improvement, UCB or random. The drivers determine how experiments are distributed and run on the system. As the code is designed to run experiments in parallel (spawning a new experiment as soon a result comes in), this requires some engineering.
Spearmint2 is similar, but more recently updated and fancier; however it has a restrictive license prohibiting wide redistribution without the payment of fees. You may or may not wish to trust the implied level of development and support of 4 Harvard Professors, depending on your application.
Both of the Spearmint options (especially the latter) have opinionated choices of technology stack in order to do their optimizations, which means they can do more work for you, but require more setup, than a simple little thing like
skopt
. Depending on your computing environment this might be an overall plus or a minus. SMAC (AGPLv3)
(sequential modelbased algorithm configuration) is a versatile tool for optimizing algorithm parameters (or the parameters of some other process we can run automatically, or a function we can evaluate, such as a simulation).
SMAC has helped us speed up both local search and tree search algorithms by orders of magnitude on certain instance distributions. Recently, we have also found it to be very effective for the hyperparameter optimization of machine learning algorithms, scaling better to high dimensions and discrete input dimensions than other algorithms. Finally, the predictive models SMAC is based on can also capture and exploit important information about the model domain, such as which input variables are most important.
We hope you find SMAC similarly useful. Ultimately, we hope that it helps algorithm designers focus on tasks that are more scientifically valuable than parameter tuning.
Python interface through pysmac.
Refs
AllenZhu, Zeyuan, Yuanzhi Li, Aarti Singh, and Yining Wang. 2017. âNearOptimal Design of Experiments via Regret Minimization.â In PMLR, 126â35. http://proceedings.mlr.press/v70/allenzhu17e.html.
Amos, Brandon, and J. Zico Kolter. 2017. âOptNet: Differentiable Optimization as a Layer in Neural Networks.â In PMLR, 136â45. http://proceedings.mlr.press/v70/amos17a.html.
Feurer, Matthias, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. âEfficient and Robust Automated Machine Learning.â In Advances in Neural Information Processing Systems 28, edited by C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, 2962â70. Curran Associates, Inc. http://papers.nips.cc/paper/5872efficientandrobustautomatedmachinelearning.pdf.
Franceschi, Luca, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. 2017. âOn Hyperparameter Optimization in Learning Systems.â In. https://arxiv.org/abs/1703.01785.
Gelbart, Michael A., Jasper Snoek, and Ryan P. Adams. 2014. âBayesian Optimization with Unknown Constraints.â In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, 250â59. UAIâ14. Arlington, Virginia, United States: AUAI Press. http://hips.seas.harvard.edu/files/gelbartconstraineduai2014.pdf.
GrĂŒnewĂ€lder, Steffen, JeanYves Audibert, Manfred Opper, and John ShaweTaylor. 2010. âRegret Bounds for Gaussian Process Bandit Problems.â In, 9:273â80. https://halenpc.archivesouvertes.fr/hal00654517/document.
Hutter, Frank, Holger H. Hoos, and Kevin LeytonBrown. 2011. âSequential ModelBased Optimization for General Algorithm Configuration.â In Learning and Intelligent Optimization, 6683:507â23. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer, Berlin, Heidelberg. https://doi.org/10.1007/9783642255663_40.
Hutter, Frank, Holger Hoos, and Kevin LeytonBrown. 2013. âAn Evaluation of Sequential ModelBased Optimization for Expensive Blackbox Functions.â In Proceedings of the 15th Annual Conference Companion on Genetic and Evolutionary Computation, 1209â16. GECCO â13 Companion. New York, NY, USA: ACM. https://doi.org/10.1145/2464576.2501592.
Li, Lisha, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2016. âHyperband: A Novel BanditBased Approach to Hyperparameter Optimization,â March. http://arxiv.org/abs/1603.06560.
MoÄkus, J. 1975. âOn Bayesian Methods for Seeking the Extremum.â In Optimization Techniques IFIP Technical Conference, edited by Prof Dr G. I. Marchuk, 400â404. Lecture Notes in Computer Science. Springer Berlin Heidelberg. https://doi.org/10.1007/9783662385272_55.
Snoek, Jasper, Hugo Larochelle, and Ryan P. Adams. 2012. âPractical Bayesian Optimization of Machine Learning Algorithms.â In Advances in Neural Information Processing Systems, 2951â9. Curran Associates, Inc. http://papers.nips.cc/paper/4522practicalbayesianoptimizationofmachinelearningalgorithms.
Snoek, Jasper, Kevin Swersky, Rich Zemel, and Ryan Adams. 2014. âInput Warping for Bayesian Optimization of NonStationary Functions.â In Proceedings of the 31st International Conference on Machine Learning (ICML14), 1674â82. http://www.jmlr.org/proceedings/papers/v32/snoek14.pdf.
Srinivas, Niranjan, Andreas Krause, Sham M. Kakade, and Matthias Seeger. 2012. âGaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design.â IEEE Transactions on Information Theory 58 (5): 3250â65. https://doi.org/10.1109/TIT.2011.2182033.
Staines, Joe, and David Barber. 2012. âVariational Optimization,â December. http://arxiv.org/abs/1212.4507.
Swersky, Kevin, Jasper Snoek, and Ryan P Adams. 2013. âMultiTask Bayesian Optimization.â In Advances in Neural Information Processing Systems 26, edited by C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, 2004â12. Curran Associates, Inc. http://papers.nips.cc/paper/5086multitaskbayesianoptimization.pdf.