AutoML
July 17, 2017 — October 2, 2020
The sub-field of optimisation that specifically aims to automate model selection in machine learning. (and also occasionally ensemble construction)
There are two major approaches that I am aware of, both of which are related in a kind of abstract way, but which are in practice different
- Finding the right architecture for your nueral net, a.k.a architecture search
- Hyperparameter optimisation which I have made into a separate notebook.
The first one I might cover .
TODO: work out if this is the same as “meta learning”? I think not; I suspect that of being transfer learning.
1 Reinforcement learning approaches
Quoc Le & Barret Zoph discuss using reinforcement learning to learn neural models:
Typically, our machine learning models are painstakingly designed by a team of engineers and scientists. This process of manually designing machine learning models is difficult because the search space of all possible models can be combinatorially large — a typical 10-layer network can have ~1010 candidate networks! […]
To make this process of designing machine learning models much more accessible, we’ve been exploring ways to automate the design of machine learning models. […] in this blog post, we’ll focus on our reinforcement learning approach and the early results we’ve gotten so far.
In our approach (which we call “AutoML”), a controller neural net can propose a “child” model architecture, which can then be trained and evaluated for quality on a particular task. That feedback is then used to inform the controller how to improve its proposals for the next round.
2 Differentiable architecture search
Various tricks I am not so familiar with. Overview in Elsken, Metzen, and Hutter (2019). See, e.g. Weight sharing for neural architecture search. 🏗️
I think there is a bunch of stuff implemented in archai (blogged).
3 Implementations
3.1 Lightwood
George - Epistemink, Lightwood
Specifically, George is interested in thinking about the incentive structures of designing, using and contributing to ML software, and trying to think about what useful and rigorous results can actually be achieved by benchmarking of algorithm performance.
3.2 auto-sklearn
auto-sklearn, The implementation of hyperparameter optimization by Feurer et al. (2015):
auto-sklearnis an automated machine learning toolkit and a drop-in replacement for a scikit-learn estimator:import autosklearn.classification cls = autosklearn.classification.AutoSklearnClassifier() cls.fit(X_train, y_train) predictions = cls.predict(X_test)auto-sklearnfrees a machine learning user from algorithm selection and hyperparameter tuning. It leverages recent advantages in Bayesian optimization, meta-learning and ensemble construction.