Designed to automate the design of machine learning (ML) algorithms, AutoML has until now focused on constructing solutions by combining sophisticated hand-designed components. However, an alternative approach to using such hand-designed components in AutoML, say Google researchers, is to search for entire algorithms from scratch.
“This is challenging because it requires the exploration of vast and sparse search spaces, yet it has great potential benefits — it is not biased toward what we already know and potentially allows for the discovery of new and better ML architectures,” says the researchers. “By analogy, if one were building a house from scratch, there is more potential for flexibility or improvement than if one was constructing a house using only prefabricated rooms. However, the discovery of such housing designs may be more difficult because there are many more possible ways to combine the bricks and mortar than there are of combining pre-made designs of entire rooms.”
As a result, say the researchers, early research into algorithm learning from scratch focused on one aspect of the algorithm – to reduce the search space and compute required, such as the learning rule – and has not been revisited much since the early 90s. However, in a new paper, Google researchers say they have demonstrated that it is possible to successfully evolve ML algorithms from scratch.
Their approach, called AutoML-Zero, starts from empty programs and, using only basic mathematical operations as building blocks, applies evolutionary methods to automatically find the code for complete ML algorithms. Given small image classification problems, say the researchers, the method rediscovered fundamental ML techniques such as two-layer neural networks with back-propagation, linear regression, and the like, which have been invented by researchers throughout the years – a result that demonstrates the plausibility of automatically discovering more novel ML algorithms to address harder problems in the future.
In their work, the researchers used a variant of classic evolutionary methods to search the
