Ken Stanley and colleagues continue to explore factorized representations. In this work they tried to learn them through normal gradient descent (in previous works they showed that GD does not learn them). To make learning successful, we've added a smart (and bold) autoencoder nearby that selectively learns from the purest examples.
Are UFOs real? Introduction to United Organizations Thomas X. Shepard, Alon Alblak, Joel Lehmann, Kenneth O. Stanley Paper: Review: Code.
The bottom line: the standard task loss weakly constrains the network's internal representations. Co-training of a low-capacity selective auto-encoder on normalized activation guides descending slope to fixed patterns and accelerates delayed generalization.
What they did: The authors proposed Unified Organizations (UFOs), adjoint nonlinear autoencoders that simultaneously learn to reproduce the normalized latent activations or attentional weights of the main network. To prevent the autoencoder from adapting to irregular features, its weights are only updated on representations with the least error, and the main network is penalized on all samples.
What we have: In image regression, UFOs replace chaotic latent activations with symmetric feature maps. In modular partitioning (where delayed generalization is called grok), selective UFOs reduce the median number of steps of the main network by a factor of 2.04–12.25 relative to weak weight decay. In most configurations, gurgling occurs without it.
Why it matters: Learning representation schemes help create regular patterns during incremental learning. However, the actual running time is not measured, and the tests are limited to small synthetic tasks with fixed positions.
A UFO arrived and sped up surfing here.