Another cool Scana work on memory-augmented meshes. Again, it's strange that no one has proposed such an architecture before (myself included)… Sequential Memory Machines Ciaran Regan, Kai Errolkumarn, Luke Darlow, Stefania Droga, Sebastian Risi, Leon Jones Paper: Review: Code: What they did: The authors presented a recurrent neural network, the Sequential Memory Machine (CMM), that separates fast computation from long-term storage using two memory matrices. The architecture extends their previous work, the Continuous Thinking Machine (CTM), which computed using the firing histories of individual neurons and their pairwise correlations. The CMM added a fixed long-term memory matrix and a common transformer that updates both memories simultaneously. What we got: In the priority sorting task, the binary cross-entropy of the CMM under test was 4.1x10^-3 compared to 1.4x10-2 for a network with an external memory differentiable neural computer (DNC) and 2.7x10^-2 for a causal transformer. In a 39x39 maze pass, CMM completely solves 61.6% of mazes (from 22.8% to 90.7% for different seeds) vs. 37.4% for CTM, and in a 20-shot linear regression it achieves 30.63% accuracy vs. 8.78% for CTM Why it's important: attention with a special sink zero token allows the recurrent network to decide when to read from long-term storage and when to rely on fast local dynamics. True, running the transformer at each iteration step increases the peak GPU memory consumption in the copy-100 task to 22.3 GiB compared to 634.5 MiB for CTM. Increase your memory here.

Another cool Scana work on memory-augmented meshes. Again, it's strange that no one has proposed such an architecture before (myself included)… Sequential Memory Machines Ciaran Regan, Kai Errolkumarn, Luke Darlow, Stefania Droga, Sebastian Risi, Leon Jones Paper: Review: Code: What they did: The authors presented a recurrent neural network, the Sequential Memory Machine (CMM), that separates fast computation from long-term storage using two memory matrices. The architecture extends their previous work, the Continuous Thinking Machine (CTM), which computed using the firing histories of individual neurons and their pairwise correlations. The CMM added a fixed long-term memory matrix and a common transformer that updates both memories simultaneously. What we got: In the priority sorting task, the binary cross-entropy of the CMM under test was 4.1x10^-3 compared to 1.4x10-2 for a network with an external memory differentiable neural computer (DNC) and 2.7x10^-2 for a causal transformer. In a 39x39 maze pass, CMM completely solves 61.6% of mazes (from 22.8% to 90.7% for different seeds) vs. 37.4% for CTM, and in a 20-shot linear regression it achieves 30.63% accuracy vs. 8.78% for CTM Why it's important: attention with a special sink zero token allows the recurrent network to decide when to read from long-term storage and when to rely on fast local dynamics. True, running the transformer at each iteration step increases the peak GPU memory consumption in the copy-100 task to 22.3 GiB compared to 634.5 MiB for CTM. Increase your memory here.