cool story All mass implementations of softmax and crossentropy have bugs that lead to rounding problems. And now, after more than a dozen years of actively using all this, they have finally been discovered. Again the question arises, why not first. Fix for Softmax and cross-entropy Jonathan T Barron, Dan Ganapragasam, Michael Channan Paper: Review: Code: N/A (standalone JAX snippets are provided in the paper).
Model: Not available What they did: The authors rewrote softmax, log-softmax, and cross-entropy (for hard and soft labels) in logarithmic space. This fixes two circular bugs in deep learning frameworks: Softmax Collapse (resetting the slope of the target logit when the model confidence is high) and Softmax Cancellation (a non-zero degree in the refinement when the students' logits already overlap the teacher's logits). What we have: on modular addition (a + b mod 31) in bf16, fp32 and fp64 constant cross-entropy gives 98.9%-100.0% test accuracy compared to 93.3%-100.0% for standard cross-entropy, 48.9%-97. 9% for zero-sum screening (subtracting the mean from the ranks) and 95.6%-97.4% for StableMax (a rational replacement of Softmax). In unregular polynomial greking (a^2 + b^2 mod 97), constant cross-entropy achieves 85% accuracy, while standard cross-entropy drops from 61% to 23%. Why it's important: Drop switching of basic loss functions removes loss jumps ("rogateka effect", slingshot) during drag and weight drift during refining in bf16 without going to fp64. True, all the experiments were performed on small problems of modular calculus, and the huge logit margins and almost zero student-teacher gaps examined here are rare in large-scale learning. Round up with and without bugs here.