There is such a funny approach as synthetic pre-education, when the model is trained on some structured synthetics (cellular automata, languages ​​with balanced visions, etc.) so that it learns some potentially useful structures in advance. Now, new work shows that if you take all tokens and arbiters into account honestly, it doesn't work. Once natural language learning kicks in, it makes all of these embedded gulfs true. I remember a joke about how my dad came and changed everything in his own way. Rethinking Pre-Pretraining Alexander Stern, Tim Rocktäschel, Acyr Locatelli, Laura Ruis Paper: Code: , Review: What: We tested whether synthetic pre-training (PPT) - the pre-limiting step on proximately generated sequences - helps with a fixed FLOP budget. I proposed a synthetic case for finding variable values ​​DeclRef and Hyperparameter optimization based on LLMTuna. In none of the tested situations with equal computing budget, PPT beats natural language learning alone. With a data deficit (100M FineWeb tokens, 1×1018 FLOPs), repeating plaintext gives 1.064 bits per BPB (BPB) on verification, and spreading 1%–10% FLOPs on PPT verifies the benchmark to 1.068–76 BPB. When reconstructing PPT on a cellular automaton (NCA) for LLaMA with 400M parameters (OpenWebText), changing the initialization of the embeddings from N(0, 1) to N(0, 0.02) improves the baseline confusion without PPT to 9.0%, completely overlapping the 5.4% gain from NCA PPT. Why it's important: Accepting synthetic waste calculation: In moving to natural language, the model writes the scholarly structures, and the previous successes of PPT largely compensate for poor allowances of scale.

There is such a funny approach as synthetic pre-education, when the model is trained on some structured synthetics (cellular automata, languages ​​with balanced visions, etc.) so that it learns some potentially useful structures in advance. Now, new work shows that if you take all tokens and arbiters into account honestly, it doesn't work. Once natural language learning kicks in, it makes all of these embedded gulfs true. I remember a joke about how my dad came and changed everything in his own way. Rethinking Pre-Pretraining Alexander Stern, Tim Rocktäschel, Acyr Locatelli, Laura Ruis Paper: Code: , Review: What: We tested whether synthetic pre-training (PPT) - the pre-limiting step on proximately generated sequences - helps with a fixed FLOP budget. I proposed a synthetic case for finding variable values ​​DeclRef and Hyperparameter optimization based on LLMTuna. In none of the tested situations with equal computing budget, PPT beats natural language learning alone. With a data deficit (100M FineWeb tokens, 1×1018 FLOPs), repeating plaintext gives 1.064 bits per BPB (BPB) on verification, and spreading 1%–10% FLOPs on PPT verifies the benchmark to 1.068–76 BPB. When reconstructing PPT on a cellular automaton (NCA) for LLaMA with 400M parameters (OpenWebText), changing the initialization of the embeddings from N(0, 1) to N(0, 0.02) improves the baseline confusion without PPT to 9.0%, completely overlapping the 5.4% gain from NCA PPT. Why it's important: Accepting synthetic waste calculation: In moving to natural language, the model writes the scholarly structures, and the previous successes of PPT largely compensate for poor allowances of scale.