They say you may not need several separate phases of training (pre-workout, mid-workout, SFT, RL) and offer uniform training. Training Beam Models with a Single Algorithm Shir Yehuda, Alvin Zhang, Fahim Tajwar, Daman Arora, Jeff Schneider, Daniel Hashvi, Ruslan Selkhotdinov, J. Andrew Bagnell, Arti Singh, Andrea Zent Paper: Code: Model: Overview: What They Did: The authors proposed a unified training framework and an asynchronous system that combines pretraining, intermediate training, SFT, and RL to lossy One maximum likelihood per token weight. Policy gradients are then expressed by adjusting the likelihood to direct reward-dependent distributions. What we got: In logic puzzles with different proportions of data shuffling, pass@1 is about 4-39% on problem sizes from the training distribution (vs. 12.24-24.54% for stepwise baselines) and about 1-13% on new problems with 12 characters (vs. 5.58% for the chosen interleaved baseline). Before RL begins, mixing intermediate training data with pretraining keeps accuracy on overall benchmarks at 41.85–42.42% (compared to 42.37% for pure pretraining and 35.00–38.49% for separate intermediate training). In Mathematical RL for a model with parameters of 0.76B, a sequential RL in the code gives 23.67–24.96% pass@1 in HumanEval+ with a mixing fraction of 0.5–5% (compared to 22.94% with stepwise training). Why it matters: Separate pipelines for pre-training, SFT and RL are interchangeable with a single continuous data stream and weighted cross-entropy. This improves subsequent adaptation and prevents forgetting. However, all experiments were performed on models with less than 1B parameters; For the 0.76B model, mixing was used only in the final RL, and no mixing schedule turned out to be best for all measures at once.

They say you may not need several separate phases of training (pre-workout, mid-workout, SFT, RL) and offer uniform training. Training Beam Models with a Single Algorithm Shir Yehuda, Alvin Zhang, Fahim Tajwar, Daman Arora, Jeff Schneider, Daniel Hashvi, Ruslan Selkhotdinov, J. Andrew Bagnell, Arti Singh, Andrea Zent Paper: Code: Model: Overview: What They Did: The authors proposed a unified training framework and an asynchronous system that combines pretraining, intermediate training, SFT, and RL to lossy One maximum likelihood per token weight. Policy gradients are then expressed by adjusting the likelihood to direct reward-dependent distributions. What we got: In logic puzzles with different proportions of data shuffling, pass@1 is about 4-39% on problem sizes from the training distribution (vs. 12.24-24.54% for stepwise baselines) and about 1-13% on new problems with 12 characters (vs. 5.58% for the chosen interleaved baseline). Before RL begins, mixing intermediate training data with pretraining keeps accuracy on overall benchmarks at 41.85–42.42% (compared to 42.37% for pure pretraining and 35.00–38.49% for separate intermediate training). In Mathematical RL for a model with parameters of 0.76B, a sequential RL in the code gives 23.67–24.96% pass@1 in HumanEval+ with a mixing fraction of 0.5–5% (compared to 22.94% with stepwise training). Why it matters: Separate pipelines for pre-training, SFT and RL are interchangeable with a single continuous data stream and weighted cross-entropy. This improves subsequent adaptation and prevents forgetting. However, all experiments were performed on models with less than 1B parameters; For the 0.76B model, mixing was used only in the final RL, and no mixing schedule turned out to be best for all measures at once.