Blue Origin · Alignment Forum
Spotting Behaviors to investigate
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
This work was largely done during Neel Nanda's MATS 10.0 Exploration Phase.
Key facts
- Evidence 1: 10/25% Data Filtering does not significantly outperform random
- The team LoRA the OLMo 3 7B mid-train directly, using a rank-64 LoRA applied to all 32 MLP and attention layers
- For each behavior, they ask OLMo-3-32B to generate responses that would score +5 and -5 under that behavior’s rubric, then cache their activations and train an L2-regularized logistic regression probe
- This work was largely done during Neel Nanda's MATS 10.0 Exploration Phase
Summary
J Rosser and Dohun Lee are co-first authors for this post with equal contribution. A natural assumption is that they can control what a LLM learns during training by controlling the data. Surprisingly, this didn't work! The team handpicked a set of behaviors where the SFT model differed from the mid-train. The team first create a lower cost “speed-run” version of the full SFT set up.