Llama · eliebak.com
J-space comparisons across open models
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
Everything on this page comes from a single measurement.
Key facts
- The 32B's rate falls as 1/t but never reaches zero, its final state scores CKA 0.84 against itself 200 billion tokens earlier
- Seeds agree on the shape (pairwise CKA 0.85–0.91) but not on the number: the three 9.7B seeds measure 344, 444 and 492
- The team re-measured two models on 4,096-token sequences, every prompt a full 4,096 tokens, distances bucketed out to Δ ≈ 3,000
- Along the compute-optimal ladder, growth stops near 2×10²⁰ FLOPs: with the seed-measured noise, the single power law is rejected at p ≈ 0.002–0.014
Summary
Multiplying by the unembedding makes the influence concrete. Two numbers summarize a dictionary. The recipe: center a dictionary's vectors, then build its table of relations K = VV T, a 4,096 × 4,096 grid whose cell (i, j) records how strongly entry i points along entry j. CKA(A, B) = ⟨K A, K B ⟩ / ‖K A ‖ ‖K B ‖. — 1.0 when the geometry is identical, near zero for unrelated random tables. Computed between every pair of a model's own layers, the same number draws a map with visible blocks, an input-side block that reads, a long middle block (the paper's "workspace"), and a small output-side block that writes.