● WRITING · 7 NOTES · PROSE AND EVIDENCE, SAME PIPELINE
Notes & write-ups
Write-ups from the bench — what worked, what collapsed under a corrected protocol, and the gates that caught it. The charts inside each post render from the same pinned results files as everything else on this site.
15 JUL 2026Grokking Is a Numerical Coin FlipOn a fully-tractable 11,856-parameter transformer learning addition mod 10, grokking reproduces as two things: conditional and fragile. Conditional, because the grok-rate is an inverted-U in weight decay (20, 27, 90, 0 percent as decay rises), so it can be tuned in or out. Fragile, because at weight decay 0.01, changing only the CPU thread count from 1 to 4, which reorders nothing but floating-point summation, flips 49 of 300 same-seed outcomes; 4 threads and 16 threads are bit-identical, so it is specifically the 1-to-4 reduction change. An exact McNemar test on the 49 flips (28 on, 21 off) gives p = 0.39: symmetric numerical noise, not a hidden knob.grokkingreproducibilitynumericsreproductionreproduce-then-probekaggle14 JUL 2026Is Decoder Crowding Real, or Just Frequency in Disguise?Steer a language model by pushing on one sparse-autoencoder feature and unrelated features light up too. A paper claims a cheap geometric property, decoder crowding, predicts that collateral damage better than the obvious baseline of how often the feature fires. Reproducing it from scratch on GPT-2-small: crowding correlates with collateral at Spearman 0.55 (paper ~0.47), and it survives the control the paper's headline skipped, a partial correlation of 0.57 after regressing out both firing frequency and activation magnitude. The honest divergence: frequency is not as dead here (0.25 versus ~0), so the margin is +0.30 rather than the paper's +0.43 to +0.47.interpretabilitysparse-autoencoderssteeringreproductionreproduce-then-probegpt-2kaggle13 JUL 2026The Coin-Flip Judge, Independently RecomputedA paper claims an LLM asked to judge the same answer pair 50 times at temperature 1.0 flips its preference 13.6 percent of the time. Recomputing every headline number from the authors' own released logs on a free Kaggle CPU reproduces it: 0.136 overall, a spread the average hides (the worst question flips 56 percent), and a temperature-0 residual that shrinks but does not vanish (2.8 / 7.9 percent). The one mismatch, flagged not buried: recomputed first-position bias lands at 70.6 percent against the paper's 72.llm-as-judgeevaluationreproductionreproduce-then-probekaggle12 JUL 2026Feature of the Day: A Gemma SAE AtlasA daily tour through the interpretable features hiding inside Gemma 2 (2B). Day 1 builds the pipeline: a Gemma Scope sparse autoencoder with 16,384 features reads layer 12, fires a median of 72 of them per token, and leaves 51 percent dormant on a 40-prompt probe. Its sharpest direction, feature 4667 (peak activation 393), is a sentence-initial discourse-marker detector that lights up on 'Therefore', 'According' and 'A' and stays near zero everywhere else.interpretabilitysparse-autoencodersgemmagemma-scopemechanistic-interpretabilitykaggle08 JUL 2026Killing the 10 GB TensorMy 596M-parameter model's single biggest training tensor was a 9.96 GB fp32 logit matrix that exists only to be reduced to one scalar. On a free Kaggle T4, fused cross-entropy deletes it: the naive path OOMs past 4,096 tokens while Liger holds flat at ~3.2 GB out to 32,768, agreeing with the fp32 reference to ~1e-7.cudamemorycross-entropyligerkaggleqwen3-0.6b07 JUL 2026The Win That Was NotMy SFT ablation produced a +0.68-perplexity win with a confidence interval tight enough to look unimpeachable. It was an artifact of scoring the two arms on different token sets, and it collapsed to +0.009 on a fixed held-out set.qwen3-0.6bsftevaluationreproduce-then-attribute07 JUL 2026What the Gates CaughtSeven ways my own study tried to fool me (a token-soup data loader, an eval-token confound, a misread smoke log, a conflated data ratio) and the mechanical checks that caught each one before it became a claim.qwen3-0.6bmethodologypre-registrationreproduce-then-attribute
Numbers in prose are rounded; the charts are not./research has every number →