Grokking Is a Numerical Coin Flip
On a fully-tractable 11,856-parameter transformer learning addition mod 10, grokking reproduces as two things: conditional and fragile. Conditional, because the grok-rate is an inverted-U in weight decay (20, 27, 90, 0 percent as decay rises), so it can be tuned in or out. Fragile, because at weight decay 0.01, changing only the CPU thread count from 1 to 4, which reorders nothing but floating-point summation, flips 49 of 300 same-seed outcomes; 4 threads and 16 threads are bit-identical, so it is specifically the 1-to-4 reduction change. An exact McNemar test on the 49 flips (28 on, 21 off) gives p = 0.39: symmetric numerical noise, not a hidden knob.
Grokking is the result where a small network trains to memorize, sits at chance on held-out data for a long time, and then abruptly generalizes long after the training loss has flatlined. It is usually shown as a single dramatic run: one seed, one loss curve, one phase transition. This post takes a paper that argues that framing is misleading (Grokking Is Conditional and Fragile, arXiv:2607.05104) and recomputes its two central claims from the authors' released per-seed records, on a free Kaggle CPU.
This is the third entry in a reproduce-then-probe series, where an autonomous loop surfaces a recent claim and a notebook recomputes it behind a gate that refuses to print a number it cannot trace to a released file. Here every headline grok-rate is bound to the paper's figure by a hard assertion, so a drift in the released data fails the run rather than slipping into the post.
The task is about as tractable as deep learning gets: an 11,856-parameter Llama-style transformer learning (a + b) mod 10. Small enough that you can run hundreds of seeds and ask statistical questions instead of telling anecdotes. "Grok" is defined as held-out accuracy reaching 0.70, and the honest unit is the grok-rate: the fraction of seeds that get there.
Conditional: an inverted-U in weight decay
The first claim is that grokking is not a switch that is either on or off but a regime you tune into. Recomputing the grok-rate at four weight-decay settings reproduces the shape exactly:
weight decay grok-rate 95% CI
0.0 2/10 = 20% [ 6, 51]% (n=10, wide)
0.01 81/300 = 27% [22, 32]%
0.1 9/10 = 90% [60, 98]% (n=10, wide)
1.0 0/10 = 0% [ 0, 28]% (n=10, wide)
Too little regularization rarely groks, too much never does, and there is a sweet spot at weight decay 0.1. The shape is unambiguous. The exact endpoint percentages are not, and the notebook says so: three of the four points are only 10 seeds, so their Wilson intervals are wide (the 90 percent point could honestly be anywhere from 60 to 98). This is the difference between a real inverted-U and a precise curve, and conflating the two is how you overclaim from small samples.
Fragile: the same seed, flipped by arithmetic
The second claim is the sharp one. At weight decay 0.01, the setting the notebook runs at 300 seeds, take each seed and run it twice, changing nothing but the CPU thread count. Thread count does not change the model, the data, or the initialization. It only changes the order in which floating-point numbers get summed, and floating-point addition is not associative, so the low bits differ. That is the entire perturbation.
common = sorted(set(t1) & set(t4)) # same seeds, 1 vs 4 threads
gain = sum(1 for s in common if (not t1[s]) and t4[s]) # 1 -> 4 turns grokking ON
loss = sum(1 for s in common if t1[s] and (not t4[s])) # 1 -> 4 turns grokking OFF
flips = gain + loss # 49
The aggregate rates barely move: 74/300 (24.7 percent) at one thread, 81/300 (27.0 percent) at four. But underneath that near-equality, 49 of 300 individual seeds (16.3 percent) change their verdict: 28 seeds that did not grok now do, and 21 that did now do not. Whether a given seed "groks" is, at this scale, partly a property of your BLAS reduction order. As a sanity check that this is genuinely about reduction order and not thread count as such, 4-thread and 16-thread runs come out bit-identical, so it is specifically the 1-versus-4 change that jitters the outcome.
Are the flips biased? An exact test
A minority of flips could still hide a bias: maybe more threads quietly helps, or hurts. The paper states this should be symmetric but does not test it, so the notebook does, with an exact two-sided McNemar test on the 49 discordant pairs.
n, k = flips, min(gain, loss) # 49, 21
p = min(1.0, 2 * sum(comb(n, i) for i in range(k + 1)) / 2**n) # 0.39
With 28 gains against 21 losses, p = 0.39. No significant directional bias: more threads neither systematically helps nor hurts grokking. The flips are symmetric numerical noise, not a hidden hyperparameter, which is what upgrades "should be unbiased" from a claim into a checked result.
The lesson
Put the two findings together and the usual way of reporting grokking looks fragile. A single-seed grokking result is a lucky draw from a distribution that is biased by hyperparameters (the inverted-U) and jittered by arithmetic (the thread flips). "Same seed" is not even a guarantee of the same outcome once a benign numerical change is in play. The honest unit of measurement is a multi-seed grok-rate with an interval attached, not one clean loss curve.
This recompute reads the authors' released per-seed outcomes rather than retraining, and it says so; a forker can re-run the training from scratch on the released checkpoint. What it does not do is explain the mechanism, why reduction order should tip a network across the generalization boundary at all, which is the genuinely interesting question this leaves open.
CPU-only and one-click forkable. Run it yourself: the notebook on Kaggle. Paper under test: Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters, arXiv:2607.05104; recomputed from the authors' released per-seed records, pinned at commit d68a0a7, with every headline number re-checked at runtime.