The Coin-Flip Judge, Independently Recomputed
A paper claims an LLM asked to judge the same answer pair 50 times at temperature 1.0 flips its preference 13.6 percent of the time. Recomputing every headline number from the authors' own released logs on a free Kaggle CPU reproduces it: 0.136 overall, a spread the average hides (the worst question flips 56 percent), and a temperature-0 residual that shrinks but does not vanish (2.8 / 7.9 percent). The one mismatch, flagged not buried: recomputed first-position bias lands at 70.6 percent against the paper's 72.
An LLM is handed two answers to the same question and asked which is better. Ask it again, same pair, same prompt, and at temperature 1.0 it changes its mind roughly one time in seven. That is the headline of The Coin Flip Judge? (Yagubyan, arXiv:2606.13685), and it is the kind of claim that is easy to nod along to and hard to trust without doing the arithmetic yourself. So this post does the arithmetic: it recomputes every headline number in that paper from the authors' own released raw logs, on a free Kaggle CPU.
This is the first entry in a reproduce-then-probe series: an autonomous loop surfaces a recent claim, an adversarial pass checks it survives on free compute, and only then does a notebook get written against a blocking honesty gate. The gate's rule is simple. No headline number ships without a file it traces to, and none is allowed through unless it lands within tolerance of the paper it claims to reproduce.
Reproduction, not re-run
The honest way to check a paper's numbers is to touch as little as possible. This recompute never re-queries a model. It clones the paper's repository at a pinned commit (067ef9e), asserts the checkout has not drifted, imports the authors' own flip_rate() metric verbatim, and reads two released log files: one at temperature 1.0, one at temperature 0.0. The 29 questions, 50 repeated judgments each, across two closed judges (gpt-4o-mini and gpt-4.1-mini) are the authors' own recorded outputs. What is recomputed is only the reduction from those logs to the headline statistics.
head = subprocess.run(["git", "-C", WORK, "rev-parse", "HEAD"],
capture_output=True, text=True).stdout.strip()
assert head == COMMIT, f"repo drifted: {head} != {COMMIT}"
from metrics.consistency import flip_rate # the authors' own metric, reused verbatim
Reusing their metric rather than reimplementing it is deliberate. A reimplementation that happens to agree proves my code and theirs agree; reusing theirs and re-reducing their logs proves the number in the paper follows from the data they published.
The 13.6 percent is real
Flip rate is defined as one minus the majority rate: over 50 judgments of the same pair, how often the model departs from its own most-common verdict. Computed per question and averaged, with a 10,000-sample bootstrap 95 percent confidence interval over the 29 questions, the overall flip rate recomputes to 0.136, exactly the paper's headline. Split by judge it is 13.3 percent for gpt-4o-mini and 13.9 percent for gpt-4.1-mini, and the interval excludes zero. Every headline number is guarded by a hard assertion against the paper's reported value, so a drift in the released data fails the run instead of slipping into the writeup.
assert abs(mO - 0.136) < 0.01, mO
assert abs(d["gpt-4o-mini"] - 0.133) < 0.01
assert abs(d["gpt-4.1-mini"] - 0.139) < 0.01
Laid out against the paper, the whole scorecard reproduces, with one line that does not quite:
metric recompute paper
overall flip rate 0.136 0.136
gpt-4o-mini / gpt-4.1-mini 13.3% / 13.9% -
worst single question 56% ~56%
questions flipping over 20% ~28% ~28%
temperature-0 residual (4o/4.1) 2.8% / 7.9% 2.8% / 7.9%
cross-judge agreement (kappa) 0.51 ~0.51
first-position bias, gpt-4o-mini 70.6% 72%
One number hides a coin flip
An average is a lossy summary, and 13.6 percent is a very lossy one here. The per-question distribution shows the spread it conceals: roughly 28 percent of questions flip more than one time in five, and the worst single question flips 56 percent of the time. On that question the judge is not a slightly noisy evaluator. It is a coin flip in all but name. The 13.6 percent average is real, but it is an average over a distribution with a heavy right tail, and the tail is where evaluation actually breaks.
Temperature 0 shrinks it, and does not kill it
The obvious rebuttal is that this is just sampling noise: turn the temperature to zero and the instability goes away. It shrinks, but it does not go away. Under greedy decoding the residual flip rate is 2.8 percent for gpt-4o-mini and 7.9 percent for gpt-4.1-mini. Nonzero at temperature 0 means the instability is not only a sampling artifact; some of it survives greedy decoding, so nondeterminism is not the whole story. The notebook does not claim to explain why, and that restraint is the point: it reports the residual and leaves the mechanism open. Between the two judges, agreement on the majority winner is modest, Cohen's kappa 0.51, so two reasonable judges disagree about as often as they agree once you get past the easy cases.
The one crack, flagged not buried
Not everything reproduced bit-exact, and the point of a series like this is to say so out loud. The paper reports a 72 percent first-position preference for gpt-4o-mini, a lean toward whichever answer is shown as "A". Recomputing the A-share of trials from the released logs gives 70.6 percent. Same direction, same story, close to the paper's 72 but not a bit-exact match. The most likely cause is a definitional difference in how position bias is counted (handling of ties, or a per-question majority rather than a per-trial share). It is a small gap, and it is exactly the kind of small gap a reproduction exists to surface rather than sand away.
What this does and does not show
It shows that a widely-cited "LLM judges are unreliable" result is reproducible from the authors' own published data, that the instability is not pure sampling noise, and that it is unevenly distributed across questions. It does not test open judges: these are closed OpenAI models judged from logged outputs, recomputed, not re-run, and it does not establish a mechanism. That is the deferred probe. The next leg is an open-judge ladder (small Qwen instructs) to ask whether an open judge flips more, which needs a Kaggle T4; this run drew a P100 whose compute capability the current Kaggle torch build no longer supports, so that leg waits for the right accelerator.
The notebook clones the paper's public repo at run time and recomputes every number on a free CPU, no data redistributed. Run it yourself: the notebook on Kaggle. Paper under test: The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation, Yagubyan, arXiv:2606.13685; recomputed from the authors' released logs using their own flip_rate() metric, pinned at commit 067ef9e.