How noisy is an LLM reviewer? We scored one post 30 times

In the post about csn-review we wrote that the reviewer's margin of error is "around one point, and sometimes a bit more". That number was an intuitive assumption. We based it on several repeated runs and on the fact that the Stitch post got different scores. So we measured it properly.

TL;DR. We scored the same article 30 times on each scale with csn-review v0.0.3. Proof came out at 6.81 with a standard deviation of 0.084. Originality came out at 4.43 with a standard deviation of 0.68, eight times wider. "Around one point" overstated the noise of Proof and understated the noise of Originality.

How we measured

• Skill: csn-review version 0.0.3, the same version as in the first post.
• Reviewer: Claude Code (Opus-5.5, high).
• Input: one randomly generated article. Every run scored the same text.
• Three launches. Each launch produced 10 Proof scores and 10 Originality scores, so we have 30 scores per scale.
• We only measured Proof and Originality. Author Authority scores the byline, and a generated article has no real author behind it.

Proof scores

Each column is one launch, each row is one of the ten scores in that launch.

26 of 30 scores are exactly 6.8. Three scores are 7.0 and one is 6.5. Launch 2 returned 6.8 ten times out of ten. The whole range is half a point.

Proof is computed from criterion levels (0 to 3) multiplied by weights, so the score can only move in steps. Each level is tied to something the reviewer can see in the text: a link, a commit, a rejected alternative with a reason. Our guess is that this is why the reviewer picks the same levels almost every time.

Originality scores

Same layout: three launches, ten scores each.

20 of 30 scores are 4.0, and the rest spread up to 6.5. Launch 1 alone covers the whole range from 4.0 to 6.5. The launch averages also differ: 4.85, 4.35 and 4.10. With ten scores per launch we can't tell yet whether that is a real drift between launches or chance.

Originality starts with the insight ledger: for each takeaway the reviewer searches the docs and the web and labels it docs, common or own. Different searches find different pages, so the same takeaway can be own in one run and common in the next. That is our working explanation. We have not yet compared the ledgers run by run to confirm it.

Standard error of the mean

For each scale we computed the mean, the sample standard deviation and the standard error of the mean, with n = 30:

The two numbers answer different questions:

• s tells how far a single review can land from the average. This is the one that matters in practice, because before publishing we run the reviewer once.
• SE tells how well we know the average itself after 30 runs. With 95% confidence the true mean is 6.81 ± 0.03 for Proof and 4.43 ± 0.25 for Originality.

The Gaussian curves

Here are normal curves built from the measured mean and standard deviation of each scale, on the same axis:

The Proof curve is a narrow spike. The Originality curve is low and wide. Below is the Originality curve alone, with the old "one point" margin around the mean shaded:

Under a normal model, about 86% of Originality scores land within one point of the mean, so roughly one review in seven lands further away. Our data agrees: 4 of 30 scores are more than one point above the mean. For Proof, one point is almost 12 standard deviations, so the margin is far wider than the noise.

The normal curve is a simplification. The real scores are discrete: Proof took only three different values. Originality is also skewed: most scores sit at 4.0, with a tail to the right. The curves show the size of the noise, not its exact shape.

What this means for the "one point" rule

The rule from the first post was a rough guess, and one number can't describe two scales this different. When we compare two single reviews of the same text, the difference between them has a standard deviation of s × √2:

• Proof: s_Δ ≈ 0.12. A difference above about 0.25 between two Proof scores is probably real.
• Originality: s_Δ ≈ 0.96. Two Originality scores need to differ by almost 2 points before we can call the difference real.

Originality noise also reaches the final score, because the final score is the mean of the scales. If a post sits near the publish threshold, it makes sense to run Originality several times and average: with four runs the standard error drops to 0.68 / √4 ≈ 0.34.

One warning about Proof. In the first post, the Stitch post got 6.5 and 5.8 on Proof from different agents, a gap this data says should almost never happen within one agent. So the Proof score probably depends on the coding agent (Claude Code or Codex). We measured only Claude Code, on one article and one version of the skill, so this is still a hypothesis.

What we plan next

We will keep improving the skill, and the scales are the main target. We want them to be:

• Cheaper. One review already costs hundreds of thousands of tokens. Measuring noise means running the same review dozens of times, so every token we cut from a scale makes the measurement cheaper too.
• More deterministic. Originality is the first candidate. The insight ledger needs a more repeatable way to find and label sources.
• More accurate. A scale can be stable and still wrong: it can return 6.8 every time for a post that deserves 5. We want the scores to match what a careful human reviewer would say.

We will repeat this measurement for future versions of csn-review, and on real posts with evidence, so each release comes with its own error bars.