Building an AI reviewer skill that scores developer blog posts

Every post on ProdStories is supposed to be first-hand: we write about code we shipped and tools we actually ran. Several developers write here, and none of us is an editor. We wanted a check that runs before every post goes live and tells us honestly whether the post is worth publishing.

So we built csn-review, an agent skill that scores a developer blog post before publication. This post covers how it works, what broke when we ran it, and what is still weak.

TL;DR. The reviewer never gives a score straight away. It first writes down facts: the evidence behind each claim, where each takeaway already exists in the docs, and what public sources say about each author. Only then does it score three separate scales from 1 to 10: Proof (did the authors do it, and can a reader check it?), Originality (what does the reader get here that the docs don't give?) and Author Authority (would a skeptical reader trust these authors on this topic?). Each scale runs in its own subagent, so a well-proven post can't make the reviewer generous about its originality. The skill is open source: captain-solver-news/skills.

The main idea: facts first, score second

Ask a model to rate an article from 1 to 10 and the score tends to land in a narrow band near the top of the scale. Studies of LLM judges report lenient, compressed scores (Thakur et al., Reference-Free Rating of LLM Responses). The number also tracks how the text sounds: in Wu & Aji, answers with factual errors were rated higher than short answers or answers with grammar mistakes. So a clean, confident post about a tool nobody ran can score as well as a messy post with a real repo behind it.

We looked at existing approaches, including Expert Panel Scoring in coreyhaines31/marketingskills. It asks several expert personas to score copy within their areas of expertise and suggest specific fixes. The repository also includes workflows that read product repositories and merged PRs, such as launch.

For our developer posts, we wanted a review that checks linked code, traces measurements to their sources, and compares the main takeaways with existing documentation. We built csn-review around those requirements: evidence and insight ledgers, separate scoring scales, and explicit score caps.

So our reviewer is not allowed to give a number directly. It works in a fixed order:

1. Ledger first. Before any score, it lists every claim the post makes, the evidence behind it, and whether it opened that evidence and it held up.
2. Criteria with described levels. Each criterion is scored 0 to 3, and each level is described by something you can see in the text, for example "at least one rejected alternative, with the reason". Words like "good" or "strong" never define a level.
3. A formula. Levels times weights, mapped to a 1–10 score.
4. Caps. Some problems limit the score no matter how good the rest is.
5. Two calibration lines in the instructions: "a solid, publishable first-hand post is a 7, most first drafts land at 4–6", and "torn between two levels, pick the lower".

Why three scales

Google describes content quality with E-E-A-T: Experience, Expertise, Authoritativeness, Trust. Originality isn't in the acronym, but Google's guide to helpful content asks about it directly: "Does the content provide original information, reporting, research, or analysis?" and "Does the content provide insightful analysis or interesting information that is beyond the obvious?" Our three scales roughly follow that: Proof covers Experience and Trust, Originality covers original information, and Author Authority covers the people behind the post. Each scale lives in its own file in references/.

Proof

Did the authors actually do the thing they write about, and can a reader check it?

points = Σ(level × weight), max 36
proof  = 1 + 9 × points / 36

The evidence ledger is the core of this scale. Every claim about what the authors built, ran, measured or broke, and every claim about how a tool works, gets a row with one of four statuses: verified (the reviewer opened it and it supports the claim), offered (evidence exists but can't be checked, such as a private repo), unsupported or contradicted. The reviewer opens GitHub links with gh, or with the public GitHub API when gh is missing. A link the reviewer can't open counts as offered. A post should not lose points because of how the reviewer's machine is set up.

Three rules stop the scale from rewarding a post that looks like experience:

• A practitioner detail counts only if a reader would meet it in their own project too. Local facts like our color palette or our file names prove the project exists. They don't count as practitioner knowledge.
• An alternative counts only if a competent reader would seriously consider it: a competing library, a built-in feature, a different architecture. "We could have written our own parser" doesn't count.
• "It worked on the first try" limits Failures and trade-offs to level 1.

Caps: a contradicted central claim or an unsourced statistic, like "73% of Node developers…", limits Proof to 3. No verified evidence at all limits it to 4. Both also cap the final score.

Originality

What does a reader get here that the tool's docs and the usual articles don't give?

This scale starts with an insight ledger. The reviewer lists the 3–6 key takeaways of the post and gives each one a status:

• docs: the official docs of the tool already say it. A URL and a quote are required.
• common: not in the docs, but widely covered, for example on the first page of search results. A URL is required.
• own: found in neither place. An undocumented gotcha, a docs claim that turned out wrong, a measured comparison, a number nobody else has published.

Two rules keep this honest. A takeaway without a URL is own, so the reviewer can't dismiss it with "it's somewhere on the internet". And an own measurement that only confirms the docs ("the library is heavy, we measured 500 KB") is docs with a number.

The criteria are New knowledge (weight 3), Real alternatives (2) and Point of view (1). If the post has no own takeaway, Originality is capped at 3. The reviewer also writes one line called "Why this post?": what the reader gets here and nowhere else. If the honest answer is "nothing", it suggests what to measure or compare to make the post worth writing.

Author Authority

This scale scores the people: Identity (weight 2), Experience in this topic (3), Recognition by others (2) and Publications in this topic (1). Experience is scored for the topic of the post. Ten years of backend work don't count toward a post about game shaders. An anonymous byline caps the scale at 2.

Our first idea was a default of 7 when the reviewer knows nothing about the author. We dropped it. With a default of 7, giving the reviewer real author data would usually lower the final score, and hiding it would raise it. Now bare text with no byline gets n/a, and the final score uses the other two scales.

The final score

final = mean of the scored scales
final = min(final, min(Proof, Originality) + 2)
final = min(final, any Proof cap that also caps the final score)

The second line is there because a plain mean lets one strong scale carry a failed one. Take a post with invented facts: Proof is capped at 3, Originality is 8, Authority is 7. The mean is 6.0, Revise, one edit away from publishing. With the "+2" rule it can't go above 5. The rule works the other way too: a verified, honest post that only repeats the docs can't get past Revise.

When the author sends a revision, the reviewer scores the whole draft again and shows which fix moved which criterion. If two rounds in a row don't raise the score, it says so directly: the missing part is experience, evidence or new knowledge, and rewording won't add it.

Each scale runs in its own subagent

A reviewer that has just checked every commit in a post becomes generous about its originality. So if the agent can run subagents, the skill runs three in parallel, one per scale, and none of them sees the others' scores. The Authority subagent gets only the byline and the topic of the post. Without subagents, the skill scores the scales one after another in the order Originality, Proof, Authority, and does not go back to change a score.

This costs real money. A run on three of our posts used nine subagents, about 540,000 tokens and 6 minutes. A second run took about 650,000 tokens.

What is still weak

• Scores move between runs. With the same rubric, the Stitch post got Proof 6.5 in one run and 5.8 in the next, and Originality 4.0 and then 4.5. The README says to treat a difference under one point as noise, and we mean it.
• Authority depends on what is public. LinkedIn answered every request with HTTP 999, so facts from LinkedIn stay offered. For a small team with no outside footprint, Authority sits around 5–6 and pulls every final score down.
• It is expensive. One review means three agents. Authority could be cached per author, but today it is computed again for every post.

How the reviewer scores our posts today

Reviewer: Codex (GPT-6.1-Sol, high), csn-review v0.0.3.

Reviewer: Claude Code (Opus-5.5, high), csn-review v0.0.3.

Why we still published the Google Stitch post

The Stitch post scored 6.0, below the 7 we treat as publishable. We published it anyway, for two reasons:

1. The reviewer's margin of error is around one point, and sometimes a bit more. A 6.0 and a 7.0 can come from the same post on different runs.
2. We can't raise Authority quickly, because that score is about the authors, not the post. Instead of tuning the skill's scale until our posts pass, we accept a low Authority score for now.

So for now we:

• give Proof and Originality more weight when we decide whether to publish;
• keep working on ourselves and growing as authors.

Try it

The skill is in captain-solver-news/skills. Install it with one command:

npx skills add captain-solver-news/skills --skill csn-review

Or copy the csn-review folder into your project's skills directory by hand. Then ask in plain words: "Review this draft before we publish", with links to the repo, commits or PRs and the name of the author. If the draft has no evidence, the reviewer asks for it once before scoring.

Warning

This post describes csn-review version 0.0.3. The skill is still experimental, and we keep improving it. By the time you read this, it has most likely changed a lot since publication.