How we use AI to check posts before publishing

We use AI to write code, to draft posts and to check them. This post is about the last part: the gate every ProdStories post passes before it goes live, and why we let any text through it, including a draft an agent wrote in ten minutes.

TL;DR. Two skills run on every draft. Our own csn-review scores whether the post is useful and whether a reader can check it. The borrowed seo-audit helps the right reader find it. We judge a post by what the reader gets from it, so we don't ask how the draft was made. In practice, generated drafts rarely pass.

Two skills, two questions

csn-review: is this post worth reading? We built this one ourselves and described it in How we built an AI reviewer that scores our blog posts before they go live. It scores three scales from 1 to 10: Proof (did the authors do it, and can a reader check it?), Originality (what does the reader get here that the docs don't give?) and Author Authority. Before it gives any score, it opens every repo and commit the post links to. A final score of 7 or more means Publish. From 5 to 6.9 means Revise, with a list of must-fix items.

seo-audit: will the right reader find it? We didn't write this one. It comes from coreyhaines31/marketingskills. It checks the title, the meta description, the headings and the internal links. It doesn't judge whether the post is good.

A generic SEO audit gives generic advice. So before it starts, the skill reads our product-marketing.md. That file says who we write for (engineers who build their own products without a team), which topics we cover, how we write, and what we never do: no newsletter prompts, no "game-changer", no invented statistics. With that file, the audit gives advice for a developer blog with zero domain authority, which is what we are.

csn-review is the main judge, and it always runs first. If a post can't pass it, there is nothing to optimize. seo-audit only polishes the surface of a post that has already passed.

We don't ask how the draft was made

Any text can go into the gate: a draft one of us spent a weekend on, or a draft an agent wrote in ten minutes. For us, the only measure of a post is what the reader gets from it. The reader can't see the hours an author spent, and those hours don't make the post more useful.

So csn-review never asks how a draft was written. It asks other questions. Which repo is this? Does the commit exist? Where did this number come from? Is this takeaway already in the official docs?

There is a second reason. Four of us write here, and none of us is an editor. When one author reviews another author's draft, the review is partly about taste and partly about not upsetting a colleague. The reviewer agent doesn't know whose draft it is reading. Its rules are written down in a public repo. When we disagree with a score, we argue with a rule, and we can change the rule in a commit.

The agent has no favorites among us, but it is still noisy. The same post can get scores a point apart on two runs. Author Authority scores the people, so a small team with little public footprint starts low. We covered both limits in the reviewer post.

Generated drafts rarely pass

When we started, we expected the opposite problem: an agent would write drafts faster than we could review them, and the reviewer would let them through. That didn't happen. The bar is high even for posts built on our own work.

The results of our csn-review runs are in the last section of the reviewer post, How the reviewer scores our posts today. Our post about a typewriter animation in React scored 7.1 and passed. Our Google Stitch post scored 6.0 and got Revise. Both posts have a real repo and real code behind them.

The reason is in the rubric. A language model writes fluent text easily. It can't produce a commit that doesn't exist or a measurement nobody took, and those are exactly what the reviewer checks:

• Every claim about what the authors built, ran or measured goes into an evidence ledger. If none of the evidence can be verified, Proof and the final score are both capped at 4.
• Every key takeaway is checked against the official docs and the first page of search results. If the post has no takeaway of its own, Originality is capped at 3.
• The final score can't be more than 2 points above the lower of Proof and Originality.

A generic generated post hits both caps and ends in Rework. To get past the gate, a draft needs things only the author has: a repo, a dead end, a number. Once those exist, it matters much less who typed the sentences around them.

What it costs

• Tokens. One run on three posts took about 540,000 tokens and 6 minutes, because each scale runs in its own subagent.
• Noise. Scores move by up to a point between runs, so we read 6.5 and 7.0 as the same result.

Why we think this should be the standard

Readers already can't tell how a text was written, and soon nobody will. "Was this written by AI?" stops being a useful question. Two questions still work: What in this post can I check? What does it tell me that the docs don't?

A publisher can answer both before publishing, with a written rubric and an agent that applies it the same way to every draft. We think every blog that publishes AI-assisted text, which will soon be every blog, needs some version of this: a public rubric, a check before each post, and evidence a reader can open. Ours is one way to do it.

Try it

The reviewer is open source:

npx skills add captain-solver-news/skills --skill csn-review

Run it on this post and tell us where it disagrees with us.