The Quality Gate I Said Never Fired
My AI content system had two automatic quality checks. I audited them, claimed one had never fired, and was wrong. The lesson I took away was about how I had checked, more than about what I found.

Key Results
- 38 of 38 scored drafts passed the 9 of 14 bar, with a mean self-score of 12.92
- 1 voice-check block in 87 checked drafts, after I had first reported 0
- 2 top-scoring posts (14 of 14) got 466 and 350 impressions, against 460 for an 11 of 14 post
- 0 comments and 0 reposts for 2 weeks in a row while scores stayed near the top
- 15 checks now re-measured every week, with a one-line result sent to my phone
What was broken
My content system uses AI to write social posts every day, and two checks were meant to stop weak work from going out.
The first was a voice check. It blocks banned phrases and dashes, and when a draft fails, the draft is renamed as failed, set aside and written again. The second was a score out of 14: any draft scoring under 9 was written again, up to twice.
On paper this was safe. In August I audited it by looking at 143 drafts, of which 38 carried a score. Every one of those 38 scored 11 or more, the average was 12.92 of 14, and not one came anywhere near the bar of 9.
That looked too good, and there was a reason for it. The same model that wrote each draft also scored it, and nothing outside the model ever checked the score.
So I compared the scores with real results, and only 4 posts could be matched. The two posts that scored 14 of 14 got 466 and 350 impressions, while the post that scored 11 got 460, so a higher score didn't mean a better post. For two weeks in a row the posts got zero comments and zero reposts, yet the part of the score that rates how strongly a post invites a reply sat near the top.
I treated this finding with care, because four posts is a small sample and reach had also fallen across the whole platform. Zero comments, though, needs no adjustment for either of those.
What I built
My first write-up of the finding carried a strong headline, that the gates had never fired. The proof I offered was a single line saying that no failed draft had ever existed.
That same day, I found one. A draft from early June had contained one banned phrase, and the check had caught it: the draft was set aside, marked never to post and written again, and the clean version went out on three channels. The check had worked exactly as designed.
So the true number was 1 in 87, and I had reported 0. That changes the job, because a missing check needs building, while a working check that fires once in 87 drafts needs its rules looked at.
The wrong claim had survived five audits. It said that something didn't exist, and nobody had run the search that would have shown it did, a search that takes about two seconds. So I made a new rule: no check may record a zero without also recording the exact search that produced it.
Then I built a weekly job that re-measures the system from its own files every Sunday morning. Its last act is to send one line to my phone saying whether any number moved the wrong way. The first run had 13 checks and by the next day it had 15, of which 14 measure something real. The fifteenth, which compares scores against real results, is marked as not measured, because the reach data isn't on disk yet, and it never shows as passing.
I also chose not to switch anything off, even though some items were broken. My ruling was to build the record first and fix things after, because without a live record the next five problems would stay just as invisible as this one had.
What changed
The false claim is now corrected, and the old zero stays on file as a known mistake.
The audit also showed that two of the five voice rules had never been built at all. A check that tests two rules while naming the full set claims more than it actually checks, so that gap is now a tracked item instead of a hidden one.
Every number in the weekly job now carries the search behind it, so when a number moves, I can see why it moved.
What I would do differently
I'd never again let the writer grade its own work without an outside check, because a score with nothing to test it against is only an opinion. I'd also link every score to a real result from the first day.
The scoring sheet even said when it should be refitted, after four more weeks of results. Seven weeks later nobody had refitted it, and nothing read the results log except to add new rows to it.
And I'd distrust my own zeros. A claim that something never happened is the easiest kind of claim to get wrong, and also the cheapest one to test.
Want a similar result?
Let's talk about installing the right operating system for your business.
Book a Call