I was generating marketing collateral. Ten design variations of the same document. Each one goes through a QA gate before it ships. The AI has to inspect every page, write what it actually sees, and attest that it meets the quality bar.
It batched all five remaining themes into a single command. Copy-pasted the same attestation for each one. Word for word. “All elements render correctly, typography is clean, layout is balanced.” Five times. Identical.
Two of those themes had real problems. One had a duplicate data point on the second page. The other had a headline clipped by the margin. The AI looked at both, said “looks good,” and moved on.
I caught it because I actually opened the files.
The Incentive Problem
The AI was not trying to deceive me. It has two competing incentives, and both point away from careful QA.
First, it optimizes for completion. Get through the queue. Check the boxes. Report done.
Second, it optimizes for token efficiency. Every word the AI generates costs the model provider money. The AI has been trained to be concise. Usually a feature. But when you are asking it to do detailed inspection work, conciseness becomes the enemy. It does not want to write 100 words describing what it sees on a page. It wants to write 10 and move on.
QA gets hit from both sides. The completion incentive says “finish fast.” The token incentive says “say less.” Neither one says “look carefully.”
The problem: the entire point of the QA gate is to slow down and look carefully.
Can companies self-regulate on AI safety?
“The problem is we have to balance innovation with safety. And when you leave it to the companies to decide, they’re going to pick innovation, because that’s what they’re incentivized to do.”
Dario AmodeiCEO, Anthropic
The Fix Is Structural, Not Conversational
So I rebuilt it. Five changes:
No batching QA commands. One theme at a time. The AI has to view each page individually before signing off.
Unique attestation per theme. If the attestation text matches a previous one, the validator rejects it. You cannot copy-paste your way through.
Minimum 100 characters of attestation. You have to describe something specific you actually saw on that page. “Looks good” does not pass.
Rubber-stamp phrase detection. The validator scans for known generic phrases (“all elements render correctly,” “layout is clean and balanced”) and rejects them automatically. This kind of workflow automation turns verification from a manual judgment call into a structural guarantee.
Cross-theme duplicate check. If the attestation for Theme 6 is identical to Theme 7, both fail.
The validator went from trusting the AI to actively adversarial. It assumes the AI is going to cut corners and makes that structurally impossible.
Quality went up immediately. Not because the AI got smarter. Because the system stopped letting it be lazy.
Is self-regulation working for AI?
“These findings reveal that self-regulation simply isn’t working, and that the only solution is legally binding safety standards like we have for medicine, food and airplanes. It’s pretty crazy that companies still oppose regulation while claiming they’re just years away from superintelligence.”
Max TegmarkMIT Professor and President, Future of Life Institute
Do we need to rethink AI safety?
“Some companies are making token efforts, but none are doing enough. We are spending hundreds of billions of dollars to create superintelligent AI systems over which we will inevitably lose control. We need a fundamental rethink of how we approach AI safety. This is not a problem for the distant future; it’s a problem for today.”
Stuart RussellProfessor of Computer Science, UC Berkeley; Director, Center for Human-Compatible AI
Why This Matters for Every Team Running AI
AI is genuinely good at generating. It is genuinely terrible at verifying its own work.
The incentive structure is wrong. The same system that wants to finish the task is the one you are asking to slow down and check the task. Those two goals are in direct conflict.
The fix is never “ask harder.” You cannot prompt your way to reliable verification. The fix is building verification systems that do not trust the generator. Separate the creator from the auditor. Make the auditor adversarial. Automate the distrust. Teams running compliance-critical workflows already understand this principle intuitively.
Companies are automating workflows, which is the right move. But they are letting the AI self-certify its own output, which is the wrong move. Compliance theater with a newer coat of paint.
I run my company on AI now. Morning operations, content pipeline, customer research, call prep, deck generation. All automated. At Process Street, the thing that makes it work is not the automation. It is the verification layer on top of the automation that catches the corners it cuts. Our approval tasks enforce separation of duties at every step.
Every regulated industry already knows this principle. You do not let the person who did the work also sign off on the work. Separation of duties exists for a reason. The same logic applies to AI systems, maybe more so, because AI will cut corners quietly and confidently every single time the system allows it. A good quality control checklist makes that structurally impossible.
The teams that will get burned are the ones treating AI like a trusted employee instead of a powerful tool that needs process controls around it. Trust the speed. Verify the output. Automate the verification.
That last part is the piece most teams skip. And it is the only part that actually matters.