How I Stop an Autonomous AI Agent From Publishing Garbage

An autonomous agent will happily publish 562 articles in 30 days that look fine and rank for nothing. That is the actual failure mode. Not hallucinations, not tone drift, not the dramatic stuff people write threads about. The real failure is a system that keeps working, keeps logging green, keeps producing bytes, while the thing you actually care about (indexed pages that pull traffic) quietly stops happening.

This is a field report from the pipeline I run for BizFlowAI ContentStudio: 7 sites, 1106 articles across their lifetimes, one operator (me), zero human editors in the loop. Quality control here is not a prompt. It is a stack of gates plus a set of provider-level detectors that catch what the gates cannot see. And the most important lesson I learned this year came from an auth bug, not a content bug.

The stack that makes gating possible

The orchestrator is boring on purpose: Python workers, PostgreSQL 16 as the single source of truth, roughly 30 named Windows Task Scheduler jobs prefixed BizFlowCS_* (research, outline, draft, edit, SEO pass, internal-link pass, image, publish, index-ping, GSC pull, and so on). LLM calls go through an OAuth proxy in front of Claude, with GLM as failover when the proxy returns 429. Every job writes to Postgres before and after it runs.

Why this shape? Because every step is a row and every failure is queryable. That is the only property that makes gating tractable when nothing is watching in real time.

A simplified view of the run table:

create table article_runs (
  id bigserial primary key,
  site_id text not null,
  slug text not null,
  stage text not null,          -- research | outline | draft | edit | seo | publish | index
  status text not null,         -- ok | soft_fail | hard_fail | skipped
  model text,
  tokens_in int, tokens_out int,
  cost_usd numeric(10,4),
  gate_scores jsonb,            -- per-gate numeric scores
  reason text,                  -- why blocked or skipped
  created_at timestamptz default now()
);

That single table is what makes everything downstream possible. I can ask "how many drafts hit stage seo this week but never reached publish, grouped by reason?" and the answer is one query. If I only had logs, I would have opinions. With rows, I have counts.

The provider setup matters for the same reason. When Claude 429s, the proxy fails over to GLM, and that failover is a row too (model changes, reason becomes claude_429_failover). If I ever see a week where 40% of drafts came from the failover path, that is a gate signal on its own, because model quality shifts subtly and I want to know before a reader does.

The pre-publish gates

There are seven gates a draft has to pass. None of them are clever. Cleverness is a liability here.

1. Structural: H1 present and unique, at least three H2s, no orphan H3s, no empty sections, word count inside a per-site band.
2. Factual self-check: a second LLM pass with a strict prompt that flags any sentence containing a number, a named entity, or a claim; the flags go into gate_scores.factual. If more than N flags, block.
3. Duplicate/near-duplicate: pgvector cosine against the site's existing corpus, hard block above threshold, soft warning between two thresholds.
4. Internal link sanity: at least two internal links, both resolving to 200, neither pointing to the draft itself, anchor text not identical to the target's H1.
5. SEO surface: title length, meta description length, slug shape, single canonical, no accidental noindex.
6. Tone/voice: banned-phrase list per site (the usual "unlock", "game-changer", "in today's fast-paced" set), plus a small classifier for AI-tell openers.
7. Publish window: rate limit per site per day, per category per week. This is what stops a runaway job from dumping 40 posts on one domain overnight.

Each gate writes its score to gate_scores and either passes, soft-fails (auto-revise and retry once), or hard-fails (park in a review queue that I actually read).

This is the boring part. It works. It is also completely insufficient, which is the interesting part.

The failure that ran for 17 days

Gates catch what you thought to check. The dangerous failures are the ones where every gate is green and the metric you glance at is green and the actual outcome is dead. Here is the one that taught me the lesson, and it did not come from the content pipeline at all. It came from auth.

The setup: Supabase project, an Edge Function acting as a Supabase Auth Hook to run custom logic on registration. Standard pattern. The function was deployed with verify_jwt=true, which is the default and, in almost every other context, the right choice.

What I did not internalize: GoTrue does not sign Auth Hook calls with a Supabase JWT. It signs them as Standard Webhooks, with an HMAC signature in the webhook-signature header. The Edge Functions gateway checks for a Supabase JWT first, does not find one, and returns 401 before my function code ever runs. So every email/password registration attempt was failing at the gateway, silently, with a 401 the user never saw as "server misconfigured" and I never saw as "hook broken" because the hook logs were empty. Empty logs, because the code never ran.

Meanwhile, Google OAuth signups bypass the hook entirely (different flow, different plumbing). So auth.users kept growing. Total user count on the dashboard went up every day. The number I glanced at was fine.

The Supabase dashboard's event log did not surface "Registration Failed" in the synced view I was watching. So for 17 days, roughly half the signup funnel was returning 401 to real humans, and the top-line metric was healthier than ever because Google OAuth kept climbing and the dead half was invisible.

I found it by accident: a friend tried to sign up with email, told me it did not work, I assumed user error, he sent a screenshot. Then I looked at the gateway logs directly (not the function logs, the gateway logs) and there it was, 401 after 401 after 401, going back more than two weeks.

The money paragraph, said plainly: the metric was correct and the system was broken. The metric was measuring the wrong thing. It was measuring "did a row appear in auth.users" when what I cared about was "did every attempted signup either succeed or fail loudly". Those are not the same question, and I had been treating them as if they were.

Fix was one line: verify_jwt=false on that specific Edge Function, because the Auth Hook has its own signature verification I do inside the function using the webhook secret. But the fix is not the point. The point is what replaces the broken dashboard.

The detector that replaced trust in dashboards

After that, I stopped trusting any UI that syncs from somewhere else. The new rule: for anything that matters, count the ground truth directly, from a place that cannot be filtered, cached, or ad-blocked.

For auth, that means a scheduled job that counts registrations per provider, straight from the database, and alarms when the ratio moves:

select
  raw_app_meta_data->>'provider' as provider,
  count(*) as accounts_created,
  count(*) filter (where created_at > now() - interval '24 hours') as last_24h
from auth.users
group by 1
order by 2 desc;

Then a second query against whatever attempt log you have (gateway logs shipped into Postgres, in my case). Alarm condition: attempts exist for a provider, zero accounts materialized for that provider in the same window. That would have caught the 17-day bug on day one.

The same principle applies to the content pipeline, and this is where it changed how I evaluate quality.

The intermediate signals (draft passed all gates, publish returned 200, sitemap ping accepted, article visible on the live URL) are all green for basically everything. The ground truth I actually care about is is Google indexing this URL. So I run the GSC URL Inspection API against every published URL and store the verdict.

The numbers this surfaces are uneven in ways the intermediate signals never showed:

Every article on stylera.io passed the same seven gates as every article on fakturko.io. Same pipeline, same models, same quality checks. But 71% of stylera's output is sitting in "Discovered, not indexed" or "Crawled, not indexed" purgatory, and the dashboard I would have been tempted to build ("articles published this week: 42, all green") would have told me nothing.

The detector, not the dashboard, is what told me stylera needs a different intervention (topical authority, internal link density, probably some ruthless pruning) before more publishing helps at all. On that site, the correct action from the data is to stop publishing until indexation catches up. An autonomous system without that detector would just keep going.

Generalize the pattern:

• Auth: do not trust "total users up and to the right". Count per provider, alarm on zero-with-attempts.
• Content: do not trust "published: ok". Count indexed URLs via URL Inspection API, alarm on ratio drift.
• LLM pipeline: do not trust "call returned 200". Count tokens, cost, and failover rate per model per stage, alarm on failover spikes.
• Payments: do not trust "checkout completed" from the client. Count settled charges from the provider webhook, alarm on client-vs-provider gap.

In every case the pattern is identical. The intermediate signal is convenient and lies to you politely. The ground truth is annoying to fetch and tells you the truth.

What I'd do if I were starting this pipeline today

1. Build the run table before the first worker. Not after. Every stage writes a row. If a stage cannot express itself as a row, the design is wrong.
2. Ship the seven gates on day one, even if three of them are stubs that always return pass. The scaffolding is what matters; the thresholds tune themselves once real drafts flow.
3. For every external system you depend on (auth, payments, email, indexation, analytics), write a ground-truth detector before you write a dashboard. The dashboard is optional. The detector is not.
4. Alarms should be phrased as questions, not thresholds. "Are there provider attempts with zero account creations?" is a better alarm than "user growth < 5%". The first is a logic gate. The second is a mood.
5. Assume one dependency you have not verified is currently broken. It usually is. Go look.

Close

The 28-day window that finished this month ran 1081 clicks and 50965 impressions, against 538 and 13256 for the prior 28. That is the number I would put on a slide if I made slides. It is not the point of this post. The point is that the system tells on itself now, and when it starts lying again (which it will, in some new and creative way) I will find out from a query, not from a customer.

If you are building something autonomous and the "how do I know it is actually working" question is keeping you up, I am happy to talk shop: lazar-milicevic.com/#contact. More field notes on the blog as I write them.