Measuring AI Assistant Recommendations in Production

Here are the numbers from the last measurement window across seven production sites: Gemini mentioned my brands 13 times out of 32 relevant prompts and recommended them 14 times. Claude mentioned them 26 times out of 94 prompts, and recommended them 3 times. Same brands, same prompts, wildly different behavior. That asymmetry is the whole reason I stopped treating "AI visibility" as a single metric.

I run this measurement in production, not as a survey. What follows is the setup I actually use, the failure that taught me to distrust any single dashboard, and the discipline I now apply to AI recommendation tracking.

What the mention/recommendation split actually means

A mention is not a recommendation, and the ratio between the two is the signal. Claude mentioned my brands 26 times but recommended them only 3 times. Gemini mentioned them 13 times and recommended them 14 (recommendations can exceed mentions because a single response often recommends a brand multiple times in different framings). The operational read: Claude is willing to name us in context but rarely puts us on the shortlist. Gemini names us less often but is far more willing to point the user at us when it does.

If I only tracked "mentions," I would conclude Claude is my strongest channel by 2x. That would be wrong. Recommendation is the conversion event. Mention is exposure. Confusing the two is the same category error as celebrating impressions while conversions are flat.

The other thing this split forces you to admit: each model has its own bias, its own retrieval, its own tolerance for naming specific vendors. Aggregating them into a single "AI visibility score" throws away the only information that matters, which is per-engine behavior. I keep them separate. Always.

The measurement stack

Here is what I actually run, component by component. Nothing here is theoretical. The numbers below are from my current production window.

Content substrate

• 1,106 articles published across 7 sites
• 562 of those published in the last 30 days
• Multi-agent content pipeline handles research, drafting, on-page optimization, and publishing

Indexation truth (Google Search Console URL Inspection API)

Indexation is uneven and it matters, because if a page is not indexed, it cannot be retrieved by anything downstream, including AI grounding.

Two sites, same pipeline, wildly different indexation. That gap is not a rounding error. It shifts which URLs are actually reachable to any retrieval system, human or model. I pull this daily via the URL Inspection API and treat "indexed" as the denominator for every downstream metric.

Traditional search performance (GSC 28-day)

• Current window: 1,081 clicks / 50,965 impressions
• Prior window: 538 clicks / 13,256 impressions

Search still moves. That doubling in clicks and quadrupling in impressions is happening in parallel to the AI visibility work. I mention this because a lot of people frame AI visibility as a replacement for SEO. It is not. It is an additional channel with its own retrieval mechanics, its own citations, and its own failure modes.

Downstream behavior (PostHog via HogQL)

HogQL lets me write SQL against event data, so I can join AI-referred sessions (identified via referrer patterns, UTM tagging on assistant-friendly URLs, and heuristics on landing behavior) with actual product events: signups, activation, first successful action. Without this join, the whole exercise is vanity.

Provider-level signals (Supabase Management API)

This is the part most measurement setups skip and it is where I got burned. More on that in a moment. The Management API lets me pull auth provider metadata, project-level configuration, and the raw account tables in a way that dashboards do not summarize away.

AI assistant probing

For Gemini and Claude I run a small harness that submits a fixed set of prompts (query intents drawn from GSC and Reddit for the relevant verticals) and parses responses for brand mentions and recommendations. Mentions are counted per response. Recommendations are counted when the brand appears in a "you should try" / "I recommend" / ordered-list-as-answer structure. Two different regex classes, two different columns. I never collapse them.

The 17-day failure that taught me to distrust surface metrics

This is a Supabase story but the lesson is general. I ship it here because it is the single clearest example I have of why aggregate metrics lie.

I had a Supabase Auth Hook configured to run on every new signup. Standard Webhooks signature, standard setup. The hook was deployed as an Edge Function, and Supabase's function config had verify_jwt=true set at the platform level.

The problem: verify_jwt=true rejected the Standard Webhooks signature with a 401 before my function code ran. The signature is not a JWT. The platform did not know that, and did not need to know that, because I had left the default in place. Every email signup fired the hook, hit the 401, and the whole registration flow failed.

Email registration success rate: 0%. For 17 days.

Here is what made it survive so long. Google OAuth signups bypassed the hook entirely because of a different code path. Those kept working. The signup dashboard kept ticking up. Impressions were up. Traffic was up. GSC looked great. The only broken thing was email registration, and there was nothing on the top-level dashboards that would tell me it was broken. The aggregate was healthy because Google OAuth carried it.

I found it by accident, while investigating something else. It was, frankly, embarrassing.

The detector I built after

The permanent fix is not "check dashboards more carefully." The permanent fix is a detector that counts registrations by provider directly from the database and alarms when a provider has attempt volume but zero successful accounts.

-- Per-provider health, last 24h
select
  raw_app_meta_data->>'provider' as provider,
  count(*) filter (where created_at > now() - interval '24 hours') as accounts_24h,
  count(*) filter (where created_at > now() - interval '7 days')   as accounts_7d
from auth.users
group by provider
order by accounts_24h desc;

Then, separately, I count signup attempts from application logs. If attempts > 0 and accounts_24h = 0 for a given provider, page me. Not "email the ops channel." Page. Because a provider going to zero while attempts continue is not a soft signal, it is a dead funnel.

The general form of this rule is what I now apply to every metric that could hide a broken sub-funnel behind a healthy total:

If a metric is the sum of independently-failing sub-metrics, alarm on each sub-metric, not the sum.

Why this applies directly to AI recommendation measurement

Go back to the opening numbers. Claude 26 mentions / 3 recommendations. Gemini 13 mentions / 14 recommendations.

Aggregate "AI mentions" across both engines: 39. Aggregate "AI recommendations": 17. Both numbers look fine. Both numbers grew from the prior window. If those are the only two numbers on a dashboard, everything is up and to the right.

But if Claude's recommendation rate collapses to 0 next week and Gemini's holds, the aggregate "recommendations" number will still look okay, and I will have missed the real event: one of the two major answer engines stopped shortlisting my brands. That is a strategic problem, not a metric problem. Aggregation hides it exactly the way "total signups" hid my dead email funnel.

So I run AI visibility measurement with the same discipline the auth failure forced on me:

1. Split mention from recommendation. They are different events. Log them into separate columns.
2. Measure per engine, always. Never aggregate Gemini and Claude into a single "AI" number for anything except a headline chart. Alarms live at the engine level.
3. Verify against ground truth in your own database. If AI referral traffic is up but registrations attributed to it are flat, the click-through is not landing. If mentions are up but recommendations are flat, exposure is not converting. Both are failures dressed as growth.
4. Watch attempt-with-zero-outcome patterns. In auth, this is "provider had attempts but zero accounts." In AI visibility, this is "engine mentioned the brand but never recommended it across a full measurement window" or "prompts referenced our category but our brand appeared zero times." Both mean a specific channel died while the aggregate hid it.
5. Treat indexation as the denominator. If bizflowai.io is at 66% indexed, my ceiling for AI retrieval on that site is 66%, not 100%. Any measurement that ignores this will overstate what the model "could have" found.

The measurement schema I actually store

To make this concrete, here is roughly the shape of the daily rollup table I write to.

create table ai_visibility_daily (
  day date,
  engine text,                      -- 'claude' | 'gemini'
  brand text,
  prompt_count int,                 -- prompts submitted
  mention_count int,                -- brand named anywhere in response
  recommendation_count int,         -- brand shortlisted / advised
  competitor_recommendation_count int,
  indexed_urls_at_measurement int,  -- ground truth from GSC
  primary key (day, engine, brand)
);

Recommendation rate is recommendation_count / prompt_count, not recommendation_count / mention_count. I care about the rate over the full opportunity, not just over the responses where we happened to be named. That framing makes it obvious when an engine has stopped shortlisting us even if it still mentions us.

What I'd do if I were setting this up from zero

• Start with the per-engine, per-brand, per-day rollup above. Do not build a dashboard first. Build the storage first.
• Wire GSC URL Inspection API in on day one. Indexation is not optional context; it is the denominator.
• Pipe database-level signals (signups by provider, activation by cohort) into the same warehouse as your AI visibility data. The join is where the real answers live.
• Set alarms on the sub-metrics that can independently fail. Per engine, per provider, per site. Aggregates are for stakeholders, not for operations.
• Do not build an "AI visibility score." It compresses out the information you need to act.

The pattern to internalize, and this is really the whole post: any healthy-looking total that is the sum of independently-failing components is lying to you until proven otherwise. The Supabase auth failure taught me that with 17 days of zero email signups hidden behind an OAuth-inflated total. AI recommendation measurement has the exact same shape. Two engines, two behaviors, two failure modes, one seductive aggregate that will make you comfortable while one of them quietly stops recommending you.

If you are building measurement for AI visibility, or you suspect your current setup is showing you a comforting aggregate instead of the truth, come say hi at lazar-milicevic.com/#contact. There are more field reports on the blog if you want to keep reading.