Measuring brand visibility in AI search answers

A monitoring platform that samples answers from several AI search engines on a schedule, extracts brand mentions and citations, applies statistical tests to week-on-week changes, and links them to diagnostics, guarded work orders and attribution.

Direct answer. We built a platform that samples the same prompts across eight AI engines on a schedule, extracts brand mentions and citations, and uses statistical tests to decide whether a change is real before it feeds diagnostics, guarded work orders, attribution and reports.

Summary

The users are B2B exporters who want to know whether AI search engines mention and cite them, and whether that changes after they act. A single query cannot answer that, so the platform is built around repeated sampling and uncertainty estimates.

Pipeline of the AI visibility platform: prompt templates are planned into weekly sampling slots, sampled from eight AI engines (three through a mutually authenticated relay), extracted into mentions and citations, analysed statistically, then fed to diagnostics, guarded work orders, attribution and reports in a console and client portal.

Problem

AI answers change from run to run. A single query shows little, and naive before-and-after comparisons confuse noise with effect. Some engines also need a separate network path because they are only reachable from a different region.

Constraints

  • Repeated sampling with uncertainty estimates, not single observations.
  • Three engines reachable only through an in-region relay, secured with mutual TLS.
  • Content suggestions must pass explicit guardrails before anyone acts on them.
  • Local development must run without any provider API key.

Architecture

PartImplementation
APIFastAPI with about 120 endpoints under one versioned prefix; the OpenAPI schema is the only contract between front end and back end
WorkersCelery with 13 queues and 19 scheduled jobs; the relay queue runs at a fixed concurrency of 1
Domain packageProviders, sampling, resolution, extraction, statistics, diagnostics, guardrails, attribution, reports
DataPostgreSQL with pgvector, 47 tables with monthly partitions; Redis; S3-compatible object storage
WebNext.js 15 internal console, client portal, and print routes for PDF reports
DeliveryMulti-arch container images; CI with lint, migration check, tests, OpenAPI contract diff and infrastructure validation

Key decisions

  1. Statistics as a first-class module. Wilson intervals, cluster bootstrap, difference-in-differences and change-point detection sit in their own module, so every reported change carries an interval instead of a bare before-and-after number.
  2. Mock provider mode by default. The whole pipeline runs locally with no keys, which makes tests and demos repeatable and keeps provider costs out of development.
  3. Guardrails on actions. A work order is issued only after it passes the guardrail checks, so a suggestion cannot reach a content team unreviewed.

Evidence

  • Counts above (endpoints, queues, schedules, tables, engines) come from the repository code and configuration.

Limits

No customer results, visibility uplift or attribution figures are published. Statistical methods are described here, not validated against client data.

Frequently asked questions

Why sample repeatedly instead of checking once?

AI answers vary between runs. Repeated sampling with confidence intervals shows whether a change is larger than normal variation.

Which engines are covered?

Eight engines: five through international APIs and three through a regional relay. Provider names depend on the deployment and are not listed here.

Can it attribute inquiries to AI visibility?

The platform has attribution connectors for inquiry forms, web analytics, CRM and unattributed traffic. This case makes no claim about attribution results.