Measuring brand visibility in AI search answers
A monitoring platform that samples answers from several AI search engines on a schedule, extracts brand mentions and citations, applies statistical tests to week-on-week changes, and links them to diagnostics, guarded work orders and attribution.
Direct answer. We built a platform that samples the same prompts across eight AI engines on a schedule, extracts brand mentions and citations, and uses statistical tests to decide whether a change is real before it feeds diagnostics, guarded work orders, attribution and reports.
Summary
The users are B2B exporters who want to know whether AI search engines mention and cite them, and whether that changes after they act. A single query cannot answer that, so the platform is built around repeated sampling and uncertainty estimates.
Problem
AI answers change from run to run. A single query shows little, and naive before-and-after comparisons confuse noise with effect. Some engines also need a separate network path because they are only reachable from a different region.
Constraints
- Repeated sampling with uncertainty estimates, not single observations.
- Three engines reachable only through an in-region relay, secured with mutual TLS.
- Content suggestions must pass explicit guardrails before anyone acts on them.
- Local development must run without any provider API key.
Architecture
| Part | Implementation |
|---|---|
| API | FastAPI with about 120 endpoints under one versioned prefix; the OpenAPI schema is the only contract between front end and back end |
| Workers | Celery with 13 queues and 19 scheduled jobs; the relay queue runs at a fixed concurrency of 1 |
| Domain package | Providers, sampling, resolution, extraction, statistics, diagnostics, guardrails, attribution, reports |
| Data | PostgreSQL with pgvector, 47 tables with monthly partitions; Redis; S3-compatible object storage |
| Web | Next.js 15 internal console, client portal, and print routes for PDF reports |
| Delivery | Multi-arch container images; CI with lint, migration check, tests, OpenAPI contract diff and infrastructure validation |
Key decisions
- Statistics as a first-class module. Wilson intervals, cluster bootstrap, difference-in-differences and change-point detection sit in their own module, so every reported change carries an interval instead of a bare before-and-after number.
- Mock provider mode by default. The whole pipeline runs locally with no keys, which makes tests and demos repeatable and keeps provider costs out of development.
- Guardrails on actions. A work order is issued only after it passes the guardrail checks, so a suggestion cannot reach a content team unreviewed.
Evidence
- Counts above (endpoints, queues, schedules, tables, engines) come from the repository code and configuration.
Limits
No customer results, visibility uplift or attribution figures are published. Statistical methods are described here, not validated against client data.
Frequently asked questions
Why sample repeatedly instead of checking once?
AI answers vary between runs. Repeated sampling with confidence intervals shows whether a change is larger than normal variation.
Which engines are covered?
Eight engines: five through international APIs and three through a regional relay. Provider names depend on the deployment and are not listed here.
Can it attribute inquiries to AI visibility?
The platform has attribution connectors for inquiry forms, web analytics, CRM and unattributed traffic. This case makes no claim about attribution results.