Shipping LLM Features: Versioned Prompts, Eval Sets and Regression Gates

Direct answer. When an LLM feature misbehaves in production, the model is rarely the root cause. More often someone edited the prompt in place, nobody validated the output format, or the model changed and nobody re-tested. We manage prompts as code, check every output against a schema and business rules before anything downstream sees it, keep an eval set per feature, and run a regression in CI whenever a prompt or model changes. If the score drops, the change does not merge.

Diagram of LLM feature delivery. The top row is the runtime path: a prompt file stored in the repository is sent through a gateway with a provider adapter, the output passes schema and rule checks, and a result or fallback is returned; a failed check retries once, then falls back. The bottom row is the change path: a prompt, model or schema change runs the eval set and a CI regression with the real model, and merges only if the baseline holds. Only the merged version runs in production.
Figure 1. Runtime path and change path. Only a prompt version that merged and passed regression runs in production.

1.0 Why it works in the demo and drifts in production

During a demo everything is in the hands of whoever built it: the prompt is in their editor, they chose the inputs, and they can tell at a glance whether the answer is good. Production removes all of that:

  • Prompts are spread across code strings, an admin panel and someone’s notes, and nobody can say which version is live.
  • Output is free text that downstream code parses as “probably JSON”, which breaks the first time the model adds a friendly sentence.
  • The provider ships a new model version and the same prompt behaves differently, and nobody notices.
  • There is no fixed set of test inputs, so “better or worse” is a matter of opinion.

2.0 Prompts live in the repository

One file per prompt, stored in the repository and reviewed with the code that calls it. The header states the purpose, input variables, expected output structure and target model. When two providers need different wording, use two files rather than conditional branches inside one template.

Runtime logs record the prompt file hash and the model identifier, so any output can be traced to the exact prompt and model that produced it.

We advise against making prompts editable in an admin panel. It looks flexible, but it bypasses review and tests and changes production logic directly. If operations staff really need to adjust something, such as tone or example text, expose it as a few bounded parameters, not the whole prompt.

3.0 Check the output before anything else uses it

LayerWhat it does
StructureRequest structured output and validate it against a JSON Schema or an equivalent type; retry once on failure, then fall back
Business rulesDeterministic code checks enum values, ranges, required fields and whether referenced IDs actually exist
SafetyA local keyword and rule layer that does not rely on the model behaving; a hit follows a fixed handling path
FallbackWhat the user gets on timeout, rate limiting or failed validation is decided in advance and covered by tests

Anything deterministic code can decide stays out of the model. If a scoring rubric has fixed deductions, a rule engine computes them and the model does what it is good at: understanding, summarising and generating. Results become reproducible, and “why did I lose points?” has an answer.

4.0 Eval sets and the regression gate

Each feature gets an eval set: typical inputs, edge cases and inputs that caused trouble before. Every item carries assertions on structure, key fields and content that must not appear. Subjective quality is graded against a rubric by a model, and a person periodically checks that the grading itself is sound.

These changes trigger a regression run in CI: the prompt, the output schema, the model or model version, and the parsing code. The threshold is written down beforehand, for example an assertion pass rate no lower than the previous baseline, and a change that misses it does not merge. Model output is non-deterministic, so running each input several times and checking consistency tells you more than a single pass.

Real calls cost money and time, so there are two layers. Everyday unit tests use a mock provider that returns deterministic data, which lets the whole pipeline run offline. The prompt regression uses the real model and runs only when prompt-related files change.

5.0 Providers and cost

  • The gateway has one adapter per provider, and business code never imports a provider SDK. Switching or adding a fallback provider touches only the adapter.
  • Pin models to a specific snapshot, not an alias such as latest. A model upgrade is a release and goes through the regression first.
  • Record token usage per feature and enforce quotas on the server, never on client-reported numbers.
  • When the system must be self-hosted, pick an inference server with an OpenAI-compatible API, such as vLLM, and use a local mock server during development so the client code does not change.

6.0 What you receive

  • The prompt files and their change history.
  • Output schemas, rule checks and fallback paths, with tests.
  • Eval sets, rubrics, the CI regression configuration and the current baseline scores.
  • The provider adapter layer and usage records.

To add AI features to a new or existing product, see Web, Cloud and Device Management. To talk through your situation, use Discuss a project.

References

Frequently asked questions

How big does an eval set need to be?

Start with the real inputs you already have plus the cases that have gone wrong before. Covering the main scenarios and the known failure modes already catches most regressions. Every production incident then adds its input to the set, so it grows with the problems instead of trying to be complete on day one.

Can we trust a model to grade another model?

With a clearly written rubric it works well for screening large batches, but the grader has its own biases. We have a person spot-check the grades regularly, and any dimension where the grader keeps disagreeing with people moves to rule-based checks or human review.

How much work is it to switch to a cheaper model?

It depends on whether the eval set exists. If it does, switching means changing the adapter configuration, running the regression and comparing scores and cost, and you can see exactly where the gap is. If it does not, building it is the first step.

Will the model provider train on our data?

That depends on the provider's data terms and the agreement you sign, and it should be checked during selection. For sensitive data, a self-hosted inference server or redaction before the request leaves your system are the usual options.