Certified answers & certified definitions

Two artifacts, easily confused, closing two different gaps.

A certified definition is a per-metric markdown page (lenses/<name>/certified/*.md): definition, grain, sources, canonical SQL, a probed verified value (services/certdefs/). It closes the decision gap — which meaning, which table, which filter counts as this metric here.

A certified answer is a human-vouched (question, SQL) pair with provenance — source, verified_by, a probed verified_value, status: active | retired (lenses/<name>/certified_answers.yaml). It closes the execution gap — the exact query a human verified, served instead of regenerated.

Matching is vector similarity, so active answers are always embedded. One certified without an embedding provider configured lands in a third, derived status — pending_embedding — which dst apply and dst test both name until dst reindex embeds it and promotes it back. It is never silently unmatchable.

Knowing isn't doing

A model with the right definition in context can still compute it wrong. The decision and the execution fail independently, which is why both artifacts exist — a well-written definition alone is not a correctness guarantee; the executable exemplar is what closes the second gap.

How certified answers serve

Three similarity bands against the active library (services/runtime/assembly.py:54):

Band	Behavior
≥ 0.95	serve the approved SQL verbatim, certification="certified", zero generation
0.90–0.95	a cheap paraphrase gate asks "would the SAME SQL answer both?" — yes promotes to certified; any provider error fails closed
≥ 0.83	fold hits into the prompt as few-shot exemplars, certification="assisted"

The caller always sees which band applied; certified responses carry who certified the pair and when (services/contracts/response.py).

Templates: one entry for a question family

A certified answer can carry {slot} placeholders in both question and SQL, so a family of questions that differ only in a literal is one entry rather than N frozen pairs (services/certify/binding.py):

- question: revenue in {period}
  sql: >
    SELECT sum(amount) FROM orders
    WHERE order_date >= {period.start} AND order_date < {period.end}
  slots:
    period:
      type: date_range          # date_range | date | enum | number
  sample_bindings:
    - period: 2026-Q2           # non-empty = testable

Everything about a template is deterministic. Slots are typed (date_range, date, enum, number; enum values declared inline), the date_range grammar is tiny and canonical (YYYY, YYYY-Qn, YYYY-MM, or an explicit start/end pair) and ranges are half-open — {slot.start} inclusive, {slot.end} exclusive. The LLM bind gate only proposes values for a matched question; nothing it says reaches SQL except through the validator and a sqlglot-rendered dialect-correct literal — never string-splicing. A template serve composes no prose either: the answer text is a deterministic frame rendered by code (services/runtime/answer.py:200).

sample_bindings must be non-empty for a template to be testable: the first binding is both the match anchor and the eval witness (services/evals/certified_suite.py:56). A template match reports certified_match: "parameterized" on the response, alongside exact and equivalent (services/contracts/response.py:143).

Where certified answers come from

Vouched from a review ruling, authored or BI-imported in certified_answers.yaml and landed by dst apply (agent-first — dst owns the slots and gates, not N brittle per-BI importers), generated per governed definition for human review, or POSTed directly — gated identically either way: the SQL must parse, pass shape checks, and touch no tables outside the lens boundary (services/api/mgmt_certify.py:145). The doctrine underneath all four paths: never auto-certify — the human approval is the trust.

The corpus is the regression suite

dst test treats every active certified answer as an oracle (services/evals/certified_suite.py): execute the stored SQL, re-ask the question through generation with certified matching disabled by construction, execute the generated SQL, and diff the executed results — set-based, column-order-insensitive. Divergence names both sides: fix the definition, or re-certify/retire. Each pair also records bindings — content hashes of the shared assets its SQL touches (services/certify/bindings.py) — so when a shared asset changes, its dependent answers are flagged for re-test, and a green run re-stamps them. The apply that lands an answer — created, or SQL re-authored — also self-tests it on the spot, unconditionally (eval_gate governs the publish gate, never the self-test), and a divergence there is an alert, never a block: divergence at certification time can be the point — the certified answer overrides generation; re-check the oracle if that's not what you meant (services/project/apply.py:695).

Retirement keeps history: a retired answer stays listed and exported but is never served, never matched, never tested. Retire rather than delete for two reasons. First, provenance: a certified answer records that a named human vouched for a number the org then served — when a definition change makes the answer wrong, deleting it erases the fact you ever served it, while retiring keeps the trail and stops the serving. Second, review-approved answers are server-origin (below) and survive file absence entirely, so status: retired is the only way to stop them from the files. Reach for deletion when the pair should never have existed.

Deletion follows the files: a pushed certified_answers.yaml owns its file-originated entries, so removing one deletes it on the next apply (counted loudly in the apply row) — review-approved answers (source: review:*) are server-origin and survive file absence, and a tree that carries no certified_answers.yaml at all leaves the whole surface untouched. One guard either way the corpus empties — the last active answer retired or deleted: under eval_gate: block the apply aborts (an empty corpus would leave the block gate waving every later apply through on a skip; keep at least one active answer, or set eval_gate: warn/off and re-apply), and under warn it publishes with a loud starvation warning. A lens that never had active answers keeps doing its first applies.

The pages, precisely

A certified-definition page's frontmatter carries metric, summary, grain, sources, usage_mode (auto = always in context, search = relevance-selected), verified_value, canonical sql, and optionally the ambiguous form (services/certdefs/__init__.py). One page is simultaneously context, ground truth, a certified answer, and governance metadata. A page carrying both a canonical sql: and an about: pointing at a non-metric fails lint (definition_double_truth, services/validate/report.py:140) — enforceable SQL belongs on the entity.