"It looks good to me" is not an eval. The pipeline that replaces vibes: a golden dataset, four kinds of evaluators, and a loop where every failure becomes a test case.
1
Golden Dataset
The foundation — a curated set where quality beats size.
Normal Cases
Edge Cases
Adversarial Inputs
Real Prod Failures
2
The Four Evaluators
Pick per check — cheap & deterministic first, expensive & human last.
Code-based Checks
Deterministic logic: schema compliance, exact matches, latency limits, tool-argument validation. Fast, reproducible, limited to the verifiable.
Ground Truth
Compare against trusted reference answers — great for classification & extraction where a known correct state exists.
LLM-as-Judge
A model scores outputs against a written rubric — correctness, groundedness, tone. Calibrate the judge against human labels before trusting it.
Human Review
Experts on ambiguous, high-risk cases — and the calibration source for everything automated. Expensive, so spend it where it matters.
3
Offline Evals
Pre-release, on the golden set — define coverage, calibrate judges, set the baseline.
Task Coverage
Judge Calibration
Baseline Scores
4
CI/CD Gate
Every prompt, model, or retrieval change runs the same evals as an experiment.
Regression Suite
A/B Experiments
Release Criteria
5
Guardrails
Inline and blocking — the only evals that stop a bad response before the user sees it.
Input Filtering
PII / Policy Blocks
Output Screening
6
Online Evals & Monitoring
Async, on sampled production traffic — real usage finds what your dataset missed.
Sampled Traffic
Drift Detection
User Feedback
Failure Capture
Don't ask "does it look good?" — ask "what's the score on the cases that used to fail?"
Vibes don't scale. Every production failure that reaches step 6 gets promoted into the golden dataset at step 1 — that loop is the whole system.