MA
← Selected systemsENGINEERING CASE STUDY

Behavioral regression · AI change assurance

ProofDiff

A behavioral regression system that determines what changed between baseline and candidate AI behavior, why it changed, how severe the change is, and whether the candidate should be released.

Core objectThe behavioral difference between baseline and candidate
Comparison surfaceoutputs · retrieval · tools · arguments · failures · cost · latency
Release modelseverity-aware regression gates with explicit uncertainty
CORE QUESTION
What changed in the behavior of my AI system, why did it change, and is it acceptable?
Owns · Behavioral change assurance

Behavior diff

The same corpus enters two systems. The difference is the product.

System model

Comparison becomes a reproducible release artifact.

Failure taxonomy

A regression needs a type before it needs a dashboard.

factualinstructionretrievaltool selectiontool argumentsreasoningformatlatencycostsafetystability

Engineering decisions

Evaluation becomes credible when uncertainty stays visible.

DECISION 01

The diff is the product

ProofDiff is not a leaderboard or generic evaluation suite; the primary object is behavioral change introduced by a candidate.

DECISION 02

Judge outputs carry uncertainty

A model judge is a measurement instrument and must be checked for repeatability, disagreement, and calibration.

DECISION 03

Compare traces, not only final text

Tool selection, retrieval, arguments, retries, and failure paths can regress while the final answer still appears plausible.

Capability surface

Behavioral change assurance, not generic benchmarking.

01Baseline/candidate experiments
02Versioned evaluation corpus
03Deterministic validators
04Semantic evaluation
05Judge reliability analysis
06Multi-judge signals
07Trace-aware regression
08Failure taxonomy
09Severity-aware gates
10Statistical confidence
11Cost/latency side effects
12Change attribution
13Historical regression intelligence

Benchmark surface

What the system is designed to measure.

MEASUREMENT CONTRACT

A change should ship only when the behavioral delta is understood, attributable, and inside the accepted regression budget.

Baseline/candidate replay over versioned corpora, deterministic validators, judge ensembles, repeated stochastic runs, and trace comparison.
01Regression recallknown degradations detected
02False positivesacceptable changes incorrectly blocked
03Judge agreementinter-judge + repeatability
04Variancestochastic behavior across repeats
05Trace attributionwhere behavior diverged
06Gate stabilityrelease verdict consistency

Evidence

Publicly defensible architecture and comparison boundaries.

SignalEvidenceInterpretation boundary
Experiment modelVersioned baseline and candidate runs over the same evaluation corpusSupportedThe public case study describes the engineered comparison boundary; private source is not exposed.
Deterministic validationSchema, required-fact, prohibited-content, tool-call, and state-invariant checksSupportedExact validators are used where exactness is meaningful; semantic behavior is evaluated separately.
Trace-aware comparisonRetrieved documents, tool calls, arguments, retries, decisions, and failure pathsSupportedTrace availability depends on the instrumented system under evaluation.
Judge reliabilityRepeatability, disagreement, calibration, and wording sensitivity are treated as measurement concernsSupportedModel-based judges are noisy instruments, not ground truth.
Release resultImproved · regressed · unchanged · acceptable change · uncertainSupportedA changed output is not automatically a regression; severity and invariants determine release impact.

Explicit boundaries

ProofDiff owns the behavioral delta.

Not runtime authorizationNot permission-policy regressionNot inference routingNot agent orchestrationNot generic monitoring