MA
← Selected systemsENGINEERING CASE STUDY

Inference economics · constrained execution

InferenceLedger

An inference decision system for choosing the most economically efficient execution path that still satisfies the workload's quality, latency, reliability, provider, and deadline constraints — then proving whether that decision was beneficial.

ArchitectureFastAPI · explainable router · provider adapter · JSONL + SQLite ledgers
Evaluation ruleNo optimization claim without a fixed-model baseline and a quality gate
Measurement modelcost · latency tails · quality floor · SLO attainment · fallback · routing regret
CORE QUESTION
Which execution path should we use?
Owns · Inference decision economics

Constraint envelope

Economics is multi-objective.

QUALITYminimum acceptable task quality
LATENCYSLO + remaining deadline budget
COSTper-request / workload budget
RELIABILITYtimeouts · errors · retries
CAPACITYprovider/model availability
RESTRICTIONSprovider · model · data policy
Savings without acceptable quality are not savings.

Decision system

Workload in. Policy evidence out.

Engineering decisions

The router is only credible when the experiment can reject it.

DECISION 01

Quality is a hard constraint, not a dashboard metric

A cheaper route is not an optimization if it violates the required quality floor.

DECISION 02

Route using observed distributions

Price tables alone do not capture timeout rates, latency tails, retries, or workload-specific quality.

DECISION 03

Every optimization claim needs a baseline

Policy improvement should be reproducible and auditable rather than anecdotal.

Optimization frontier

The best route is the cheapest feasible point — not the cheapest model.

Capability surface

The product owns inference economics with evidence.

01Multi-objective routing
02Workload-aware model selection
03Deadline-aware execution
04Observed provider profiles
05Policy-driven fallback
06Exact cost accounting
07Quality evaluation
08Baseline-vs-candidate lab
09Decision ledger
10Pareto analysis
11Policy replay
12Optimization regression gates

Benchmark surface

What the system is designed to measure.

MEASUREMENT CONTRACT

Optimization is accepted only when economics improve without violating the workload's quality and SLO envelope.

Fixed baselines, policy replay, workload segmentation, repeated provider runs, and quality-gated comparisons.
01Cost / successprovider usage, retries, cache effects
02Latency tailsp50 · p95 · deadline attainment
03Quality floortask validators + semantic evaluation
04Routing regretchosen path vs best feasible path
05Fallback ratetimeouts · errors · policy recovery
06SLO attainmentquality × latency × reliability

Evidence

Observed values and their boundaries.

SignalEvidenceInterpretation boundary
Workload5 JSON-contract requestsVerifiedA smoke workload, not broad semantic evaluation.
Measured provider cost$0.00371000 → $0.00111450 · −69.96%SupportedOne provider endpoint and one run per strategy.
Latency p502976 ms → 2579 msSupportedSmall sample; not a general latency claim.
Latency p9511588 ms → 8681 msSupportedReported for transparency; statistically weak at n=5.
Deterministic quality100% → 100% · 5/5 successes both runsSupportedJSON field equality only; not semantic-quality parity.
Provider retries0 → 0SupportedObserved in the committed smoke runs only.

Explicit boundaries

Integrates with the stack without becoming the stack.

Not a generic API gatewayNot a general model-evaluation platformNot a RAG frameworkNot an agent orchestratorNot a GPU/Kubernetes scheduler