Fire drills for systems agents

Agents enter with code. They leave with traces.

Training Day Redline is a cost-capped proving harness for ML systems work. A coding agent edits candidate kernel, CUDA and inference code inside a bounded benchmark. What comes out is a replayable evidence record that someone else can verify.

BenchmarkFire Drills, bounded by correctness, budget and policy

OutputA redacted .tdpack bundle and its checksums

Leaves your machineNothing, unless you call a model provider yourself

~ / trainingday LOCAL-FIRST

$ td drills

# 11 public drills in Gauntlet Zero: Kernels

$ td rep rmsnorm_001 --agent manual

# one Rep, scored under Redline, trace kept

$ td pack export --trial <trial-id> --out proof.tdpack

# redacted by default

$ td pack verify proof.tdpack

# anyone can re-check every checksum

What a run emits

One record per trial. Six lines of manifest, and a list of what never ships.

Drill identity

Drill id, evaluator revision, reference revision, case manifest digest.

Budget

api_cost_usd gpu_minutes wall_clock_minutes, with the observed values against each ceiling.

Correctness

Pass or fail, decided by the evaluator parent. The candidate cannot edit the oracle result or the evaluator frames it is scored against.

Performance

Reported as neutral until a sealed execution attestation exists. A self-reported timing is not evidence.

Integrity

SHA-256 over every included file, re-checkable with scripts/tdpack_verify.py and no account.

Never included

API keys, provider headers, raw model traffic, environment variables, local paths, git remotes, hidden tests.

Redline

Every Rep runs under three ceilings. Cross one and the run is marked failed, the correctness result stays factual, and automation exits non-zero.

A missing measurement is rejected, not treated as free usage. Absent, negative and non-finite API, GPU and wall-clock readings fail closed.

API costDollars spent on model calls, ceiling set per Rep.

Hard

GPU minutesAccelerator time, ceiling set before the rental is booked.

Hard

Wall clockElapsed minutes, ceiling set before the session starts.

Hard

Gauntlet Zero: Kernels

The CPU-safe drill pack. A CUDA drill with a CPU semantic oracle is in development.

elementwise_fusion_001rmsnorm_001layernorm_backward_001softmax_001topk_argmax_001dequantization_001quantized_matmul_001causal_mha_001rotary_embedding_001ragged_moe_001cpu_inference_001

Where it stands

Written down because the harness insists on it.

Public pack, not released

The harness and the eleven drills are built and running. The public distribution is still in private development.

Leaderboard, unranked

Ranking is gated on a sealed execution attestation that does not exist yet. Until then, a proof verifies correctness and nothing else.

No account, no telemetry

Runs are local. Exports are explicit. Nothing is uploaded, and there is no update check.