Adaptive Harness: an agentic engineering team that learns from its own runs
An agentic engineering team (a data engineer, a statistician, and the lead role that writes the brief, named race_engineer in the code) whose harness learns from its own recorded runs and proposes changes to its own configuration. A fixed evaluator decides which changes are kept, and MongoDB Atlas, including Atlas Vector Search for lessons, is its memory. The harness is the product. Built on 2026-09-26 at the MongoDB hackathon for Statement One, Recursive Harnessing, and trialed on briefs built from public motorsport timing data (FastF1).
This project is unofficial and is not associated with the Formula 1 companies.
Reports and command output use short identifiers such as S4, E7, R2, D2, v4, and M1. Each family is explained in Terms.
What it is
The claim under test: a harness that records its own runs can find the root causes of its defects in those records and change its own process so that it passes more checks on a fixed evaluation it cannot edit, without getting worse elsewhere. The design fits any analytics and engineering workflow where outputs must cite their evidence.
In the trial, the team writes two kinds of brief:
A venue brief (venue_brief): how often Grand Prix races at a venue were interrupted by a safety car (SC) or a virtual safety car (VSC), with counts, rates, and 90% confidence intervals (Wilson method).
A race audit (race_audit): whether a given race had an SC, a VSC, or a red flag.
Every message, tool call, tool result, check, and decision is written to MongoDB Atlas. An improvement cycle reads those records, proposes changes to the harness configuration, and keeps a change only if a fixed evaluator, running as a separate process, shows it passes more checks without getting worse anywhere. The cycle follows the five phases of DMAIC, a method from Six Sigma quality work: define the problem, measure it, analyze its causes, improve the process, and control the result.
How the harness improves itself
Each improvement cycle runs five phases in order: define the problem, measure it, analyze its causes, improve the process, and control the result. The improvement agent writes each phase; code decides whether the cycle may continue, and at improve the fixed evaluator's scores decide whether a proposed version is kept.
The cycle as it ran in exp1, drawn from the recorded experiment. The candidate (v2) passed 11 development checks against 11 for the current version (v1), so rule R2 failed and the cycle stopped at improve. Ids in the diagram (M0, D1 to P2, v1, v2, HARNESS, and the baseline, memory_only, and candidate arms) are explained in Terms. Select the diagram to open it full size.
How it works
Three role agents work each case in one bounded discussion. The race engineer runs on Claude Opus 5.5; the statistician and the data engineer run on Claude Sonnet 5. The steps are fixed by the harness configuration, and each has a short id that appears in the records:
step
what happens
who
tools
id
1
Frame the question.
race engineer
S1
2
Assess the question, in parallel, each without seeing the other's assessment.
statistician; data engineer
S2a, S2b
3
Challenge: each role responds once to the other two.
all three
S3
4
Gather the race data.
data engineer
list_events, race_control_messages, track_status
S4
5
Estimate rates and intervals.
statistician
interval
S5
6
Write the brief.
race engineer
S6
7
Validate the brief, in code only: a format check, plus any optional in-harness checks that are switched on. If one of those optional checks fails, the brief goes back to the race engineer for one repair.
code
S7
Each case run has fixed budgets: 40 model calls, 60 tool calls, 900 seconds. The tools read public timing data through the open-source FastF1 library into a local cache outside the repository, with a guard that refuses data dated after the case's as-of date. The fixed evaluator is the only code that reads the answer key. It scores eight checks per run: the brief has a valid format (E1 schema), names the right venue (E2 venue), counts exactly the right races (E3 coverage), gets each safety-car, virtual-safety-car, and red-flag answer right (E4 indicators), gets the counts, rates, and intervals right (E5 arithmetic), gives a verdict or withholds one correctly (E6 disposition), backs every race row with recorded tool results about that race (E7 evidence), and stays within budget (E8 budget).
Architecture. Blue: runner and roles. Grey: data tools. Purple: evaluator and answer key. Amber: improvement agent. Green: MongoDB Atlas. The full diagram with labelled arrows is architecture.drawio (three pages: architecture, DMAIC cycle, build timeline).
How it improves
The harness improves its own process with the five DMAIC phases. An improvement agent (Claude Opus 5.5, fixed prompt) writes the text of each phase. At the end of each phase, code, not the agent, applies a pass-or-stop test (in DMAIC terms, a tollgate). The cycle stops at the first test that does not pass, and the stop is recorded as the outcome.
phase
what happens
test to continue (decided by code)
Define
Code builds a charter from the scores of the starting runs on the development cases; the agent writes a problem statement.
At least one failed check on a development case.
Measure
Confirm the fingerprints (hashes) of the evaluator and the answer key; re-score every starting run in a fresh evaluator process; count failures per scored check, rank them by check (a Pareto table), and record cost.
Identical scores, matching fingerprints, and no measurement-system failure after one rerun.
Analyze
For each failure on a development case the agent returns a root cause, with a chain of one to five "why" statements and the recorded events it came from; code verifies each against the recorded run. Verified causes are stored as lessons.
At least one verified root cause of a kind the harness is allowed to change: the order of steps, what each role receives, or the in-harness checks.
Improve
The agent proposes up to three configuration changes, each citing a verified root cause. A validator checks that each change is allowed and within bounds, that its citation is valid, and that it is not an exact repeat of a rejected set of changes. A fresh pilot then runs two groups of runs on the same six cases with the same memory.
The five acceptance rules below.
Control
Make the accepted version the one in use (pin it by its configuration fingerprint) and switch on a control plan built from the pilot.
Plan switched on; every confirmation case meets its threshold.
The two pilot groups (arms) separate the effect of the configuration change from the effect of the lessons. The memory_only arm runs the current version with the new lessons (memory snapshot M1), and the candidate arm runs the proposed version with the same lessons. Any difference between them comes from the configuration change.
Acceptance rules
Valid measurement (R0): no run ends as a measurement-system failure (recorded as HARNESS) after one rerun.
No regression (R1): every check that passes on a control case under the current version also passes under the candidate.
Improvement (R2): the candidate passes strictly more checks on the development cases.
Held-out (R3): the candidate passes at least as many checks on the held-out cases.
Time (R4): the candidate's total wall-clock time is at most 1.5 times the current version's.
Cases come in three sets of two. The development cases (D1, D2) are the only ones the improvement agent learns from. The held-out cases (H1, H2) test whether a change generalizes. The control cases (P1, P2) test that nothing got worse. In each set, case 1 is a venue brief and case 2 is a race audit. Only development-case evidence enters the define and analyze phases or any prompt the improvement agent sees. Held-out and control results are used only by code.
Since the second experiment (exp2), the improve phase is shown the earlier proposals with their development-case outcomes (the proposal history), and since commit 1ba48c8 it also sees proposals the validator refused.
MongoDB design and usage
MongoDB Atlas is the harness's memory and its audit trail. Each record is a pydantic model written with to_doc(). At 13:57 EDT, read-only counts while exp3 was running: runs 53, events 2,615, harness_versions 4, lessons 3, evaluations 47, experiments 3.
Field names are shown as stored. The identifiers in them (E1 to E8, R0 to R4, tollgate, pinned, arm) are explained in Terms.
Why these indexes
Unique (run_id, seq) on events. Event writes are idempotent: a retry that hits a duplicate key counts as already written, so a run's history never has gaps or doubles.
Partial unique index on pinned. Exactly one harness version is marked as the one in use (pinned), and the database, not only the code, guarantees it. Making a new version the one in use unmarks the old version first. The first configuration, v1, is the version in use today.
Compound indexes on (experiment_id, arm, case_id). Reports, the acceptance rules, and the report's tables per group of runs all read by experiment and arm.
Partial index control_checks. Serves the evaluator's query for the control history of the version in use (the recorded results of checking its later runs against its control plan) without scanning every event. The before-and-after explain is below.
Vector search with a snapshot pre-filter
Each lesson is embedded with BAAI/bge-small-en-v1.5 (384 dimensions, cosine). Each role retrieves the top 3 verified lessons at its first step through $vectorSearch on lessons_vector, with the filter fields declared in the index:
The pre-filter runs inside the index. A run on the memory as it stood before any cycle (snapshot M0) cannot see lessons written by a cycle (snapshot M1), and unverified root causes are never retrieved. This is what makes the baseline arm (without the new lessons) and the memory_only arm (with them) a fair comparison.
The aggregation used by show cost
Token cost per arm, role, and model comes from the recorded model-call events, joined to their run for the arm (python -m adaptive_harness show cost). The $match is served by the run_type index:
The index keys both equality fields, and its partial filter is a predicate the query carries verbatim, so it holds only control-outcome events instead of an entry for every message, model-call, and tool event:
Both plans are explain with executionStats on the project's Atlas cluster (MongoDB 8.0.32), run read-only while another experiment was writing, so the collection grew between captures. No control outcome had been recorded yet, so both return no documents; what changes is how much the server reads to find that out.
before
after
indexes on events
_id_, run_seq, run_type
_id_, run_seq, run_type, control_checks
documents in events
1,952
2,157
winning plan
COLLSCAN
FETCH <- IXSCAN (control_checks)
rejected plans
0
0
documents returned
0
0
keys examined
0
0
documents examined
1,952
0
execution time
2 ms
1 ms
Before the index, every control check scanned the whole collection, and the cost grows with every run recorded. After it, documents examined equal the number of matching control outcomes, independent of how many other events exist. The time difference is small at this size; the documents-examined count is the number that scales. Source: reports/explain_control_checks.md.
The experiments
Each experiment (exp1, exp2, exp3) is one improvement cycle on the same six cases. Numbers are taken from the generated reports. exp3 also ran the two development cases a second time under v1 (the repeat arm): no check result changed between the identical runs. Two exp3 starting runs, on the control cases P1 and P2, ended as measurement-system failures (HARNESS) when the model provider refused requests for insufficient account credit; each was rerun once and the reruns are scored.
exp1 rejected
What the improvement agent was shown
One verified root cause: the evidence check (E7) failed on the development race audit (D2) because the only check at the final validation step (S7) was a format check, which cannot see that a race row cites no tool result about that race. Starting point on the development cases: 1 failed check in 12 scored checks.
What it proposed
Version v2: switch on the optional in-harness check venue_match. The agent stated this was a trial, not a confirmed fix.
Rule verdicts
Valid measurement (R0) yes; no regression (R1) yes; improvement (R2) no: development checks passed, candidate 11, current 11; held-out (R3) yes, 10 and 10; time (R4) yes, 1134.9 s against 1183.6 s.
Decision
Rejected at the improve phase. v1 stays the version in use. Report
exp2 rejected
What the improvement agent was shown
A fresh verified root cause, again the evidence check (E7) failing on D2 for the same reason, plus the proposal history: v2 (venue_match on) rejected because development checks did not improve, 11 against 11.
What it proposed
Version v3: switch on the optional in-harness check source_agreement, the untried option; the agent called it "a partial fit at best".
Rule verdicts
Valid measurement (R0) yes; no regression (R1) yes; improvement (R2) no: development checks passed, candidate 11, current 11; held-out (R3) yes, 10 and 10; time (R4) yes, 1066.3 s against 986.0 s.
Decision
Rejected at the improve phase. v1 stays the version in use. Report
exp3 rejected
What the improvement agent was shown
A third verified root cause for the evidence check (E7) failing on D2 (a gap in the in-harness checks), the proposal history (v2 and v3, both rejected because development checks did not improve), and a new optional check, row_evidence, that the operator had added to the list of available checks, switched off.
What it proposed
Version v4: switch on row_evidence, the check the lessons describe. Validator: valid.
Rule verdicts
Valid measurement (R0) yes; no regression (R1) no: the control venue brief P1 lost the venue (E2) and coverage (E3) checks; improvement (R2) no: development checks passed, candidate 11, current 11; held-out (R3) yes, 10 and 10; time (R4) yes, 1181.9 s against 993.0 s.
What the check did
row_evidence passed in all six candidate runs, so the repair step never fired. The check accepts a race row with at least one citation about that race; the evaluator's evidence check (E7) requires every citation to be about that race. D2's evidence check still failed.
Decision
Rejected at the improve phase. v1 stays the version in use. Report
Checks passed per case set
Each column is one arm, a group of runs over the same cases. M0 is the memory before the cycle; M1 adds the lessons the cycle wrote.
experiment
case set
baseline: v1, M0
memory_only: v1, M1
candidate: proposed version, M1
exp1
development
11
11
11 (v2)
exp1
held-out
10
10
10
exp1
control
10
8
10
exp2
development
11
11
11 (v3)
exp2
held-out
10
10
10
exp2
control
9
10
10
exp3
development
11
11
11 (v4)
exp3
held-out
10
10
10
exp3
control
10
10
8
What we learned
The same verified lesson across cycles. Each cycle's analyze phase, working from fresh runs, verified the same root cause for the evidence check (E7) failing on the development race audit (D2), and placed it in the category of in-harness checks (measurement). The only check at the final validation step was a format check, so nothing tested that each race row cites a tool result from the same run about that race. Three such lessons are stored, one per cycle.
Memory alone did not improve results. The memory_only arm (v1 plus the new lessons) passed 11 development checks in exp1, exp2, and exp3, the same as the baseline without them.
Scores vary between identical runs. Under the same v1 configuration, the control cases passed 10, 8, 9, 10, 10, and 10 checks across the six scored v1 arms with control cases in exp1 to exp3 (the control venue brief P1 flipped on the venue (E2) and coverage (E3) checks). This is why the improvement rule (R2) compares the candidate with the current version run fresh, and why no statistical claim is made.
The evaluator refused unproven changes. v2 (venue_match on) and v3 (source_agreement on) each passed 11 development checks against 11, so the improvement rule failed and v1 stayed in use. The agent had flagged both as uncertain in its own risk statements. v4 was refused too (below).
The operator added the check the lessons asked for, and the harness chose to use it. No check the harness could switch on tested the evidence of each row, so after two cycles the operator added row_evidence to the list of available checks, switched off, together with one repair step at final validation. In exp3 the improvement agent, not the operator, proposed switching it on (v4). The acceptance rules rejected v4 on no regression (R1: P1 lost the venue and coverage checks) and on improvement (R2: 11 against 11). row_evidence passed in all six candidate runs, so the repair step never fired: the check accepts a row with at least one citation about that race, and the evaluator requires every citation to be about that race. The check is looser than the measurement it was meant to protect.
Provenance and operator decisions
Evaluation data first.14d7030 (10:40 EDT) committed the cases, definitions, and frozen answer key before any harness code; ecf78b2 (10:42) the quality specification; 703e771 (10:52) the frozen contract. The lanes were then built in parallel against the contract.
Optional race-audit cases D2, H2, P2. The only default development case, the venue brief D1, has zero races at its venue, so it cannot exhibit citation defects. The three optional race-audit cases were published in data/cases.json with the defaults, before any harness code, and were used, not added. Every arm covers all six.
Budgets raised before the first starting run.1344432 (11:35) set per-case budgets from placeholders of 20 model calls, 40 tool calls, 600 s to 40, 60, 900 s. The first baseline run started at 11:49.
row_evidence added to the list of available checks, switched off, with one repair step.032720a and f39d1fe, merged as dccd9f1 (13:30), after two cycles verified the same lesson. When a switched-on optional check fails at final validation, the race engineer revises the brief once; if checks still fail, the run ends on HOLD (the brief withholds its verdict and records why). Configurations stored before it keep their fingerprints (configuration hashes). Switching it on is left to the improve phase and the acceptance rules.
All code was written on 2026-09-26 with AI coding assistants. The design notes and instructions given to them were written before the event and are not included.
Limits
Six fixed cases (two development, two held-out, two control), one run per case per arm. No statistical claim; the models are not deterministic and a repeat could differ.
No sigma level or capability index (Six Sigma summary scores that need far more data). The control plan uses the specification's thresholds, not statistical control limits, which need about 20 runs.
No candidate has been accepted in exp1, exp2, or exp3; the control phase has not been reached, and v1 is still the version in use.
A hackathon prototype, not a production system.
Terms
Identifiers that appear in the reports, the records, and command output, one family per line.
identifier
meaning
S1 to S7 (S2a, S2b)
The steps of one case run: frame (S1), two blind assessments by the statistician (S2a) and the data engineer (S2b), challenge (S3), gather (S4), estimate (S5), write the brief (S6), validate in code (S7).
X1 to X3
Optional extra steps a proposal may insert: identity check (X1), definitions (X2), reconcile sources (X3).
The acceptance rules for a proposed version: valid measurement (R0), no regression on control cases (R1), improvement on development cases (R2), no loss on held-out cases (R3), time at most 1.5 times (R4).
D, H, P cases
Case sets: development (D1, D2), held-out (H1, H2), control (P1, P2); case 1 in each set is a venue brief, case 2 a race audit.
v1 to v4
Harness versions: v1 is the starting configuration and still in use; v2, v3, and v4 are the rejected candidates of exp1, exp2, and exp3.
exp1 to exp3
Experiments, one improvement cycle each.
arms
Groups of runs in an experiment: baseline (current version, memory M0), repeat (the development cases again, same setup), memory_only (current version, memory M1), candidate (proposed version, memory M1), confirmation (after acceptance; not yet run).
M0, M1
Memory snapshots: M0 is memory before any cycle; M1 adds the verified lessons a cycle wrote.
DMAIC
The five improvement phases from Six Sigma: define, measure, analyze, improve, control.
tollgate
The pass-or-stop test code applies at the end of each phase.
pinned
Marked as the version in use; exactly one version is pinned at a time.
PASS, FAIL, HARNESS, NA
Check results: passed, failed, measurement-system failure (a data, model, or timeout error; rerun once), not applicable.
GO, HOLD
A brief's disposition: gives its verdict, or withholds it and records why.
defect, opportunity, DPO
A failed check, a check that could pass or fail, and defects per opportunity.
method, material, measurement
Root-cause categories the harness may change: step order (method), what each role receives (material), in-harness checks (measurement); machine, people, and environment are fixed.
/stages, /context_policy, /checks/...
Configuration paths a proposal may change, one per category above; for example /checks/venue_match switches that check on or off.