Adaptive Harness: an agentic engineering team that learns from its own runs

An agentic engineering team (a data engineer, a statistician, and the lead role that writes the brief, named race_engineer in the code) whose harness learns from its own recorded runs and proposes changes to its own configuration. A fixed evaluator decides which changes are kept, and MongoDB Atlas, including Atlas Vector Search for lessons, is its memory. The harness is the product. Built on 2026-09-26 at the MongoDB hackathon for Statement One, Recursive Harnessing, and trialed on briefs built from public motorsport timing data (FastF1).

This project is unofficial and is not associated with the Formula 1 companies.

Reports and command output use short identifiers such as S4, E7, R2, D2, v4, and M1. Each family is explained in Terms.

What it is

The claim under test: a harness that records its own runs can find the root causes of its defects in those records and change its own process so that it passes more checks on a fixed evaluation it cannot edit, without getting worse elsewhere. The design fits any analytics and engineering workflow where outputs must cite their evidence.

In the trial, the team writes two kinds of brief:

Every message, tool call, tool result, check, and decision is written to MongoDB Atlas. An improvement cycle reads those records, proposes changes to the harness configuration, and keeps a change only if a fixed evaluator, running as a separate process, shows it passes more checks without getting worse anywhere. The cycle follows the five phases of DMAIC, a method from Six Sigma quality work: define the problem, measure it, analyze its causes, improve the process, and control the result.

How the harness improves itself

Each improvement cycle runs five phases in order: define the problem, measure it, analyze its causes, improve the process, and control the result. The improvement agent writes each phase; code decides whether the cycle may continue, and at improve the fixed evaluator's scores decide whether a proposed version is kept.

The DMAIC cycle as it ran in experiment exp1: define, measure, and analyze passed, and the cycle stopped at improve because the fixed evaluator scored the candidate version v2 no better than the current version v1 on the development cases (rule R2). Control was not reached, so v1 stayed in use.
The cycle as it ran in exp1, drawn from the recorded experiment. The candidate (v2) passed 11 development checks against 11 for the current version (v1), so rule R2 failed and the cycle stopped at improve. Ids in the diagram (M0, D1 to P2, v1, v2, HARNESS, and the baseline, memory_only, and candidate arms) are explained in Terms. Select the diagram to open it full size.

How it works

Three role agents work each case in one bounded discussion. The race engineer runs on Claude Opus 5.5; the statistician and the data engineer run on Claude Sonnet 5. The steps are fixed by the harness configuration, and each has a short id that appears in the records:

stepwhat happenswhotoolsid
1Frame the question.race engineerS1
2Assess the question, in parallel, each without seeing the other's assessment.statistician; data engineerS2a, S2b
3Challenge: each role responds once to the other two.all threeS3
4Gather the race data.data engineerlist_events, race_control_messages, track_statusS4
5Estimate rates and intervals.statisticianintervalS5
6Write the brief.race engineerS6
7Validate the brief, in code only: a format check, plus any optional in-harness checks that are switched on. If one of those optional checks fails, the brief goes back to the race engineer for one repair.codeS7

Each case run has fixed budgets: 40 model calls, 60 tool calls, 900 seconds. The tools read public timing data through the open-source FastF1 library into a local cache outside the repository, with a guard that refuses data dated after the case's as-of date. The fixed evaluator is the only code that reads the answer key. It scores eight checks per run: the brief has a valid format (E1 schema), names the right venue (E2 venue), counts exactly the right races (E3 coverage), gets each safety-car, virtual-safety-car, and red-flag answer right (E4 indicators), gets the counts, rates, and intervals right (E5 arithmetic), gives a verdict or withholds one correctly (E6 disposition), backs every race row with recorded tool results about that race (E7 evidence), and stays within budget (E8 budget).

Architecture The runner holds three roles and the seven steps of a case run. The data engineer calls data tools over the FastF1 cache. The runner writes runs and events to MongoDB Atlas and reads from it the configuration in use and the verified lessons. The evaluator, a separate process, alone reads the answer key, reads outputs and events, and writes evaluations. The DMAIC improvement agent reads development-case evaluations and writes lessons, harness versions, and experiments. Runner: one bounded discussion per case run Race engineerClaude Opus 5.5frames; writes brief StatisticianClaude Sonnet 5assesses; estimates Data engineerClaude Sonnet 5assesses; gathers data frame → 2 blind assessments → challenge → gather → estimate → brief → validate Coordinator: context per role from the context policy; budgets 40 model calls, 60 tool calls, 900 s; records every event Data tools (as-of guard)list_events, race_control_messages, … FastF1 cachepublic timing data, outside the repo gather-step tool calls and results Fixed evaluatorseparate process, eight checks Answer keyread only by the evaluator DMAIC improvement agent (Claude Opus 5.5) fixed prompt per phase; code decides whether each phase passes; sees development-case evidence only MongoDB Atlas runsone per case run, totals, output eventsunique (run_id, seq) harness_versionsexactly one version in use lessonsvector index, filtered by memory snapshot evaluationsresult of each check, hashes experimentsphases, gate results, rule verdicts events in use lessons
Architecture. Blue: runner and roles. Grey: data tools. Purple: evaluator and answer key. Amber: improvement agent. Green: MongoDB Atlas. The full diagram with labelled arrows is architecture.drawio (three pages: architecture, DMAIC cycle, build timeline).

How it improves

The harness improves its own process with the five DMAIC phases. An improvement agent (Claude Opus 5.5, fixed prompt) writes the text of each phase. At the end of each phase, code, not the agent, applies a pass-or-stop test (in DMAIC terms, a tollgate). The cycle stops at the first test that does not pass, and the stop is recorded as the outcome.

phasewhat happenstest to continue (decided by code)
DefineCode builds a charter from the scores of the starting runs on the development cases; the agent writes a problem statement.At least one failed check on a development case.
MeasureConfirm the fingerprints (hashes) of the evaluator and the answer key; re-score every starting run in a fresh evaluator process; count failures per scored check, rank them by check (a Pareto table), and record cost.Identical scores, matching fingerprints, and no measurement-system failure after one rerun.
AnalyzeFor each failure on a development case the agent returns a root cause, with a chain of one to five "why" statements and the recorded events it came from; code verifies each against the recorded run. Verified causes are stored as lessons.At least one verified root cause of a kind the harness is allowed to change: the order of steps, what each role receives, or the in-harness checks.
ImproveThe agent proposes up to three configuration changes, each citing a verified root cause. A validator checks that each change is allowed and within bounds, that its citation is valid, and that it is not an exact repeat of a rejected set of changes. A fresh pilot then runs two groups of runs on the same six cases with the same memory.The five acceptance rules below.
ControlMake the accepted version the one in use (pin it by its configuration fingerprint) and switch on a control plan built from the pilot.Plan switched on; every confirmation case meets its threshold.

The two pilot groups (arms) separate the effect of the configuration change from the effect of the lessons. The memory_only arm runs the current version with the new lessons (memory snapshot M1), and the candidate arm runs the proposed version with the same lessons. Any difference between them comes from the configuration change.

Acceptance rules

Cases come in three sets of two. The development cases (D1, D2) are the only ones the improvement agent learns from. The held-out cases (H1, H2) test whether a change generalizes. The control cases (P1, P2) test that nothing got worse. In each set, case 1 is a venue brief and case 2 is a race audit. Only development-case evidence enters the define and analyze phases or any prompt the improvement agent sees. Held-out and control results are used only by code.

Since the second experiment (exp2), the improve phase is shown the earlier proposals with their development-case outcomes (the proposal history), and since commit 1ba48c8 it also sees proposals the validator refused.

MongoDB design and usage

MongoDB Atlas is the harness's memory and its audit trail. Each record is a pydantic model written with to_doc(). At 13:57 EDT, read-only counts while exp3 was running: runs 53, events 2,615, harness_versions 4, lessons 3, evaluations 47, experiments 3.

collectionpurposekey fieldsindexes
runsOne document per case run._id run_id, harness_version, config_hash, memory_snapshot, arm, experiment_id, model assignment, fastf1_version, status, totals, output(experiment_id, arm, case_id); (harness_version, started_at); (case_id, started_at)
eventsEvery model call, tool call, tool result, message, decision, check, and error, in order._id "run_id:seq", type, stage_id, role, content, refs, context_manifest, usage, model, content_sha256unique run_seq (run_id, seq); run_type (run_id, type); partial control_checks (content.version, type) where content.control exists
harness_versionsEach configuration and why it exists.config, config_hash, parent_id, changes with reasons and cited lesson, validation, status, pinned, control_planpartial unique one_pinned on pinned where pinned is true; status
lessonsRoot causes from the analyze phase.defect, cause_category, why_chain, origin_event_id, source_event_ids, text, status, scope, snapshot, embedding(scope, status, snapshot, created_at); defect.run_id; Atlas Vector Search lessons_vector
evaluationsThe evaluator's verdicts.run_id, case_id, arm, checks E1 to E8 (PASS, FAIL, HARNESS, NA), opportunities, defects, evaluator_sha256, key_sha256run_id; (experiment_id, arm, case_id)
experimentsOne document per improvement cycle.cases, arms, dmaic.{phase}.{status, artifact, tollgate}, acceptance R0 to R4, decision, candidate_version, report_pathcreated_at

Field names are shown as stored. The identifiers in them (E1 to E8, R0 to R4, tollgate, pinned, arm) are explained in Terms.

Why these indexes

Vector search with a snapshot pre-filter

Each lesson is embedded with BAAI/bge-small-en-v1.5 (384 dimensions, cosine). Each role retrieves the top 3 verified lessons at its first step through $vectorSearch on lessons_vector, with the filter fields declared in the index:

{ "$vectorSearch": {
    "index": "lessons_vector", "path": "embedding",
    "queryVector": [...], "numCandidates": 100, "limit": 3,
    "filter": { "$and": [
      { "status":   { "$eq": "verified" } },
      { "scope":    { "$eq": "neutralization_brief" } },
      { "snapshot": { "$eq": "M1" } } ] } } },
{ "$addFields": { "score": { "$meta": "vectorSearchScore" } } }

The pre-filter runs inside the index. A run on the memory as it stood before any cycle (snapshot M0) cannot see lessons written by a cycle (snapshot M1), and unverified root causes are never retrieved. This is what makes the baseline arm (without the new lessons) and the memory_only arm (with them) a fair comparison.

The aggregation used by show cost

Token cost per arm, role, and model comes from the recorded model-call events, joined to their run for the arm (python -m adaptive_harness show cost). The $match is served by the run_type index:

[ { "$match": { "run_id": { "$in": [...] }, "type": "model_call" } },
  { "$lookup": { "from": "runs", "localField": "run_id",
                 "foreignField": "_id", "as": "run" } },
  { "$unwind": "$run" },
  { "$group": { "_id": { "arm": "$run.arm", "role": "$role", "model": "$model" },
                "calls": { "$sum": 1 },
                "input_tokens":  { "$sum": { "$ifNull": ["$usage.input_tokens", 0] } },
                "output_tokens": { "$sum": { "$ifNull": ["$usage.output_tokens", 0] } },
                "cache_read_tokens": { "$sum": { "$ifNull": ["$usage.cache_read_tokens", 0] } } } },
  { "$sort": { "_id.arm": 1, "_id.role": 1, "_id.model": 1 } } ]

control_checks explain, before and after

Each time the evaluator checks a run of the version in use, it reads the recorded control outcomes of that version:

db.events.find({
  "type": "check",
  "content.control": { "$exists": true },
  "content.version": "v1"
})

The index keys both equality fields, and its partial filter is a predicate the query carries verbatim, so it holds only control-outcome events instead of an entry for every message, model-call, and tool event:

db.events.createIndex(
  { "content.version": 1, "type": 1 },
  { name: "control_checks",
    partialFilterExpression: { "content.control": { "$exists": true } } }
)

Both plans are explain with executionStats on the project's Atlas cluster (MongoDB 8.0.32), run read-only while another experiment was writing, so the collection grew between captures. No control outcome had been recorded yet, so both return no documents; what changes is how much the server reads to find that out.

beforeafter
indexes on events_id_, run_seq, run_type_id_, run_seq, run_type, control_checks
documents in events1,9522,157
winning planCOLLSCANFETCH <- IXSCAN (control_checks)
rejected plans00
documents returned00
keys examined00
documents examined1,9520
execution time2 ms1 ms

Before the index, every control check scanned the whole collection, and the cost grows with every run recorded. After it, documents examined equal the number of matching control outcomes, independent of how many other events exist. The time difference is small at this size; the documents-examined count is the number that scales. Source: reports/explain_control_checks.md.

The experiments

Each experiment (exp1, exp2, exp3) is one improvement cycle on the same six cases. Numbers are taken from the generated reports. exp3 also ran the two development cases a second time under v1 (the repeat arm): no check result changed between the identical runs. Two exp3 starting runs, on the control cases P1 and P2, ended as measurement-system failures (HARNESS) when the model provider refused requests for insufficient account credit; each was rerun once and the reruns are scored.

exp1 rejected

What the improvement agent was shown
One verified root cause: the evidence check (E7) failed on the development race audit (D2) because the only check at the final validation step (S7) was a format check, which cannot see that a race row cites no tool result about that race. Starting point on the development cases: 1 failed check in 12 scored checks.
What it proposed
Version v2: switch on the optional in-harness check venue_match. The agent stated this was a trial, not a confirmed fix.
Rule verdicts
Valid measurement (R0) yes; no regression (R1) yes; improvement (R2) no: development checks passed, candidate 11, current 11; held-out (R3) yes, 10 and 10; time (R4) yes, 1134.9 s against 1183.6 s.
Decision
Rejected at the improve phase. v1 stays the version in use. Report

exp2 rejected

What the improvement agent was shown
A fresh verified root cause, again the evidence check (E7) failing on D2 for the same reason, plus the proposal history: v2 (venue_match on) rejected because development checks did not improve, 11 against 11.
What it proposed
Version v3: switch on the optional in-harness check source_agreement, the untried option; the agent called it "a partial fit at best".
Rule verdicts
Valid measurement (R0) yes; no regression (R1) yes; improvement (R2) no: development checks passed, candidate 11, current 11; held-out (R3) yes, 10 and 10; time (R4) yes, 1066.3 s against 986.0 s.
Decision
Rejected at the improve phase. v1 stays the version in use. Report

exp3 rejected

What the improvement agent was shown
A third verified root cause for the evidence check (E7) failing on D2 (a gap in the in-harness checks), the proposal history (v2 and v3, both rejected because development checks did not improve), and a new optional check, row_evidence, that the operator had added to the list of available checks, switched off.
What it proposed
Version v4: switch on row_evidence, the check the lessons describe. Validator: valid.
Rule verdicts
Valid measurement (R0) yes; no regression (R1) no: the control venue brief P1 lost the venue (E2) and coverage (E3) checks; improvement (R2) no: development checks passed, candidate 11, current 11; held-out (R3) yes, 10 and 10; time (R4) yes, 1181.9 s against 993.0 s.
What the check did
row_evidence passed in all six candidate runs, so the repair step never fired. The check accepts a race row with at least one citation about that race; the evaluator's evidence check (E7) requires every citation to be about that race. D2's evidence check still failed.
Decision
Rejected at the improve phase. v1 stays the version in use. Report

Checks passed per case set

Each column is one arm, a group of runs over the same cases. M0 is the memory before the cycle; M1 adds the lessons the cycle wrote.

experimentcase setbaseline: v1, M0memory_only: v1, M1candidate: proposed version, M1
exp1development111111 (v2)
exp1held-out101010
exp1control10810
exp2development111111 (v3)
exp2held-out101010
exp2control91010
exp3development111111 (v4)
exp3held-out101010
exp3control10108

What we learned

Provenance and operator decisions

Limits

Terms

Identifiers that appear in the reports, the records, and command output, one family per line.

identifiermeaning
S1 to S7 (S2a, S2b)The steps of one case run: frame (S1), two blind assessments by the statistician (S2a) and the data engineer (S2b), challenge (S3), gather (S4), estimate (S5), write the brief (S6), validate in code (S7).
X1 to X3Optional extra steps a proposal may insert: identity check (X1), definitions (X2), reconcile sources (X3).
E1 to E8The evaluator's eight checks per run: schema (E1), venue (E2), coverage (E3), indicators (E4), arithmetic (E5), disposition (E6), evidence (E7), budget (E8).
R0 to R4The acceptance rules for a proposed version: valid measurement (R0), no regression on control cases (R1), improvement on development cases (R2), no loss on held-out cases (R3), time at most 1.5 times (R4).
D, H, P casesCase sets: development (D1, D2), held-out (H1, H2), control (P1, P2); case 1 in each set is a venue brief, case 2 a race audit.
v1 to v4Harness versions: v1 is the starting configuration and still in use; v2, v3, and v4 are the rejected candidates of exp1, exp2, and exp3.
exp1 to exp3Experiments, one improvement cycle each.
armsGroups of runs in an experiment: baseline (current version, memory M0), repeat (the development cases again, same setup), memory_only (current version, memory M1), candidate (proposed version, memory M1), confirmation (after acceptance; not yet run).
M0, M1Memory snapshots: M0 is memory before any cycle; M1 adds the verified lessons a cycle wrote.
DMAICThe five improvement phases from Six Sigma: define, measure, analyze, improve, control.
tollgateThe pass-or-stop test code applies at the end of each phase.
pinnedMarked as the version in use; exactly one version is pinned at a time.
PASS, FAIL, HARNESS, NACheck results: passed, failed, measurement-system failure (a data, model, or timeout error; rerun once), not applicable.
GO, HOLDA brief's disposition: gives its verdict, or withholds it and records why.
defect, opportunity, DPOA failed check, a check that could pass or fail, and defects per opportunity.
method, material, measurementRoot-cause categories the harness may change: step order (method), what each role receives (material), in-harness checks (measurement); machine, people, and environment are fixed.
/stages, /context_policy, /checks/...Configuration paths a proposal may change, one per category above; for example /checks/venue_match switches that check on or off.
venue_match, source_agreement, min_races_for_rate, row_evidenceOptional in-harness checks run at validation (S7), separate from the evaluator; all off in v1.
SC, VSCSafety car, virtual safety car.