A multi agent document triage pipeline, and the runs that scored it.

Table 1 Four runs, scored against the manifest

The live and the offline run reach the same 13 / 9 / 4 split and give the same stated reason on every document, because the rules and the routing are pure functions of the extracted fields. The model can only influence classification and extraction. The last row is an earlier revision kept on purpose: the provider connection dropped mid run, so the degradation path fired on a real fault rather than a scripted one. It disagrees with the manifest by design, because 11 documents were marked degraded and sent to a person instead of being auto approved on data the pipeline could not read.

Table 2 Every document in the live run

All 26 documents from the live run: what the classifier decided, what the extractor reported for its own confidence, which business rules fired, and the routing decision against the expected one from the ground truth manifest. Type accuracy is scored over 25, not 26, because D25 is deliberately ambiguous and the manifest asserts no single correct type for it.

Table 3 The raw audit trail for one document

D07 is a duplicate of an invoice already seen earlier in the same batch. Below is every audit line the run wrote for it, unedited from evidence/live-run/events.jsonl with only the absolute file path shortened. Read top to bottom it is the whole control flow: the supervisor dispatches each stage, the classifier and extractor call the model and report their own confidence, the validator applies the rules with no model call and finds the duplicate as a fatal finding, and the router turns that into a decision carrying a policy version.

{"ts": "2026-07-30T11:09:01.168+00:00", "run_id": "live-final-verified", "doc_id": "D07", "node": "supervisor", "event": "dispatch", "detail": {"target": "classify", "delay_s": 0.0, "note": "dispatching classify", "attempts": {}, "degraded": false}}
{"ts": "2026-07-30T11:09:01.168+00:00", "run_id": "live-final-verified", "doc_id": "D07", "node": "classifier", "event": "start", "attempt": 1, "detail": {"model": "anthropic:claude-opus-5:effort=low"}}
{"ts": "2026-07-30T11:09:05.456+00:00", "run_id": "live-final-verified", "doc_id": "D07", "node": "classifier", "event": "classified", "attempt": 1, "detail": {"doc_type": "invoice", "confidence": 0.95, "reason": "The email embeds a full tax invoice with invoice number INV-2201, dates, line item, VAT and total due of R8 050.00.", "usage": {"model": "claude-opus-5", "input_tokens": 981, "output_tokens": 71, "latency_ms": 4287}}}
{"ts": "2026-07-30T11:09:05.460+00:00", "run_id": "live-final-verified", "doc_id": "D07", "node": "supervisor", "event": "dispatch", "detail": {"target": "extract", "delay_s": 0.0, "note": "dispatching extract", "attempts": {"classify": 1}, "degraded": false}}
{"ts": "2026-07-30T11:09:05.464+00:00", "run_id": "live-final-verified", "doc_id": "D07", "node": "extractor", "event": "start", "attempt": 1, "detail": {"doc_type": "invoice", "model": "anthropic:claude-opus-5:effort=low"}}
{"ts": "2026-07-30T11:09:10.847+00:00", "run_id": "live-final-verified", "doc_id": "D07", "node": "extractor", "event": "extracted", "attempt": 1, "detail": {"doc_type": "invoice", "confidence": 0.97, "fields_populated": 11, "fields_requested": 11, "usage": {"model": "claude-opus-5", "input_tokens": 2734, "output_tokens": 166, "latency_ms": 5382}}}
{"ts": "2026-07-30T11:09:10.849+00:00", "run_id": "live-final-verified", "doc_id": "D07", "node": "supervisor", "event": "dispatch", "detail": {"target": "validate", "delay_s": 0.0, "note": "dispatching validate", "attempts": {"classify": 1, "extract": 1}, "degraded": false}}
{"ts": "2026-07-30T11:09:10.850+00:00", "run_id": "live-final-verified", "doc_id": "D07", "node": "validator", "event": "start", "attempt": 1, "detail": {"doc_type": "invoice"}}
{"ts": "2026-07-30T11:09:10.850+00:00", "run_id": "live-final-verified", "doc_id": "D07", "node": "validator", "event": "validated", "attempt": 1, "detail": {"passed": false, "severity_counts": {"fatal": 1}, "findings": [{"code": "DUPLICATE_SUBMISSION", "severity": "fatal", "message": "Same business key as document D01, already processed in this run.", "field": null}], "policy_version": "2026.07.1"}}
{"ts": "2026-07-30T11:09:10.850+00:00", "run_id": "live-final-verified", "doc_id": "D07", "node": "supervisor", "event": "dispatch", "detail": {"target": "route", "delay_s": 0.0, "note": "dispatching route", "attempts": {"classify": 1, "extract": 1, "validate": 1}, "degraded": false}}
{"ts": "2026-07-30T11:09:10.851+00:00", "run_id": "live-final-verified", "doc_id": "D07", "node": "router", "event": "routed", "attempt": 1, "detail": {"decision": "reject", "reasons": ["DUPLICATE_SUBMISSION: Same business key as document D01, already processed in this run."], "policy_version": "2026.07.1", "confidence": 0.95, "degraded": false}}
{"ts": "2026-07-30T11:09:10.851+00:00", "run_id": "live-final-verified", "doc_id": "D07", "node": "supervisor", "event": "finish", "detail": {"target": null, "delay_s": 0.0, "note": "all stages complete", "attempts": {"classify": 1, "extract": 1, "validate": 1, "route": 1}, "degraded": false}}

Note The design change this evidence caused

D06 is an invoice with no due date. In the first schema revision the business required fields were not nullable, so the model had no way to report the absence and supplied a date centuries out instead. The arithmetic downstream was then perfectly correct about nonsense, reporting unusually long payment terms. Nothing crashed and the document still went to a person, because the extractor honestly dropped its confidence to 0.50 and the confidence floor caught it, but the stated reason described a commercial observation instead of a missing field. A schema that cannot express absence does not prevent a missing value, it converts one into a fabricated one, and a fabricated value that is well formed is invisible to shape checking. The two concerns were separated: every field is nullable on the wire so the model can tell the truth, requiredness is enforced as a business rule in Python, and a plausibility rule was added ahead of the payment terms warning as a backstop. In the run above D06 reports FIELD_MISSING on due_date and extractor confidence is 0.95, because the model no longer has to guess. Both changes are covered by regression tests that name the observed failure.