The Argument
The argument
Medicine already knows how to close a loop — it does it thousands of times a day, and chases the ones left open. The AI recommendation is the one event in the chart that never got a close. The reason is access, not rigour.
Medicine runs on tickets
Every workflow the chart trusts is a ticket. Something opens; something specific is allowed to close it. Open-without-close isn’t a gap — it’s an error the system chases until it dies.
The encounter
Visit beginsOPEN
the visit
Note signedCLOSED
The lab order
Order placedOPEN
specimen → analyzer
Result filedCLOSED
Every open has a designated close — and the chart hunts unclosed tickets for a living.
The ticket nobody closes
Run the same schematic on an AI recommendation. The event happens. The action happens. And the ticket that should close it — the follow-up, the outcome — is never even opened. Not failed: unlogged.
The AI answer
RecommendationOPEN
clinician acts
Order · script · plan
…then?
Follow-up · outcomeSTILL OPEN
The close exists in the workflow — not in the record. Treat the recommendation like a lab order: open it, watch it, close it on evidence.
Everyone grades the model. No one grades the outcome.
Every instrument and product on the chart lives on one side of a wall: they grade artifacts of the session — answers, notes, sandboxed orders. Whether the thing was done lives on the other side, inside the chart. The literature is lopsided in exactly this shape: of 4,609 clinical LLM studies, 1,048 touched real patient data and 19 were prospective randomised trials; an earlier review found 5% used real patient-care data at all. Governance has noticed — CHAI and the Joint Commission now require monitoring for “changes in outcomes” — but a mandate is not an instrument: none of it says what to measure, over what window, against what denominator. The wall is EHR access.
Grading the model
0 instruments & products — the full timeline. everyone.
THE WALL — EHR · PATIENT-DATA ACCESS
Grading the outcome
the same timeline — one entry here; 19 prospective trials in 4,609 studies
The follow-up lives in the EHR. Almost no evaluator can see the EHR. That’s the whole story.
| Era | when | who it's for | what it grades | who sees the data | the loop |
|---|---|---|---|---|---|
| Exam era 2019–24 | item recall | clinician-facing | session artifacts | academia | never checked |
| Rubric era 2023– | conversation | patient-facing | session artifacts | frontier labs | never checked |
| Task era 2024– | task | clinician-facing | sandboxed actions | academia | never checked |
| Deployment era 2025– | usage → outcome | clinician-facing → the whole workflow | session artifacts → patient outcome | vendor streams | the only place it could close — today, privately |
| Product launches '22–'26 | |||||
Distilled from a working note (Aug 2026) on where healthcare-AI evaluation goes next. Chart geometry from
the deck's era cascade; sources are linked inside the deck.
Companions: the visit-clock monitoring map ·
the long-form essay. Print gives one chapter per page.