Does an invoice agent learn from its corrections?
40 synthetic suppliers each send 5 invoices in date order. After each one, a reviewer corrects whatever the agent got wrong; the next invoice from that supplier can be extracted with the earlier invoices and their corrections in view. Every model runs under every memory policy, and all are scored on the same 120 invoices (invoice #3 onwards in each series, the first point where every policy has its full history).
- Claude Sonnet 5.5: on the 19 invoices that followed a corrected one, showing the vendor's earlier invoices took fields correct from 89.4% to 96.7% and invoices with every field right from 15.8% to 84.2%. A field corrected last time was wrong again 59.1% of the time without memory and 4.5% with it. 13 invoices went from wrong to right and 0 from right to wrong.
- Claude Opus 5.5: on the 12 invoices that followed a corrected one, showing the vendor's earlier invoices took fields correct from 95.7% to 98.4% and invoices with every field right from 25.0% to 83.3%. A field corrected last time was wrong again 53.8% of the time without memory and 0.0% with it. 7 invoices went from wrong to right and 0 from right to wrong.
- Most accurate setup on the cost frontier: Claude Opus 5.5 with “Previous invoices in the prompt, after a correction” — 98.4% of fields correct, $59.13 per 1,000 invoices.
- Cheapest setup nothing beats on price: Claude Sonnet 5.5 with “Previous invoices in the prompt, after a correction” — 96.7%, $33.99 per 1,000.
After a correction: does seeing the previous invoices fix it?
Each vendor's invoices are extracted in order with its earlier invoices — file, verified values and the reviewer's corrections — in the prompt, then corrected. Scored here: the invoices whose vendor's previous invoice needed a correction, against the same invoices extracted with no memory.
| Model | Invoices | Fields correct | Every field right | Repeat errors | Quirk fields | Silent errors /100 | Per 1,000 | Fixed / broke | Sent back: dates | Sent back: lines | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5effort: medium | 19 | No memory | 89.4% | 15.8% | 59.1% | 59.3% | 47.37 | $19.45 | 13 / 0 | 5.3% | 0.0% |
| With memory | 96.7% | 84.2% | 4.5% | 100.0% | 78.95 | $33.99 | 5.3% | 10.5% | |||
| Claude Opus 5.5effort: medium | 12 | No memory | 95.7% | 25.0% | 53.8% | 55.6% | 16.67 | $43.99 | 7 / 0 | 0.0% | 0.0% |
| With memory | 98.4% | 83.3% | 0.0% | 100.0% | 0.0 | $59.13 | 0.0% | 0.0% |
Repeat errors: a field the reviewer corrected on the previous invoice, wrong again on this one. Fixed / broke: invoices that went from at least one wrong field to all right, and the reverse. Sent back: answers returned to the agent for one re-read — dates that failed the date checks (due before invoice date, due date not matching the terms, invoice dated before the vendor's previous one), or no line items where the vendor's previous invoice had them. With no memory there is no previous invoice, so only the first two date checks apply.
Accuracy against cost
Each point is one model with one memory policy. On the cost frontier — nothing is both cheaper and more accurate — labelled at the right. Beaten on both, or an oracle ceiling (it knows the answers, so it is not a choice). Score is accuracy minus a penalty for confident mistakes (100 is every field right and certain).
Table view
| Model | Policy | Score | Fields correct | Per 1,000 invoices | Frontier |
|---|---|---|---|---|---|
| Claude Sonnet 5.5 | Previous invoices in the prompt, after a correction | 94.5 | 96.7% | $33.99 | ◆ |
| Claude Opus 5.5 | Previous invoices in the prompt, after a correction | 97.7 | 98.4% | $59.13 | ◆ |
Does it learn? Accuracy by invoice from the same vendor
Invoice #1 is always extracted with no history, so every policy starts at the same point.
Claude Sonnet 5.5 (effort: medium)
Claude Opus 5.5 (effort: medium)
Table view
| Model | Policy | #1 | #2 | #3 | #4 | #5 |
|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | Previous invoices in the prompt, after a correction | 99.0% | 98.9% | 99.6% | 100.0% | 100.0% |
| Claude Opus 5.5 | Previous invoices in the prompt, after a correction | 99.2% | 99.9% | 99.6% | 100.0% | 100.0% |
Every model under every policy
Fields correct · fields at risk from vendor quirks correct · cost per 1,000 invoices. ◆ on the cost frontier. Change against the same model with no memory in brackets.
| Model | Previous invoices in the prompt, after a correction |
|---|---|
| Claude Sonnet 5.5effort: medium | 96.7% ◆ quirks 100.0% $33.99 |
| Claude Opus 5.5effort: medium | 98.4% ◆ quirks 100.0% $59.13 |
All metrics
| Model · policy | Score | Correct | Straight through | Sent to review | Silent errors /100 | Repeat errors | Quirk fields | Control vendors | Requests /invoice | Per 1,000 | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 · memory-after-correction | 94.5 | 96.7% | 68.4% | 42.1% | 78.95 | 4.5% of 22 | 100.0% | 100.0% | 1.0 | $33.99 | 8.87s |
| Claude Opus 5.5 · memory-after-correction | 97.7 | 98.4% | 83.3% | 25.0% | 0.0 | 0.0% of 13 | 100.0% | — | 1.0 | $59.13 | 10.76s |
Field accuracy by vendor quirk
| Model · policy | ambiguous dates | customer number | ref as po | ship to first | trading name |
|---|---|---|---|---|---|
| Claude Sonnet 5.5 · memory-after-correction | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| Claude Opus 5.5 · memory-after-correction | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% |
How it works
Method
Documents. Each vendor keeps its name, address, layout, region, currency and invoice-numbering scheme across its series; the numbers, dates and lines change. Most vendors have one or two habits that trip extraction up the same way every time: a brand in the letterhead with the legal name only in the footer; a customer number printed above a generically-labelled invoice number; the PO printed as “Your ref.” next to the vendor's own “Our ref.”; day-first dates from a US vendor (or month-first from a UK one); a delivery address for another company printed before the bill-to block. A quarter have none, as a control. 10 of 40 vendors here have no quirk. Documents are born-digital PDFs, office scans and phone photos.
Memory. After the agent extracts an invoice, a simulated reviewer corrects every wrong field to the answer key. The vendor's history then holds that invoice's file, the verified values, and the list of corrections (“invoice_number: extracted “K56586” → corrected to “NW2026-0469””). A policy decides how much of it the next request includes: none, the previous invoice, the previous two, or text only (values and corrections without the earlier file, for far fewer input tokens). The history each policy sees is the one it produced itself.
Retry policies. “Retry with two” extracts with one earlier invoice; if the answer would be sent to review — a field under 0.85 confidence, or a failed totals/line/date rule — it extracts again with two, and keeps the second answer. Both requests are paid for. The “if wrong” variant retries whenever the first answer is wrong anywhere: impossible in production, it is the ceiling for any retry trigger.
Scoring. Every field is one judgement: right or wrong, and the confidence given. Score = mean of correct − (correct − confidence)², ×100. Right and sure is +1, an honest “not sure” is about 0, wrong and sure is −1. Amounts must match exactly in minor units; identifiers exactly ignoring case and spaces; names and addresses allow formatting differences. Repeat errors: fields corrected on the vendor's previous invoice that are wrong again. Quirk fields: the fields a vendor's quirk puts at risk. Control vendors: all fields on vendors with no quirk.
Cost is every request an invoice took, at the configured per-token prices.
Dataset c5f41982b3cc (generator v1, seed 2026).