Research · Accounts payable

An invoice agent that learns from every correction.

We built an AI agent that reads supplier invoices, checks its own work, and remembers what your team fixed last time. Then we measured it, model by model, and published everything.

By: transparent.dev
Models evaluated: Claude Sonnet 5.5, Claude Opus 5.5
Test set: 200 invoices · 40 suppliers · 11 countries
Data: results.json · all metrics

Finding 01

99.7%

of invoice fields read correctly

Claude Opus 5.5, 200 invoices

Finding 02

confident mistakes once it knows a supplier. Anything doubtful goes to a person.

Claude Opus 5.5, per 100 invoices

Finding 03

54% → 0%

of corrected mistakes repeated on the supplier's next invoice

without memory → with memory

Finding 04

2.9¢

per invoice, with Claude Sonnet 5.5. 5.6¢ with Claude Opus 5.5.

model costs, at list price

01 — The problem

The problem

Most of accounts payable is retyping: supplier, invoice number, dates, lines, totals. AI is now good at that. What it isn't good at, out of the box, is your suppliers.

Every supplier has habits. One prints a brand at the top and its legal name in the small print. One writes dates the other way round. One puts its own reference where you expect your PO. A generic AI gets the same thing wrong every month, and every month someone on your team fixes it.

So we asked a simple question: if the agent can see a supplier's last invoices, and what your team corrected on them, does it stop making the same mistake? And which model gives the most accuracy for the money?

Short answer: yes, and it depends on what a mistake costs you. Here's the evidence.

02 — Supplier habits

Every supplier has habits

We generated 200 realistic invoices from 40 suppliers across 11 countries, five per supplier, in date order: emailed PDFs, office scans and phone photos. Most suppliers were given a habit that trips up extraction. These are real answers from our runs.

Brand in the letterhead

On the page: NORTHSTAR DIRECT across the top; the registered name in the footer
First time: Northstar Direct
After one correction: Chase-Lynn

The legal name is the one that has to match the supplier record.

Month-first dates from a UK supplier

On the page: Invoice date 05/02/2026, Net 30
First time: 5 February 2026
After one correction: 2 May 2026

Both readings are real dates. Only the supplier's history settles it.

Labels in the tax number

On the page: ABN: 07 246 609 939
First time: ABN: 07 246 609 939
After one correction: 07 246 609 939

A reviewer fixes it once; the agent keeps to it from then on.

Sample emailed pdf invoice from the test set — Emailed PDF · AU · delivery address before the bill-to

Sample office scan invoice from the test set — Office scan · AU · its reference where the PO goes

Sample phone photo invoice from the test set — Phone photo · FR · its reference where the PO goes

03 — How it works

How the agent works

No magic, and nothing your auditors won't understand. One model call reads the invoice; plain rules check it; a person sees only what's doubtful; and their corrections become the agent's memory.

Read

The invoice — PDF, scan or photo — becomes a fixed record: supplier, numbers, dates, lines, totals. Every field carries a confidence.

Remember

The supplier's last two invoices come along, with what your team corrected on them, so it reads this one the way your team would.

Check

Plain rules, no AI: totals add up, dates fit the terms, lines are there. If something fails, the agent re-reads once.

Route

Confident and consistent goes straight through. Anything doubtful goes to a person, with the reason on screen.

Learn

Whatever the reviewer fixes is remembered for that supplier's next invoice. No retraining, no waiting.

04 — Results

What the models did

Same 200 invoices, same prompt, same checks, same memory. Only the model changes. We run every new model through exactly this before we'd recommend it.

Claude Sonnet 5.5

effort: medium

Best value

99.5%

of fields right, all invoices

98%

of invoices fully right once it knows the supplier

confident mistakes per 100 invoices, once it knows the supplier

$29.39

per 1,000 invoices · 9s each

Claude Opus 5.5

effort: medium

Most accurate

99.7%

of fields right, all invoices

99%

of invoices fully right once it knows the supplier

confident mistakes per 100 invoices, once it knows the supplier

$55.93

per 1,000 invoices · 11s each

A confident mistake is a wrong field the model was at least 85% sure of — the kind that would be paid without anyone looking. Costs are model usage at list price, including every re-read.

Figure 1

Memory makes the second invoice better than the first

Invoices with every field right: a supplier's first invoice, with nothing to remember, against its later ones, with the history in view.

New supplierKnown supplier

Table view

Model	New supplier	Known supplier
Claude Sonnet 5.5	60.0%	98.1%
Claude Opus 5.5	75.0%	98.8%

Figure 2

Corrected once, fixed for good

Of the fields a reviewer corrected on a supplier's previous invoice, how many were wrong again on the next one. Same invoices, read with and without the history.

Without historyWith history

Table view

Model	Invoices	Without	With	Fixed / broken
Claude Sonnet 5.5	19	59.1%	4.5%	13 / 0
Claude Opus 5.5	12	53.8%	0.0%	7 / 0

The choice isn't "best model". Claude Opus 5.5 made no confident mistakes once it knew a supplier, at $55.93 per thousand invoices. Claude Sonnet 5.5 gets 98% of those invoices fully right for $29.39. Where a wrong payment is expensive, the difference pays for itself; where volume matters more, it doesn't. We'd measure both on your invoices and show you.

05 — Experiments

How we got here

Good results come from many small experiments, measured honestly — including the ones that didn't work. This is the log from this project, in order. It's the same process we'd run on your workflow.

Claude Haiku 5.5Adopted

How much history should the agent see?

Does showing the supplier's previous invoices and their corrections help, and is one invoice enough?

What we tried. Six memory policies on the cheapest model: none, the previous invoice, the previous two, the previous invoice as text only, and 'one, then two if flagged'. Scored on the same 120 invoices.

What we found. One earlier invoice cut repeat mistakes from 47% to 16%. Two earlier invoices added cost without adding accuracy: the model began copying line items from them. Retrying with two only when the first answer looked doubtful got the most invoices fully right.

Decision. Adopted: memory, at most two invoices, with checks rather than more context.

Policy	Invoices with every field right	Repeat mistakes	Cost per 1,000
No memory	28%	47%	$0.98
Previous invoice	66%	16%	$1.33
Previous two	70%	14%	$1.62
One, retry with two if flagged	77%	21%	$1.85
Previous invoice, text only	55%	20%	$1.08

results/runs/claude-haiku-5-5/*.json

Claude Haiku 5.5Adopted

When a model's answers fail to parse

Why did 9 of the first 12 invoices fail outright?

What we tried. Read the raw responses. Haiku 5.5 wrapped a correct invoice in the wrong envelope: as a string of JSON, with a stray brace, or with an extra field, and repeated it on retry.

What we found. Nothing was misread; the formatting was broken. A narrow repair of exactly those slips, with every field still strictly validated, took failures to zero.

Decision. Adopted in the harness, and flagged for production: switching to this model needs the same repair.

Smoke runs (4 suppliers)

Claude Sonnet 5.5Rejected

History in the prompt, or a tool to look it up?

Is it better to let the agent fetch a supplier's earlier invoices only when it wants them?

What we tried. Gave the agent a lookup tool instead of history in the prompt, then measured the 66 invoices that followed one needing a correction.

What we found. The lookup fixed 51 of 55 invoices that had mistakes and broke none. But the agent looked up on every invoice and never went back further than one, so the tool only added a round trip, doubling the cost.

Decision. Rejected: history goes in the prompt whenever it exists. Same benefit, simpler, cheaper.

Same 66 invoices	No memory	With lookup
Invoices with every field right	17%	94%
Repeat mistakes	75%	5%
Cost per 1,000	$16.65	$33.54

results/runs/claude-sonnet-5-5-medium/lookup-after-correction.json

Claude Sonnet 5.5Adopted

Teaching it a supplier's date format

Why did a corrected day/month swap come back on the next invoice?

What we tried. The prompt told the agent to read ambiguous dates by the supplier's country, which overrode the correction. We made earlier corrections take precedence, and added date checks: due date after invoice date, due date matching 'Net 30', invoice not dated before the supplier's previous one. A failed check sends the answer back once.

What we found. The swapped dates that had survived a correction were fixed. The checks sent 9 of 200 answers back for a re-read.

Decision. Adopted.

Run before the line-items check (since replaced; figures as reported then)

Claude Sonnet 5.5Adopted

Catching a dropped line-item table

Sometimes a whole table of lines came back empty, on clean PDFs too. Can we catch it?

What we tried. If the supplier's previous invoice was itemised and the answer has no lines, send it back once. Made the 'Net 30' check exact for dates that could be read either way.

What we found. On the re-read, two of three dropped tables came back in full and the third in part, and the last date swap was caught. Across all 200 invoices, 99.5% of fields were right.

Decision. Adopted. Next: a check that works on a supplier's first invoice, when there is no history to compare.

results/runs/claude-sonnet-5-5-medium/memory-after-correction.json

Claude Opus 5.5Adopted

Is the bigger model worth twice the price?

Same setup, the next model up. What does the extra cost buy?

What we tried. Ran Claude Opus 5.5 at the same effort, with the same prompt, memory and checks, on the same 200 invoices.

What we found. Once Opus knew a supplier, it made no confident mistakes at all: every error it made was flagged for review, where Sonnet had 15 confidently wrong fields. No Opus answer needed a re-read; 16 of Sonnet's did.

Decision. Both stay: Opus where a wrong payment is expensive, Sonnet where volume matters more.

results/runs/claude-opus-5-5-medium/memory-after-correction.json

06 — For your team

What we'd do for your team

This project is a working example of how we build AI for accounting and bookkeeping: measured, explained, and yours to keep.

Measure on your documents

We build a test set from your real invoices and suppliers, with known answers, so every claim is a number you can check.

Evaluate every new model

When a new model ships, it runs through your test set within days. You switch when it's better for your money, not because it's new.

Tune for value

Prompts, memory and checks are experiments, not guesses. We keep what earns its cost and tell you what we dropped.

Build it into your workflow

Into your inbox, your review queue and your accounting system, with every run and result stored where you can see it.

07 — Method

How we measured

The invoices. 200 synthetic supplier invoices from 40 suppliers, five each in date order, in 11 regional formats and five layouts, as PDFs, office scans and phone photos. Every answer is known by construction. 10 suppliers have no special habit, as a control.

The reviewer. After each invoice, every wrong field is corrected to the true value, and the next invoice from that supplier sees those corrections. Results compare the same invoices read with and without that history.

Scoring. Every field is judged right or wrong against the answer key — amounts to the cent, identifiers exactly, names allowing formatting differences. A confident mistake is a wrong field at 85% confidence or more.

The caveats. The invoices are synthetic, so your suppliers will differ, which is why we measure on yours. Each figure is one run. Runs vary, mostly when a model occasionally drops a whole table of lines: two runs of the same model differed by three points of field accuracy for that reason alone, which is why we keep re-reading and repeat runs in the process. Every run, figure and line of code is kept, and the full results are here.