Human-in-the-Loop AI Automation: How to Design the Review Queue

Human-in-the-loop automation means the machine acts where it can evidence the answer and declares uncertainty everywhere else. A person watching every transaction defeats the point. What you want is a system that knows the difference between a case it has solved and a case it has guessed, and makes the guessed ones cheap for a human to resolve.

Operations buyers raise this before they raise price. A back-office brief of this kind arrives as a list of workflows: supplier invoice PDFs turned into purchase invoice entries in the ERP, bank payment notification emails reconciled against open invoices, sales orders synced from the ERP into a spreadsheet, delivery planning generated from order delivery dates, and monthly file creation and archiving. Attached to that list is a version of the same requirement: cut the manual data entry, but keep exception handling and manual review wherever the information cannot be reliably identified. That requirement is the specification. Everything below is how you build against it.

The blunter version of the question is what happens when the AI gets an invoice wrong. Claiming it will not is dishonest. Models misread scans, suppliers redesign their templates, and a total obscured by a stamp stays obscured. The answer is a routing rule. Confident, reconciled cases post automatically. Everything else lands in a review queue with the document and the machine's best guess side by side. For the wider picture of which back-office processes are worth automating at all, we covered that in the back-office automation guide.

Score confidence per field. A single invoice can have a certain supplier, a certain invoice number, a certain net total, and a VAT line the model is guessing at. Document-level scoring gets this wrong in both directions: it throws a clean document into review because of one bad field, or it posts the bad field because the other nine were clean.

Two signals decide the routing, and they are not equally trustworthy. The model's own reported confidence is useful but soft. Deterministic checks are hard: line items sum to the net total, VAT arithmetic holds, the supplier matches a record you already hold, the invoice number has not been seen before, the purchase order exists and its amounts match. Which checks apply depends on the document type, and the field-by-field checks that hold on a supplier invoice are the set most back-office builds start from. Arithmetic is the cheapest lie detector in the system, and it costs nothing to run on every document.

We apply the same discipline to our own tooling. The free AI-readiness check on our homepage runs eight deterministic checks with no LLM scoring anywhere in the pipeline, so two runs of the same page return the same result. Where a rule can decide, do not ask a model. Reserve the model for the part that genuinely needs reading comprehension.

Thresholds come from measurement. Run the system in shadow mode for the first weeks: the machine proposes, a person decides everything, and the system logs agreement per field. After a few hundred documents you know which fields the model gets right often enough to release and which ones stay in review. That log is also the baseline for every accuracy claim you make afterwards.

Some cases route to a person regardless of confidence, because the cost of being wrong is not symmetric. A missed automation opportunity costs a few minutes. A wrongly posted payment costs money and trust, and takes far longer to unwind than it took to create.

The standing list is short and worth writing into the specification before any code exists. The first document from a counterparty you have no record of. Any document where the payment details differ from what you already hold for that supplier. Amounts above the approval limit the company already uses for human sign-off, because the automation should inherit the policy rather than quietly bypass it. Credit notes and anything that reverses or amends an earlier posting. Suspected duplicates, where the same supplier and amount appeared recently under a different invoice number. Documents that arrive through a channel the workflow does not normally use.

Write the list into the specification as business rules. Thresholds move as the system learns. This list stays fixed until the business itself decides to change it.

A review item should be resolvable on one screen, with the source document on one side and the machine's proposal on the other. Every second the reviewer spends switching tabs, opening a PDF from a mail client or logging into the ERP to check something is a second that makes the queue feel like work, and queues that feel like work stop getting emptied.

Each proposed field carries a pointer back to where it was read: the page and the region of the document. Highlight it. The reviewer's eye should land straight on the evidence. Show, in plain words, which field triggered the review and why, so the person knows what they are being asked to judge rather than re-checking every field on the document out of caution.

Give exactly three actions: approve, correct a field, or reject with a reason code. Keep it usable from the keyboard, because a reviewer clearing forty items with a mouse will not stay enthusiastic. The reason codes matter more than they look: they are the data that later turns an exception into a rule.

One design rule here comes from a different part of our work. When we run blind AI visibility checks, we never name the brand in the question, because naming it produces false positives: the model repeats what you fed it. Human reviewers have the same failure mode. A pre-filled form shown without the source document gets confirmed rather than checked. People agree with whatever is already in the box. Put the evidence next to the guess, or you have built a rubber stamp with an audit log.

Group similar exceptions together. Twenty invoices from one supplier where the same field failed is one decision repeated twenty times, and a reviewer who sees them as a batch resolves them far faster than the same twenty scattered through a chronological list.

The audit trail exists to answer one question months later: why does this posting look like this. Record it per field. Store the extracted value, where in the document it came from, which model and version produced it, the reported confidence, which deterministic checks passed and which failed, whether a person touched it, who, when, and what the value was before they changed it.

Make the log append-only. A correction adds a record; it never overwrites the original reading. The history is what lets you answer an auditor without re-reading a year of invoices, and it is also what lets you find every affected posting quickly when a supplier turns out to have changed a template three months ago.

The second use is calibration. Every correction is a labelled example that tells you which field the model gets wrong and on which document type. Collect enough of them and you have per-field agreement rates, which is the only defensible ground for releasing a threshold.

Corrections are what shrink the queue. The model does not improve on its own. Three mechanisms do the work, and all three depend on the review screen capturing structured corrections instead of free text.

The first is per-counterparty memory. Once a person has confirmed where a given supplier prints its invoice number and how it formats dates, that layout is stored and the next document from that supplier arrives close to solved. The second is rule promotion: a reason code that appears twenty times is a deterministic rule waiting to be written, and a rule runs faster and returns the same answer every time. The third is recalibration. Fields where people agreed with the machine across the last few hundred documents get their threshold released. Fields where people keep correcting stay in review, permanently if that is what the data says.

Resist quoting an automation rate before the shadow period has produced one. Document variety decides it more than model quality does. A company buying from thirty regular suppliers converges quickly. A company with a long tail of one-off vendors will keep a bigger queue, and should.

Keep measuring after go-live. Sample the automatically posted entries and re-check them by hand, because an empty queue and a correct queue are different things. The harder half of that habit is reading the result plainly when it is unflattering. On aiasemu.ee, the lawn-care business we run on our own automation, where AI measures a lawn from a satellite photo and prices the job in seconds, one week of PostHog data recorded 57 address searches, 39 refined estimates, 9 bookings started and 2 completed. Our reading of that is a site working as a lead machine more than as a booking machine. The same discipline is what shrinks a review queue: measure, say what the number actually shows, then change the design.

Start with one workflow that runs often, follows rules, and irritates someone every week. Supplier invoice entry is usually the strongest first candidate: high volume, a clear correct answer, and an ERP API to post into. Matching bank payment notifications against open invoices is a close second, because bank notification emails from a given bank follow a consistent shape, which gives the matching logic something stable to work from, and the exceptions, partial payments and missing references, are exactly the cases a person should see.

You almost never need to replace the systems you already run. Common small-business ERP systems expose APIs, and the automation is built around them. A well-defined single workflow is typically a one to three week build, and what a build of that size costs turns mostly on how varied the documents are. Larger programmes covering several workflows are quoted per project, once we have seen the actual document set rather than a description of it.

Build the review queue in the same sprint as the extraction, never after it. A queue added at the end becomes a second inbox: no owner, no service level, no keyboard shortcuts, and within a month everyone approves in bulk without looking. Name the owner, agree how quickly items must be cleared, and put the queue depth on a dashboard someone actually sees.

If the internal objection is that the company is too small for this, the answer is the single-workflow start: automate one process, measure the hours it returns, then decide whether to extend. It also settles the scoping question operations buyers raise before price, which is whether the partner knows their specific systems. Send the process list with the systems named and the volumes attached, and the conversation starts with facts. The wider argument for why machine-readable operations matter beyond the back office is in our guide on AI recommendations.