Shadow mode is a period in which an AI system runs in parallel with the team, on the same real cases, without deciding anything. It is the safest way to test an AI system: at the end, what the system would have done is compared with what people did. Errors surface while they still cost nothing, the team sees the system get its own cases right, and the decision to change the process rests on numbers.
Why not switch the system on from day one
An AI system tested on hand-picked cases can behave differently with real work. Formats change from supplier to supplier, scanned documents come in poor quality, and rare cases appear that nobody thought to include in the tests.
The models themselves have known limits here. In the AI Index 2026 from Stanford University, none of the 15 best models gets above 70% at extracting and calculating values from scanned tax certificates. That does not rule out a good document system, but it means designing it with review on the hard cases, and finding out which ones they are before touching the process.
Switching the system on from day one turns each of those errors into a problem for the team or the customer. In shadow mode, the same error is just a row in a comparison table.
How to set up the shadow period
What goes in
The system receives the same cases as the team, as they arrive: the emails, the invoices and the requests. The team works exactly as before and does not see the system’s output, so it is not influenced by it. The system writes what it would have done to a separate log, with the confidence level of each result.
How long it lasts
A few weeks, with enough volume to cover the normal variety of the process. With hundreds of cases a week, you reach conclusions in two or three weeks. A process with peaks (month-end close, tax season, a campaign) should include at least one peak.
What gets compared
For each case, the system’s result against the team’s result. If they match, the case counts as correct. When they do not, someone who knows the process decides who was right, because sometimes the system is right and the person made the mistake.
At the end, the comparison shows, by type of case, the share correct and the share the system would have sent for review. It also shows the number that weighs most, the errors the system would have made with high confidence.
Measure by group as well as on average
A system can get 95% of cases right on average and fail almost every time in a small group, such as invoices from new suppliers or requests that come in by phone and are typed up by hand. The average hides these groups, and they are exactly the ones that annoy the team most.
That is why shadow-mode results are split by document type, by client and by supplier. When a group fails a lot, you fix the rules for that group, or always send it for review, or leave it out of the system. The choice depends on the group’s volume and on what an error costs, a line of reasoning we develop in where AI fails, and how to decide what to review.
When the system starts working for real
The decision is written down before the shadow period starts, so it is not taken out of enthusiasm or fatigue. It usually has three conditions:
- The share of high-confidence results that are correct passes a value agreed with the client.
- There are no high-confidence errors in the types of case where an error is expensive.
- The volume going to review fits the time the team has.
Once all three are met, the system starts deciding on high-confidence cases, and the rest stay with the team. When they are not, the system is adjusted and the shadow period is repeated for the group that failed, without holding back the groups that already work.
The effect on the team
Shadow mode is also a way of introducing change. The team loses nothing during the period, sees the system get right the cases it handles every day, and takes part in deciding who was right when the two disagree. By the time the system starts working, people already know where it is good and where it needs checking.
This answers one of the most common reasons a system goes unused, which is distrust of the output. The others are in why your team isn’t using the AI system. The shadow-period figures are compared with the baseline measured before automating, and how to design the rules the system applies is in how to automate document data extraction with AI.
Frequently asked questions
What is shadow mode in an AI system?
A period in which the system runs in parallel with the team, on the same real cases, without deciding anything. At the end, what the system would have done is compared with what people did.
How long should it last?
Long enough to cover the normal variety of the process. With hundreds of cases a week, two or three weeks are usually enough. If the process has peaks, the period should include at least one.
Does the team do double work during shadow mode?
No. The team works as before, and the system runs in parallel. The only extra work is deciding who was right when the system and the team disagree.
What happens if the system fails on one type of case?
The rules for that group are fixed, or the group always goes for review, or it stays out of the system. The groups that work can go ahead without waiting for the others.
Putting an AI system to work on a critical process?
Test it in shadow mode first. Book a call and we will design the period, the comparisons and the conditions for going ahead with you.
