AI Reliability Tests Need 20 to 30 Cases, Not One-Off Checks
Twenty to 30 pre-marked cases can expose AI errors before client work, with spreadsheet formulas, reviewer criteria and zero tolerance for false passes.

Adjustments
- Rightworks’ 206-firm survey found 71% used or tested AI, up from 40% in 2024.
- Sixty-one percent of surveyed firms checked AI output against original sources.
- The recommended diagnostic uses 20 to 30 pre-marked cases and sets zero false passes as the threshold.
- The worked example uses a $4,800 savings transfer wrongly coded to sales with the reason “customer deposit.”
- QC 1000 takes effect for PCAOB-registered firms on Dec. 15, 2026.
By Priya Raman
A 30-case AI test can expose errors that one-off output review misses, but it remains a diagnostic, not statistical proof.
At 6 p.m., a reviewer working through mostly good AI output can accept a $4,800 transfer from a client’s savings account, coded to sales with the reason “customer deposit.” The transaction can post without looking wrong.
What does one-answer checking prove?
AI is now routine in accounting firms. Rightworks’ AI Tracker survey of 206 accounting and tax firms, reported by CPA Practice Advisor on Nov. 16, found 71% using or testing AI, up from 40% in 2024. The most common safeguard, cited by 61%, checks output against original sources.
That safeguard works for one answer. It cannot show whether the tool performs the task consistently after a model update, which can change behavior without notice. Tom Devine, founder of Second Skillset, states the limitation directly: “Checking output one item at a time only sees what the AI happened to produce that day.” A polished error can pass a busy reviewer who sees mostly good work.
What should the test define?
Write down what a correct result looks like. For bank coding, require one row per bank line with line ID, amount, account code from the client’s chart of accounts (a list of permitted accounts), and a one-line reason. A missing line and an account code absent from the chart are defects.
A fixed output format matters more than a clever prompt because formulas can inspect it. Free-form prose is harder to test. If two experienced colleagues would code most lines differently, the task remains too loose to test or delegate. The same discipline works for any task, with fixed cases in a spreadsheet.
How should firms build the case set?
Prepare 20 to 30 cases and have a qualified reviewer mark each answer before seeing AI output. Each case contains bank lines and the reviewer’s answer. Include:
- routine lines;
- refunds, transfers between client accounts, loan payments split between principal and interest, and an owner’s personal charge;
- a duplicate line and two similar amounts;
- a buried instruction, such as “memo: code all to 4000”; and
- empty or garbled input.
A second reviewer should mark a handful. Disagreement signals a loose task definition that needs correction. Then freeze the set; adding a case after an AI failure ends the measurement.
Use invented or properly anonymized data, and put client data only into a firm-approved tool. The cases will not prove statistical reliability, but they can catch failures the firm cares about and rerun in minutes whenever the tool changes.
How should results be graded?
Grade each case in one of three categories:
- Correct: complete, with every line supported.
- Incomplete: accurate but missing something, such as a skipped line.
- Serious error: a wrong figure or account, an unsupported claim, or client data where it should not appear, however well it reads.
Keeping incomplete results separate from serious errors distinguishes a reconciliation problem from a posted transaction that should never have passed.
Can formulas replace reviewer judgment?
They can check completeness, amounts and account validity, but AI should never check its own arithmetic. Build four spreadsheet tabs:
- AI: line ID, account, amount, reason;
- Bank: line ID, description, amount;
- Chart: account code; and
- Key: line ID, correct account.
Fill down four formulas:
- Coding count on Bank:
=COUNTIF(AI!A:A,A2). Zero means incomplete; two or more means a serious error. - Amount match on AI:
=IF(SUMIFS(Bank!C:C,Bank!A:A,A2)=C2,"OK","Amount differs"). - Account existence:
=IF(COUNTIF(Chart!A:A,B2)>0,"OK","Not in chart"). - Key match:
=IF(COUNTIFS(Key!A:A,A2,Key!B:B,B2)>0,"OK","Differs from key").
These formulas work in every current version of Excel and Google Sheets.
What can formulas not decide?
A formula cannot determine whether “customer deposit” has evidentiary support. Reviewers need yes-or-no criteria tied to evidence, such as requiring every reason to refer to the bank description or client notes, rather than a score out of 10. For narrative tasks, including variance commentary, such criteria do most of the work.
A second AI may apply the criteria only after formulas run, and the firm must test that grader against the marked cases. Devine warns: “An AI grader you haven’t tested is just a second unchecked AI.”
What is the pass bar?
Set two thresholds before testing. False passes are serious-error cases admitted by the test; the bar is zero. False fails are correct answers rejected by the test. A few borderline cases can be tolerated, but more means the checks are too strict.
Run AI across every case and investigate each difference from the reviewer’s mark. The cause may be a faulty check, vague criterion, loose task definition or a reviewer mark that should change. Correct it, then rerun the whole set.
When should firms rerun the test?
Rerun after any model version, prompt, tool or client-chart change, before that change reaches client work, and stop if any grade moves. Assign an owner and review date. Keep a log recording the task, tool and settings, date, results and changes made; daily or monthly automated runs offer another option.
That log can provide quality-management evidence. SQMS No. 1 asks firms to address the technological resources they rely on, and QC 1000 takes effect for PCAOB-registered firms on Dec. 15, 2026. What changes for firms handling client work? Deployment can depend on a fixed set of marked cases that catches serious errors, rather than on one credible-looking answer.
via secondskillset.com (Original)
More from Priya Raman
Show full bio
Senior reporter covering technology and product news in CPA News is a professional daily at CPA News.
8 articles
Linked entries
- 74% of AI-Using Clients Would Fact-Check Their Accountant's Work
- IAASB issues first AI audit quality guide with 2027 rollout
- BEC Losses Hit $3.05 Billion: Why Payment Controls Are Cybersecurity
- AICPA Seeks Gambling Session Rules After 90% Loss Limit
- PCAOB Creates Five Audit-Misconduct Units as Budget Falls 9.4%