AI judges are good, not good enough to decide alone
Many compliance AI tools use a language model to judge whether evidence meets a requirement: in effect, “LLM-as-a-judge.” The research is encouraging and cautionary at once.
- Agreement is high. The original MT-Bench study found GPT-4 as a judge achieved “over 80% agreement” with human preferences, the same level as agreement between humans (Zheng et al., 2023).
- Biases are real. The same study examined “position, verbosity, and self-enhancement biases.” A later study identified 12 bias types and found “significant biases persist in certain specific tasks” (Ye et al., 2024).
- Reliability is unsolved. A 2024 survey concludes that “ensuring the reliability of LLM-as-a-Judge systems remains a significant challenge” (Gu et al.).
In a compliance setting, a 20% disagreement rate is not a rounding error. It is the share of findings a human must catch.
Humans have biases too: automation bias
Adding a human reviewer does not help if the reviewer simply accepts the AI’s answer. A systematic review in JAMIA defines automation bias as “the tendency to over-rely on automation” and notes “there is often a failure to recognize the new errors” automated systems introduce, with workload, task complexity and time pressure as key mediators (Goddard et al., JAMIA 2012). NIST’s Generative AI Profile likewise warns that “humans may over-rely on GAI systems” (NIST AI 600-1).
What regulation expects in 2026
- EU AI Act Article 14. Human oversight “shall aim to prevent or minimise the risks to health, safety or fundamental rights,” and overseers must “remain aware of the possible tendency of automatically relying or over-relying on the output” (Article 14). The Digital Omnibus on AI, in force since July 27, 2026, moves stand-alone high-risk obligations from August 2, 2026 to December 2, 2027 (Lewis Silkin).
- FDA. For device quality systems, FDA’s Computer Software Assurance guidance applies its risk-based approach to “artificial intelligence/machine learning tools” (FDA).
- NIST. The Generative AI Profile lists human-AI configuration as a core risk area (NIST AI 600-1).
Designing oversight that actually works
| Pattern | How it helps | Counters |
|---|---|---|
| Evidence before verdict | Show the cited passage first, then the AI’s judgment | Automation bias: reviewers judge the evidence, not the label |
| Risk-tiered review | Full review for high-risk requirements; sampling for low-risk ones | Workload, a key mediator of automation bias |
| Mandatory reason on override | Reviewers record why they accept or reject | Rubber-stamping; creates audit trail |
| Calibration sets | Periodically insert cases with known answers | Silent drift in AI or reviewer accuracy |
| Position and length checks | Vary the order and length of evidence in evaluations | LLM judge position and verbosity biases |
| Disagreement tracking | Monitor AI-human disagreement by requirement type | Shows where AI should not be trusted |
Checklist for human-in-the-loop compliance AI
- Decision rights defined: AI recommends, a named person decides
- Every AI finding cites its source passage
- Review depth set by the risk of the requirement
- Reviewers trained on automation bias and the system’s known weaknesses
- Overrides and reasons logged
- AI-human agreement measured on a fixed test set and after any model change
- Escalation path for disagreements and uncertain cases
- Oversight design documented for regulators and auditors
Built for human decisions
AskDona is designed for this pattern: it shows the passage behind every finding before the verdict, so your reviewers judge the evidence and make the call.
Run a sample document assessment and compare its findings with your team’s own judgments.
FAQ
What is LLM-as-a-judge? Using a language model to evaluate whether an answer or piece of evidence meets a standard, instead of, or alongside, a human judge.
How accurate are LLM judges? In the MT-Bench study, GPT-4 reached “over 80% agreement” with human preferences (Zheng et al.), but later research documents persistent biases (Ye et al.).
What is automation bias? “The tendency to over-rely on automation” (Goddard et al.), which can make human review ineffective.
When do EU AI Act high-risk obligations apply? After the 2026 Digital Omnibus, stand-alone high-risk systems apply from December 2, 2027 and product-embedded systems from August 2, 2028 (Lewis Silkin). Confirm against the official text on EUR-Lex.
References
All pages opened and checked by October 9, 2026.
| Source | Type | Date | Used for |
|---|---|---|---|
| Zheng et al.: Judging LLM-as-a-Judge | Preprint | Jun 9, 2023 | 80% agreement, biases |
| Ye et al.: Justice or Prejudice? | Preprint | Oct 2024 | 12 biases |
| Gu et al.: Survey on LLM-as-a-Judge | Preprint | Nov 23, 2024 | Reliability challenge |
| Goddard et al.: Automation bias (JAMIA) | Peer-reviewed | Jan 2012 | Automation bias |
| NIST AI 600-1 | Standard | Jul 2024 | Over-reliance risk |
| EU AI Act Article 14 (unofficial copy) | Regulation text | 2024 | Human oversight |
| Lewis Silkin: Digital Omnibus on AI | Law firm | Jul 27, 2026 | New high-risk dates |
| FDA: CSA guidance (HHS copy) | Regulator guidance | Feb 3, 2026 | AI tools in quality systems |