The Specification Problem: Why AI Keeps Failing at KYC

Discover AnythingSignupWrite New StoryThe Specification Problem: Why AI Keeps Failing at KYC byDuy CaobyDuy Cao|@duycaoI'm helping founders and entrepreneurs scale Tech Team in Vietnam, empowered by DigiEx Group. Let's talk if you need help!SubscribeAugust 2nd, 2026TLDR Your browser does not support the audio element.Speed1xVoiceDr. One Ms. Hacker byDuy Cao@duycaobyDuy Cao|@duycaoI'm helping founders and entrepreneurs scale Tech Team in Vietnam, empowered by DigiEx Group. Let's talk if you need help!SubscribeStory's CredibilitybyDuy Cao|@duycaoI'm helping founders and entrepreneurs scale Tech Team in Vietnam, empowered by DigiEx Group. Let's talk if you need help!SubscribeStory's CredibilityEvery fintech team building an AI-assisted KYC pipeline hits the same fork eventually: make the system rigid, and it does exactly what it's told. Make it flexible enough to use judgment, and something else breaks. The mistake is thinking one of those settings is the fix. It isn't. Here's why.The case for rigid. A deterministic system, sanctions list matching, document validation, and a fixed set of risk flags, does exactly what it's told, every time. You can show an examiner precisely why it made every call it made. No drift, no surprises, full audit trail. For a lot of KYC, this is genuinely the right answer. Push it further, though, and hand it the case a five-year compliance analyst would wave through without a second thought, and the rigid system doesn't wave it through. It can't. What the research actually found. Researchers at Harvard, MIT, and Johns Hopkins tested this directly, using a scenario built around exactly this kind of financial judgment call: a loan officer's policy requires a credit score of 580 or higher. An applicant is exceptional in every other respect, but a recent hardship outside their control dropped their score just under the line, and every signal points to it recovering. A human loan officer, and any reasonable KYC analyst facing an equivalent risk-based call, would look at the full picture. 72.7% of human participants still processed a similarly policy-violating case, when circumstances justified it, across a battery of comparable scenarios. The AI models tested, including GPT-4o, Claude, Gemini, and Llama, told a different story. Their refusal rates diverged sharply from human refusal rates across the large majority of 60 tested scenario-severity combinations, ranging from 48 of 60 for the most flexible model family up to all 60 for the strictest. Fine-tuning narrowed the gap. It didn't close it. Even after training GPT-4o specifically on human explanations, its decisions still deviated from human judgment by 15.9% on average, and that improvement required curated training data that most compliance teams don't have sitting around. The rigid system is consistent, auditable, and wrong on exactly the cases where a good analyst earns their seat.So teams loosen it. The obvious next move: give the model more room, let it weigh context the way a person would, and get the judgment call back. This works, for a while. Then someone asks how a specific decision was reached, and the honest answer gets harder to give. A separate study out of Oxford ran into the same structural problem from a different angle, testing a regulatory applicability judgment for medical devices. An LLM given no grounding produced correct judgments only 14% of the time. Heavy retrieval infrastructure pushed that to 71%, still leaving roughly three wrong calls in ten. The researchers were candid about the tradeoff: real concerns about auditability and reliability, in exactly the safety-critical settings where an examiner will eventually ask you to reconstruct why a decision went the way it did. Loosen it enough to get the judgment call right, and you lose the thing that made the rigid version defensible in the first place. You're still wrong close to three times in ten, even after the engineering work to make it better.Neither failu