What are friends for?
There’s an old joke in maths about the hierarchy of purity, immortalised by G.H. Hardy’s A Mathematician’s Apology: the best mathematics is pure, abstract, and unfettered by constraints. Work in industry and you get the dunce hat. I’m still earning mine in AI consulting, a market I’d sum up in four words: high adoption, low maturity (Gartner, 2024, 2025). In healthcare and finance that gap bites, because when an AI makes a call, the people on the receiving end, credit officers or doctors, need to trust it and to understand it.
xkcd 435, “Purity” by Randall Munroe, licensed under CC BY-NC 2.5.
One perk of the dunce hat is getting to read the research on exactly this problem. It also helps to know the right people, like my old classmate John’s partner, Yueqing “Chloe” Xuan, whose work on Explainable AI (XAI) includes a sharply titled paper: “Comprehension Is a Double-Edged Sword” (Xuan et al., 2025).
The Catch
Say you’re building an AI that decides who qualifies for a loan. What does it take for the officer signing off to trust it? Usually, understanding: we believe a decision once the logic makes sense to us. Chloe argues that’s only half the test. Real understanding also means knowing what an explanation leaves out, and resisting the urge to fill that silence with an assumption.
So how do you measure a wrong assumption? Her team put 200 people in front of AI explanations and, for each, asked them to judge a claim as true, false, or “cannot tell”:
Pick true or false on a claim that was never actually stated, and you’ve quietly filled a gap the explanation never closed. Every answer also carried a confidence rating, to see whether the wrong readers at least knew they were guessing.
Confidently Wrong
Two findings matter here.
First, people read what an explanation says far better than they notice what it leaves out. Handed a clear explanation, they over-read it, filling gaps with plausible but unsupported assumptions. And counterintuitively, the easier it was to follow, the more this happened. “Easy to follow” and “easy to misread” are close cousins.
Second, they were confident in the wrong places. If anything, the people who got it wrong were surer of themselves than the people who got it right.
That’s the double-edged sword: a clear explanation earns trust, then quietly oversells itself. The lesson we can use is that testing whether people understand an explanation isn’t enough. You have to test what they wrongly assume.
Where This Leaves Us
“Understood” and “safely understood” are not the same claim, and usually only one of them gets measured. We treat a clear explanation as a solved problem, when clarity is the very thing that lets a wrong reading pass unchallenged.
That gap bites hardest where we can least afford it. In lending, hiring, and healthcare, an explanation isn’t a courtesy, it’s increasingly the law. The GDPR obliges “meaningful information about the logic involved” in an automated decision (European Parliament and Council of the EU, 2016), and Canada’s proposed AIDA points the same way (Government of Canada, 2023). But a rule that demands an explanation quietly assumes the explanation will be read correctly. Chloe’s work is a warning that the assumption isn’t free: the more fluent the explanation we write to satisfy the rule, the more room it leaves to be confidently misread.
So the next time you sign off on a system because its explanations “make sense”, notice that you’ve only tested the easy half. The hard half is what resurfaces later, as a complaint, an appeal, or a regulator’s question. That half is still being worked out, which is where you come in.
References
European Parliament and Council of the European Union. (2016). Regulation (EU) 2016/679 (General Data Protection Regulation), Articles 13-15 and 22. Official Journal of the European Union, L119. https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:02016R0679-20160504
Gartner. (2024, May 7). Gartner survey finds generative AI is now the most frequently deployed AI solution in organizations. https://www.gartner.com/en/newsroom/press-releases/2024-05-07-gartner-survey-finds-generative-ai-is-now-the-most-frequently-deployed-ai-solution-in-organizations
Gartner. (2025, June 30). Gartner survey finds 45% of organizations with high AI maturity keep AI projects operational for at least three years. https://www.gartner.com/en/newsroom/press-releases/2025-06-30-gartner-survey-finds-forty-five-percent-of-organizations-with-high-artificial-intelligence-maturity-keep-artificial-intelligence-projects-operational-for-at-least-three-years
Government of Canada. (2023). The Artificial Intelligence and Data Act (AIDA) - Companion document. Innovation, Science and Economic Development Canada. https://ised-isde.canada.ca/site/innovation-better-canada/en/artificial-intelligence-and-data-act-aida-companion-document
Hardy, G. H. (1940). A Mathematician’s Apology. Cambridge University Press.
Xuan, Y., Small, E., Sokol, K., Hettiachchi, D., & Sanderson, M. (2025). Comprehension is a double-edged sword: Over-interpreting unspecified information in intelligible machine learning explanations. International Journal of Human-Computer Studies, 193, 103376. https://doi.org/10.1016/j.ijhcs.2024.103376