Somewhere in the last two years,
"the AI flagged it" quietly became a stand-in for "we
checked." It shows up in incident reports, vendor risk reviews, hiring
decisions, and customer refund logs. It sounds like due diligence. In an actual
audit, it's closer to a shrug, and regulators have started treating it that
way.
The Phrase That Doesn't Hold
Up
"AI approved it"
answers a different question than the one auditors, regulators, and courts
actually ask. They don't want to know what the system did. They want to know
who was accountable for letting it do that, and whether a qualified person
reviewed the outcome before it affected a customer, an employee, or a filing. A
model output is not a control. A person checking that output, on a defined
cadence, with the authority to override it, is a control. Those two things get
conflated constantly, and the gap between them is where compliance programs are
quietly failing.
The Federal Trade Commission has
made clear it isn't interested in the marketing language companies use to
describe their AI. In May 2026, the agency settled with Cox Media Group
and two smaller marketing firms for $930,000 over claims that an
"AI-powered" ad targeting service listened to conversations through
smart devices. It didn't. The service was built on purchased email lists
dressed up in AI language. The lesson for compliance teams isn't about smart
speakers — it's that regulators are willing to dig past the label and test
whether the AI claim actually matches what the system does and who's accountable
for its output.
What "Oversight"
Actually Requires
The default assumption in a lot
of organizations is that keeping a human "in the loop" automatically
satisfies oversight obligations. Researchers at the Harvard Business School AI
Institute recently challenged that assumption directly. Their analysis of
human-AI decision workflows found that organizations can't realistically keep a person
reviewing every AI touchpoint once a system operates at scale, and
that the smarter question isn't whether a human is nominally present, but which
specific decisions actually warrant human review given the volume and stakes
involved. A rubber-stamp review process that exists on paper but never
meaningfully overrides an output isn't oversight. It's documentation of an
unexamined assumption.
This is roughly the same
conclusion built into the government's own guidance. The National Institute of
Standards and Technology's AI Risk Management Framework
is voluntary, but it has become the reference point most auditors and insurers
measure against when they ask whether a company's AI governance is
"reasonable." The framework doesn't ask whether a human touched the
process somewhere. It asks whether the organization can trace an AI-influenced
decision back to a specific person with the authority and information needed to
catch a bad output before it caused harm.
Building an Audit Trail That
Actually Holds Up
For most mid-sized businesses,
closing this gap doesn't require a chief AI officer or a six-figure governance
platform. It requires three things most companies already have the pieces for.
First, write down where AI
touches a consequential decision — hiring, credit, pricing, customer refunds,
medical or legal guidance — and name the specific person accountable for
reviewing that output before it goes final. Not a department. A name.
Second, keep the record. If a
human reviewed an AI recommendation and approved or overrode it, log the date,
the reviewer, and the reasoning in plain language. That log is the difference
between a defensible decision and an unverifiable claim if a regulator or a
plaintiff's attorney ever asks.
Third, test the AI claims you're
making publicly against what the system can actually demonstrate. If your
marketing says a tool "detects fraud with 99% accuracy," someone in
your organization needs to be able to produce the evidence behind that number,
not just the vendor's spec sheet.
None of this is exotic. It's the
same discipline compliance teams have applied to financial controls for
decades, applied to a newer kind of decision-maker. The businesses that get
burned aren't usually the ones using AI aggressively. They're the ones who assumed
a helpful tool and a compliant process were the same thing.
Comments
Post a Comment