Human-in-the-loop oversight crucial for enterprise AI: Experts Explain

ZDNET’s key takeaways

• Courts and agencies demand accountability for flawed enterprise AI.
• AI-native software is adding human escalation to complex workflows.
• Four experts explain how human-in-the-loop AI oversight is evolving.

In February 2025, the FTC finalized a $193,000 settlement against DoNotPay, the platform that claimed to be the world’s first robot lawyer. According to the FTC, DoNotPay didn’t ever test its output against any advice a licensed attorney would produce. DoNotPay was also hit with a class-action lawsuit almost immediately after it went live for offering unauthorized legal services without a bar license in California.

With leading AI companies making bold and outlandish statements about what their platforms can do, the chasm between propaganda and outcomes is a matter of some debate. It also makes for murky legal ground because a lot of these complications don’t have existing legal precedent.

When you’re tasked with signing off on recurring purchases for a large corporation, this is the kind of gray area that executives try to avoid. This has created a market gap for responsibly designed AI systems that offer a clear chain of accountability, which a new category of software is rushing to fill. ZDNET spoke to four leading experts engaged in studying or designing such systems to talk about the underlying technology and whether the safety they promise holds up in real life.

What is human-in-the-loop (HITL)?

Human-in-the-loop (HITL) is an emerging design pattern among AI platforms where autonomous agents route complex decisions through human review before executing a task or generating a response.

It’s basically a system that’s designed to create an accountability chain for sound decision-making in corporate settings, especially in sensitive matters like healthcare operations, regulatory compliance reviews, high-value financial transactions, or legal decision-making. But the definition is rather broad, which means companies use this term fluidly for a lot of different systems and architectures.

For example, a SaaS vendor that performs periodic security audits on its AI system can claim it has human oversight, but that doesn’t mean there’s a human in the loop at all times during crucial tasks. Lately, however, a growing number of companies are baking in real-time human oversight as an architectural stopgap, especially if they serve IT and DevOps, vibe coding workflows, or executive decisions in regulated industries like healthcare or finance.

“The model is one component in a workflow,” explained Akash Thakur, an SRE architect and AI reliability engineer based in Canada. He believes that the bigger problem lies in how most systems handle an AI model being uncertain or wrong, not the model itself. Instead of treating AI model failures as a worst-case scenario, HITL serves as a real-time auditing system that catches the model when it’s likely to make a mistake — before a customer or regulator finds out the hard way.

The things most AI routing systems get wrong

The most common flaw among HITL systems is treating the model’s own confidence score as the sole trigger for escalating a response for human review. Daniel Gamber, CEO of the AI document processing platform Cambrion, says that confidence scores measure the wrong thing entirely.

“A confidence score tells you the machine could read the text,” he explained. “It tells you nothing about whether the number is actually right.”

Thakur described the same reliability blind spot, emphasizing that “model confidence isn’t the same as being correct.” If a model is confident even when it is wrong, it will always sail under the threshold that sets off a human review, letting mistakes fly past without the tiniest red flag. He also added that human reviews can be expensive in their own right, but they can’t insure you against legal or compliance issues if they aren’t implemented properly.

Even when a system correctly flags an AI response as needing review, the routing process that decides how to address the flag is often too binary, said Asim Husain, co-founder of Alterion and a former VP of engineering at Google. In fact, he described four types of error responses that HITL systems should plan for once an AI workflow has crossed a defined threshold:

• Notify someone and let it through,
• Mask the sensitive part and proceed,
• Hold for approval, and
• Quarantine or kill the session outright.

The ideal response depends on an organization’s risk tolerance, which also needs to be accounted for in the design. Beyond response severity, the system should also know who to escalate to, Husain added. “A destructive database mutation should land with the platform or security team that owns that system. A financial transaction above a threshold routes to whoever owns transaction controls.”

Bias, audit trails, and the tangled web of compliance

Through all these conversations, another common thread emerged. “If a human ‘reviewer’ has never once overturned the system, that’s not oversight,” said Thakur. “A human gets placed at the end of an automated process to ‘review’ decisions, but they’re reviewing hundreds of items with no real time or context to judge any of them, so they approve by default.”

Even when an HITL system correctly escalates AI workflows to the right professional, there’s a considerable chance that the person reviewing the output is already biased in favor of the AI, approving responses with no more than a cursory glance in the interest of expediency.

The EU Artificial Intelligence Act, which also happens to be the most well-developed piece of modern legislation in its category, makes demonstrable oversight a core requirement for compliance.

“When high-risk AI systems fulfil the essential requirements of a regulation of the European Parliament and of the Council on horizontal cybersecurity requirements for products with digital elements, they should be deemed compliant with the cybersecurity requirements set out in this Regulation in so far as the achievement of those requirements is demonstrated in the EU declaration.”

“’Demonstrable’ is doing a lot of work in that phrase,” Husain added, saying that an organization only clears its compliance requirements if it can prove in a court of law or before a regulatory agency that it has implemented real oversight with a clear audit trail of physical evidence. That means training human reviewers to challenge AI output when it could be wrong while also maintaining detailed records of the human review process that decides whether the output needs to be altered or it’s fit to be submitted as is.

Sometimes, however, the problem is the exact opposite. A solid human review protocol, even when coupled with good escalation triggers and routing logic, may still be held back by the bias inherited by the model itself during training.

For example, according to a 2024 Government Accountability Office report on the IRS’s automated audit-selection system, Black taxpayers were audited three to five times more often than non-Black taxpayers, even though there was no direct race input field in the system data.

While the exact reason is still unknown, a deeper review by Stanford researchers suggested that the disparity likely emerged from aggressive scrutinization of the earned income tax credit claimed by low- to middle-income families and the practice of self-filing tax returns. The IRS acknowledged the issue in its 2024 annual report, vowing to overhaul its systems to address this bias.

Developing an HITL solution that holds up to scrutiny

To understand what successful HITL design looks like, I decided to look through some popular SaaS platforms that are praised for implementing a more thorough version of this system with no major allegations of bias in their recent history.

Nominal is an automated accounts payable platform that pulls data from your business ledger, parses through it using a proprietary model with custom logic, and waits for human approval before any actual transactions can take place. The key difference in its implementation is that human approval is a mandatory requirement for transaction clearance, with artificial intelligence existing only to provide a layer of context for the human reviewer instead of executing fund transfers on autopilot.

“AI accelerates capability, not accountability,” said Eric Vaughan, CEO of IgniteTech, an AI and SaaS management company for large corporations. “Models do not like to say they do not know. They lean toward answering.”

Vaughan’s solution, which he has implemented in his products MyPersonas and Eloquens AI, is grounding — an explicit instruction that tells the model to escalate for human oversight unless the answer is already included in the company’s existing knowledge base.

Instead of trying to satisfy the user by giving them whatever response they are looking for, Vaughan thinks AI should be confined to a defined body of knowledge and refuse to answer unless the required information is explicitly present in that existing body. He also measures the entire product against a single core metric, which is called grounding accuracy. “Our grounding accuracy runs in the mid 90s, we have worked very hard on that.”

Gamber’s company takes a similar approach by escalating documents for review when a value can’t be grounded in the source or fails an internal consistency check. “We escalate on whether we can actually ground the value, not on how the model feels about it,” he explained. He offered examples where this approach has proven useful, like when a document has inconsistent dates, a missing signature, or tax calculations that just don’t add up.

Once the AI is grounded to a defined data source, the next step is to set up escalation to match the organization’s internal risk thresholds. “A $1,000 transaction limit means something completely different to a regional bank than to a Fortune 50 retailer,” Husain explained.

Innovaccer Gravity, a platform that automates benefits administration for medical care providers, does something similar by letting its customers decide how each checkpoint in the oversight loop is set up. For example, ambulatory care centers can insert staff reviews at any point in the referral-to-treatment chain instead of being forced to inherit a fixed model that they cannot control.

Grounding, mandatory approval, intelligent routing systems, and tamper-evident audit trails all exist to ultimately solve the same pain points, but for varying organizational sensibilities. “Every one of these features exists to keep a specific person accountable for every consequential decision,” Vaughan offered. While risk thresholds vary from company to company, the things that could benefit from human oversight generally do not.