Governance

Human-in-the-loop AI: when an agent should ask before it acts

Four situations where an AI agent should stop and ask a person first, what the person should see, and how to keep the queue short enough to stay careful.

By The Cadrix team6 min read
Hands resting on a laptop keyboard at a bright table, about to approve or stop
Photo by Selina on Unsplash

"Human in the loop" is one of those phrases everyone agrees with and few people define. Put a person everywhere and the automation is pointless. Put a person nowhere and one bad run can email a customer, move money or overwrite a record.

The useful question is narrower: for this particular step, what happens if the agent is wrong, and can we undo it? That question tells you where a person belongs. This post gives you four triggers that work in practice, a way to decide what the person sees, and a common mistake to avoid.

What the phrase should mean#

A human in the loop is someone with the authority and the information to say no before an action takes effect. That is different from a person who gets a notification afterwards, and different from a person who clicks "approve" on a screen that shows them nothing.

Two things have to be true for the oversight to be real. The decision has to come before the action, and the person has to be able to disagree without breaking the workflow. If rejecting a request causes an error, people will stop rejecting.

Four triggers that justify a pause#

You do not need a long policy to start. Most useful approval steps fall into one of these four groups.

1. Money moves. Refunds, payments, credits, discounts, purchase orders. Agents are good at reading a request and preparing the action. They are not accountable for the budget. A threshold works well here: small amounts go straight through, larger ones wait for a person.

2. Something leaves the building. An email to a customer, a message on WhatsApp, a post, a document shared with an outside party. You cannot unsend these. Internal drafts are different: if a person edits the draft anyway, an extra approval just adds delay.

3. A system of record changes. Updating a customer's bank details, closing an account, changing a status in your finance or HR system. The cost of a wrong write is often discovered weeks later.

4. The agent is unsure, or the input is odd. A request that does not match any known pattern, a document the extraction step could not read cleanly, a classification where the top two choices are close. Here the pause is not about the action. It is about the agent admitting it is guessing.

A fifth category exists, which is "anything the law says a person must review", and we come back to it below. Everything else, such as reading, searching, summarising and drafting, can usually run without a pause.

Not every run needs a person#

The mistake we see most is approving every run. It feels safe. It also produces a queue that nobody can keep up with, and the approver starts clicking through. The EU AI Act names this risk directly: for high-risk systems, the people overseeing them must be enabled to stay aware of the tendency to rely too heavily on the system's output, which the Act calls automation bias (Article 14(4)(b)).

You can avoid it by sorting runs before they reach a person. Put a condition ahead of the approval step. Cases that are clearly fine continue. Cases that are large, unusual or low-confidence go to a human. The aim is an approval queue small enough that each item gets read.

What the person should see#

A pause is only as good as the context attached to it. Before you build the step, write down what you would need to know to say yes with a straight face. Usually that is:

  • The exact action, in plain words: "Refund 84.00 to the card ending 4417."
  • The reason the workflow wants to do it, and the input it is based on.
  • The thing the agent was unsure about, stated rather than hidden.
  • The choices on offer: approve, edit, or reject with a reason.

Editing matters more than people expect. If the only options are yes and no, a nearly correct draft gets rejected and someone redoes the work by hand. Letting the approver fix one line and continue is faster and teaches you where the agent is weak.

What the rules and standards say#

You do not have to be in a regulated sector to borrow the idea. Two public references are worth knowing.

The Model Context Protocol specification, which many agent tools use to connect to outside systems, says there "SHOULD always be a human in the loop with the ability to deny tool invocations", and that clients should ask for confirmation on sensitive operations. OWASP's guidance on prompt injection lists "require human approval for high-risk actions" among its mitigations, next to least privilege and separating untrusted content.

For regulated use, the EU AI Act requires high-risk AI systems to be designed so that people can oversee them effectively, including being able to override or reverse an output and to stop the system (Article 14). Whether a given workflow counts as "high-risk" depends on its use, not on the technology, and the application dates for those obligations were changed in 2026. If you think your use case is covered, check the current text and ask your legal adviser. This post is not legal advice.

Building it in Cadrix#

Here is a refund workflow built on this pattern. It uses only features that exist today.

  1. Start the workflow from a message or an API call that carries the customer's request and order reference.
  2. Agent reads the request, looks up the order through a web request or a tool you have connected, and prepares a proposed refund with a short reason.
  3. Condition compares the amount with your limit. Under the limit, go to step 5. Over it, or if the agent flagged the case as unclear, go to step 4.
  4. Human Input pauses the run. The approver sees the description you wrote, such as the amount, the order and the agent's reason, and can approve, edit or reject. The run shows as awaiting approval and appears in the Needs you queue.
  5. Action issues the refund through your payment tool or a web request, then sends the customer a confirmation.
  6. Rejection path: send the reason back to the customer or a support queue instead of failing.

Every step, including who approved and what they changed, is part of the run trace, so you can answer "who allowed this?" later.

A quick test for your own workflows#

Take any step and ask three questions. If the agent is wrong, who is harmed? Can it be undone in minutes? Would a reasonable colleague want to see it first? Two "yes, harm" answers or an irreversible action means a pause. Otherwise let it run, and keep the trace so you can look afterwards.

Sources#

Get new posts by email

Practical notes on governed AI workflows, in your inbox. Unsubscribe whenever you like.

Opens your email app to send the request. We use your address only to send the blog. See the privacy policy.

Keep reading