Most diagrams of agentic work contain a reassuring box somewhere near the end.
It says Human Review.
Once that box appears, the architecture feels safer. The agent may research, interpret a request, call tools, prepare a recommendation and perhaps execute much of the workflow, but a person is still there before the consequence. Control appears to have been preserved.
I keep coming back to a fairly basic problem with this picture. It assumes that the person at the checkpoint will know when the system has failed.
That is a much stronger assumption than simply putting someone in the workflow.
Imagine an AI system that produces an obviously absurd answer. Oversight is easy. The reviewer sees the problem and rejects it.
Now imagine something much more ordinary: the answer is polished, plausible and mostly correct. One assumption underneath it is wrong. A source is stale. A tool call quietly failed. A constraint was interpreted differently from the way the business intended. An intermediate step was skipped, but the final response still looks complete.
What exactly is the human meant to notice?
This is where the phrase human in the loop becomes less reassuring. The person may see the final answer but not the work that produced it. They may understand the business problem but not recognize the particular way the AI has failed. They may recognize the problem and still lack the authority to stop what happens next.
In some workflows, the human may end up carrying accountability for a process they could neither properly inspect nor meaningfully interrupt.
That is not a hypothetical edge case in governance. It is the central weakness in treating human presence as evidence of human control.
The EU AI Act is unusually concrete on this point. Its human-oversight provisions for high-risk systems do not stop at requiring a person to be involved. They contemplate people who can understand relevant system capabilities and limitations, detect anomalies and unexpected performance, remain alert to automation bias, interpret outputs, disregard or override them, and intervene or stop the system.EU AI Act · Article 14
The regulatory language is specific because the distinction matters.
A human checkpoint is not the same thing as effective human oversight.
The more capable AI systems become, the less useful it is to imagine failure only as a visibly bad answer.
An agent can fail while producing something that looks perfectly reasonable.
It can choose the wrong objective and execute it competently. It can complete nine of ten steps and report completion. It can apply the right rule to the wrong context. It can draw from an authoritative source that is no longer current. It can satisfy the immediate request while violating a boundary the human assumed was obvious.
The interesting failures are often not nonsensical. They are plausible. And plausibility changes the oversight problem.
If the human sees only the polished result, much of what would make the failure detectable may already have disappeared from view: the assumptions, alternatives, tool results, discarded evidence, uncertainty and deviations from the expected path.
This is why observability matters so much in agentic work.
A person cannot challenge an assumption they never see. They cannot notice an unusual tool action if the system hides it. They cannot distinguish successful completion from a confident completion claim if the workflow presents both in exactly the same way.
This is not simply a transparency argument. Showing more information does not automatically create better oversight either. A screen full of traces, logs and intermediate reasoning can overwhelm a reviewer just as easily as a polished summary can hide too much.
What does the human need to see in order to know when their judgment is required?
That is a very different design objective from giving the user access to everything.
Recent work on agent oversight is beginning to show how much practical work sits behind that question. A 2026 study of experienced developers using software agents found oversight happening before, during and after agent execution: developers constrained actions in advance, co-planned work, monitored agents while they operated and reviewed results afterwards. They also developed heuristics because directly evaluating agent-generated work could itself become difficult.Human oversight of software agents
That is closer to what real oversight looks like: not one approval button at the end, but a pattern of control distributed around the work.
There is a temptation to treat this as something entirely new because agents are new.
It is not.
Automation researchers have spent decades studying what happens when people move from doing work to supervising systems that do it for them.
One of the better-known findings is the out-of-the-loop performance problem.
In a 1995 study, Mica Endsley and Esin Kiris examined how automation affected operators' ability to understand what was happening and resume control when the automated system failed. They found that operators working under automation could lose situation awareness and become slower at responding when they had to take control. The shift from active involvement toward passive monitoring was an important part of the problem.Endsley & Kiris, 1995
That research came from a very different technological era, but the underlying tension feels remarkably current.
If you are no longer doing the work, what keeps you capable of recognizing when the work has gone wrong?
Agentic AI extends the question into knowledge work.
The person may no longer be supervising a machine following a fixed operational sequence. They may be supervising a system that interpreted the objective, formed a plan, chose tools, produced intermediate judgments, adapted when something changed and decided when the task was complete.
That is a more difficult thing to supervise. And it introduces an uncomfortable possibility.
The more we delegate the work that develops understanding, the more we may weaken the understanding on which later oversight depends.
This is still an emerging question in the context of AI agents, and we should not overstate the evidence. But it is increasingly being taken seriously. A 2026 position paper by Margaret Mitchell, Avijit Ghosh and Samir Passi argues that current agent designs can make effective oversight harder while extended automation may also weaken some of the cognitive capacities required for it.Mitchell, Ghosh & Passi, 2026
Their proposal is not simply to keep humans in the loop, but to design agent systems around the cognitive requirements of the people expected to oversee them.
That leads to what may become one of the defining governance problems of agentic work.
Suppose an agent is unreliable.
People check it. They challenge its work. They watch what it does. They learn where it tends to fail.
Now suppose the agent becomes very good.
Most of the time the result is correct. The recommendations are sensible. Actions complete successfully. The human learns that intervention is rarely necessary.
That sounds like progress, and it is.
But something else may happen at the same time.
Review becomes routine. Attention moves elsewhere. The human gets less practice dealing with the difficult cases. The system's answers become more persuasive because they are usually right.
Then, eventually, the unusual case arrives.
And the unusual cases are exactly the ones that may require the judgment the system has made less necessary during ordinary work.
As automation becomes more dependable in ordinary situations, the human may become less prepared for the situations in which oversight matters most.
There is no reason to turn this into an argument against automation. The productivity gains can be real. Delegation can be sensible. There is little value in keeping people manually busy merely to prove that they remain involved.
But it does change what organizations need to think about when they claim that a human remains in control.
Control cannot mean only that the person has a button available. It has to include whether they retain enough awareness and capability to know when to use it.
That capability is not static. It has to be sustained.
Capability and dependency
FE-005 — The Limitless Problem: Intelligence Supplements, Cognitive Dependency, and Human AgencyFE-005 examines what happens when externally supplied intelligence improves performance while the underlying human capability may become more conditional.
At this point the problem starts to look less like workflow design and more like capability.
If a human is expected to govern an AI system, what must that person actually be able to do?
They first need enough visibility to notice that something deserves attention. But visibility is only useful if they can make sense of what they see.
A domain expert may recognize that a recommendation conflicts with the reality of the situation. Someone familiar with the AI system may notice that an apparently confident answer rests on weak evidence. An experienced operator may recognize that an agent has moved beyond the mandate it was given.
Then comes the harder part: challenge. Not challenge in the sense of disagreeing with AI by default, but the ability to test what matters. To ask which assumption would change the decision. To know when a second source is warranted. To distinguish a polished explanation from a well-supported one. To recognize when the agent's claim of completion needs independent confirmation.
And when the problem is real, the person has to be able to intervene. Sometimes that will mean rejecting a recommendation. Sometimes stopping an action, narrowing a permission, asking for another source, escalating to a specialist or reverting to a manual process.
Finally, someone still has to own what happens.
As we worked through these requirements, a useful chain emerged:
Observe → Interpret → Challenge → Intervene → Own consequence
I do not think this should be treated as another five-step governance recipe. Its value is simpler. It shows how easily the phrase human oversight can break.
If the human cannot observe the relevant activity, the chain breaks. If they can see it but cannot interpret it, the chain breaks. If they understand it but have been trained to accept the system rather than challenge it, the chain weakens.
If they recognize the problem but cannot stop anything, the oversight is largely ceremonial. And if accountability becomes ambiguous after delegation, human presence has not preserved consequence ownership.
The weakest link may be different in every organization.
That is why simply counting checkpoints tells us so little.
The person performing oversight does not need to become an AI engineer.
That would be the wrong conclusion.
In many consequential settings, domain expertise remains more important. The doctor needs medical judgment. The engineer needs engineering judgment. The financial controller needs to understand the financial consequences. The compliance professional needs to understand the regulatory context.
AI Fluency matters because these people increasingly have to exercise their expertise through a system that is participating in the work.
They need to know enough about that collaboration to remain effective.
Someone supervising a low-risk drafting assistant needs a different level of capability from someone supervising an agent that can change a customer record, make a financial commitment, alter a regulated artifact or trigger an operational action.
So the relevant idea is not generic AI skill. It is role-appropriate AI Fluency.
We use AI Fluency at Coaincentives Labs to describe the discipline of governed human–AI collaboration: retaining intentionality, authorship, challenge, verification and responsibility while using AI to extend what a person can do.
That human capability layer becomes increasingly important when the role changes from simply using an AI tool to governing work the AI can perform with some autonomy.
This is where the “human in the loop” assumption becomes testable. Can the person recognize when the AI contribution is weak? Can they identify what needs verification? Do they know what has been delegated and what has not? Can they challenge an answer that looks plausible? Do they know when the task has crossed from routine execution into a situation that requires human judgment?
If not, placing them at the checkpoint may create an illusion of control.
There is an important limit to this argument.
AI Fluency is necessary for meaningful human oversight. It is not sufficient.
A highly capable person still cannot supervise effectively if the system hides material actions, if logs are unavailable, if the human has no override authority, if permissions are excessively broad, or if the workflow asks one person to review thousands of decisions at a pace that makes genuine judgment impossible.
The human has to be capable. The system also has to make that capability usable.
Agentic Readiness foundation
FE-003 — From Human Agency to Agentic ReadinessFE-003 develops Agentic Readiness as the capability to work with increasing AI autonomy while preserving human agency, judgment, accountability and traceability.
This may be the point at which the usual human-versus-technology framing becomes unhelpful.
If oversight fails, it is easy to blame either side. The user was not sufficiently trained. The model was not sufficiently transparent.
Neither explanation is complete.
Effective oversight emerges from the combination of the person, the system and the surrounding operating model.
The human needs the right capability. The interface needs to surface what matters. Permissions need to constrain what can happen without intervention. Actions need to be traceable enough to inspect. Consequential steps need sensible escalation and reversal paths. Decision rights need to be explicit. And the workload has to leave enough space for judgment to be real.
This also explains why “more human review” is not a credible answer to every AI risk.
If a person has to inspect every action an agent performs, we have either recreated the work we intended to automate or created an approval queue that people will learn to clear mechanically.
The point is not maximum human involvement. It is effective human governance.
That requires calibration. Low-consequence, reversible work can tolerate more autonomy and lighter review. Higher-consequence or difficult-to-reverse actions deserve stronger controls, better evidence and clearer intervention points.
Some risks should be constrained technically rather than left to human judgment. Monitoring can help identify exceptions. Permissions can narrow exposure. Independent checks can verify important outcomes.
The human should be brought in where human judgment actually changes the quality or legitimacy of the decision.
This is defense in depth rather than human review as a universal safety blanket.
And it leads to a different management question: under what conditions can the human still exercise meaningful control?
Management-system implication
FE-004 — Enterprise AI Enablement Is a Management System Refresh, Not a Tool RolloutFE-004 explains why AI capability depends on management systems, incentives, decision rights, learning loops and governance rather than technology deployment alone.
Once human oversight is treated as a capability rather than a box in a process map, another implication follows.
Organizations need evidence that it works.
A policy stating that “all AI-generated decisions must be reviewed by a human” tells us almost nothing about the quality of that review.
The useful test is closer to practice.
NIST's AI Risk Management Framework points in this direction. It calls for human oversight processes and operator or practitioner proficiency to be defined, assessed and documented rather than simply declared.NIST AI RMF
The broader implication is useful: governance controls become stronger when organizations collect evidence about whether the control functions in practice.
This is also where measurement of AI Fluency becomes relevant to Agentic Readiness.
AFA is not an agent-safety system, and it should not be presented as one.
Its relevance is more bounded.
If an organization increasingly depends on people to direct, challenge, verify and govern AI-supported work, it needs ways to understand whether those human capabilities are actually present and developing.
That is one part of the readiness problem. The rest sits in the architecture, permissions, monitoring, workflow design, escalation and accountability around those people.
Measurement foundation
FE-006 — AI Adoption Is Rising. Where Is the Evidence of Capability?FE-006 asks what evidence leaders need before they can claim that AI adoption is building durable human capability rather than simply increasing visible activity.
Metacognitive foundation
LN-003 — Offloading Cognition Is Not the Same as Offloading MetacognitionLN-003 distinguishes useful cognitive offloading from delegating the planning, monitoring, verification, and reflection that make human oversight meaningful.
We will probably continue using the phrase human in the loop.
It is useful shorthand. It reminds us that consequential automation should not make human agency disappear.
The danger comes when the phrase becomes evidence that the problem has already been solved.
A human can be in the loop and still be unable to see what matters. They can see what matters and fail to recognize its significance. They can recognize the problem and lack the authority to intervene. They can retain nominal accountability while the practical conditions for exercising judgment have quietly disappeared.
As AI systems take on more execution, this distinction becomes more important, not less.
Agentic Readiness therefore cannot be measured only by what the AI can do.
It also depends on whether the organization has preserved the conditions under which people can still govern what the AI does.
That includes system observability. It includes bounded authority. It includes escalation and reversibility. And it includes humans with enough role-appropriate AI Fluency to know when their judgment is required.
Putting a human in the loop is easy to draw. Making that human an effective control is a design, capability and governance problem.
The control is not the presence of a human. It is whether meaningful human judgment can still change what happens next.
AI Fluency Accelerator (AFA) Coach helps people develop AI Fluency through meaningful work. AFA Assessment measures performance in a deliberately bounded, goal-based assessment session. Skill Accolades can turn demonstrated capability into evidence-backed, verifiable recognition. Assessment produces a session-bound performance result; qualifying patterns demonstrated over time can support Proof-of-Skill recognition.
AI Fluency is a leading indicator of Agentic Readiness: the capability to work with increasing AI autonomy while preserving human agency, judgment, accountability, and traceability.