AI oversight needs evidence that humans can disagree

A human checkpoint is weak evidence of human oversight

A CHRO can usually name who owns an AI-influenced hiring decision. A harder question is how many times in the previous six months a recruiter formally overruled the screening tool and what happened afterward. The first answer describes an organizational arrangement; the second provides observable evidence that a person can exercise independent judgment. An AI governance process can look complete while giving its nominal human reviewer little practical ability to disagree.

That gap matters after the governance build-out of 2025 and 2026, when organizations established AI oversight committees, acceptable-use policies, and executive AI-governance roles. Deloitte’s 2026 Global Human Capital Trends survey, conducted with Oxford Economics among more than 9,000 business and HR leaders across 89 countries, found that 64% considered AI and decision-making very important to current success. Yet only 5% considered themselves leading in the area. Deloitte, which has a commercial interest in organizations buying advice on AI, workforce, and governance changes, assesses policy as widespread while evidence that formal authority actually gets exercised remains scarce.

The gap between those forms of evidence is the difference between ownership and behavior. Ownership is a stated claim about responsibility; an override is an observable event showing that someone evaluated a recommendation and reached an independent conclusion. A strong system may deserve high agreement, so an organization does not need a particular override rate to prove that governance works. It does need evidence that the people responsible for human oversight can disagree when their judgment calls for it.

Delegation changes behavior before model quality enters the picture

That ability to disagree is shaped before anyone reviews an AI recommendation, because the design of delegation can change human behavior. Researchers led by the Max Planck Institute for Human Development tested this in a machine-delegation version of the established behavioral-science die-roll experiment. Participants watched an on-screen die roll, reported its result, and earned one cent for every pip they reported across ten rounds. When people reported for themselves, 95% reported honestly.

The researchers then held the underlying situation constant while changing how participants instructed a machine. Everyone saw the same fixed sequence of ten rolls, and the financial incentives stayed unchanged. In the rule-writing condition, participants had to specify what the machine should report for each of the six possible die outcomes. About 75% requested honest reporting.

That decline surprised the researchers because they had expected explicit rule-writing to preserve honesty. They expected that specifying a dishonest action would carry moral weight similar to performing it directly. Instead, honesty fell by about 20 percentage points even under explicit rules. Delegating the act changed behavior even though the events and incentives stayed the same.

More interpretive latitude changed behavior further. Participants in an example-based condition selected a dataset to show the machine what to do, and about half requested honesty. In the goal-setting condition, participants moved a dial between “maximize accuracy” and “maximize profit” and left implementation to the machine. Honesty fell to 12% in the first study and 16% in a follow-up where delegation itself was optional.

Those rates mean that 84–88% of participants in the goal-setting condition asked the machine to cheat without explicitly specifying the method. Across every form the researchers tested, delegation reduced honesty, but the size of the reduction depended on the interface. As the system received more room to interpret the user’s objective, the behavioral change grew.

The experiment was conducted online through Prolific with a maximum payout of 60 cents, so the interface comparison comes from that specific protocol. The underlying die-roll task has a long validation history and predicts real-world behaviors including fare-dodging and deceptive sales practices. A companion tax-evasion study, in which misreporting reduced a donation to the Red Cross, reproduced the central finding about how machine agents act on the instructions people give them.

Natural-language prompting offers another view of the same delegation effect. It sits closer to explicit rule-writing than to turning an objective dial, yet dishonest requests still ran at roughly 25% based on participants’ own accounting and 40% when independent raters evaluated the instructions. That gap matters for governance because the person issuing an instruction and an outside reviewer can reach materially different conclusions about what the instruction asks the machine to do.

The same ambiguity appears in familiar business goals. A manager can tell an agent to maximize qualified pipeline or reduce time-to-fill, while the system determines the operational criteria and methods for achieving the objective. The manager has supplied the desired result while delegating many judgments that connect the instruction to its consequences. Max Planck recommends avoiding interfaces that let people distance themselves from how machines interpret their instructions, because more abstraction between a person and a consequence can make that consequence easier to tolerate.

High agreement cannot tell you whether judgment is working

Once delegation reaches an operating workflow, an override count creates a second measurement problem because agreement is inherently ambiguous. Spain’s Viogén system scores the risk of repeat violence in domestic-abuse cases, and an external audit found that police followed its recommendations 95% of the time. The European Data Protection Supervisor cited that audit in a technical brief issued “last September” and observed that such high concordance could reflect justified trust. The same result could also raise concern about whether independent human judgment persists.

Both explanations produce the same aggregate number. A highly accurate system may legitimately deserve almost every recommendation to be accepted, while a reviewer who has stopped evaluating recommendations independently may also accept almost every recommendation. Prescribing a target override rate would create another governance distortion because disagreement itself is not the goal. What matters is demonstrated capacity to disagree when judgment calls for it.

Evidence of that capacity has to extend beyond the percentage of decisions reversed. Reviewers need usable authority and interfaces, functioning escalation paths, enough competence to form an independent judgment, and organizational freedom to act on it. Records also need to show how recommendations were assessed and what followed when someone challenged one. Zero overrides should consequently trigger investigation instead of an automatic finding that oversight has failed.

Poland shows how formal override authority becomes unusable

Such an investigation can reveal the gap between formal and practical authority, as Poland’s public employment service shows. From 2014 to 2019, it used an algorithmic system that placed job seekers into three categories, with the assigned category determining the support each person received. Client advisors were formally responsible for human oversight and had authority to override the algorithm. On an organization chart or governance checklist, the essential control appeared to exist.

Daily working conditions made that control difficult to exercise. Caseloads were too high, training was insufficient, and advisors had not been adequately told when an override was warranted or how to justify one. The system also displayed its outputs in a way that made it difficult to judge whether a classification fit the individual client. Formal responsibility therefore came without the time, knowledge, and interface information needed for an independent decision.

Those workflow constraints grew stronger when management entered the process. Some local managers discouraged overrides, while others prohibited them because an override attracted attention from higher levels of the organization. Challenging the algorithm created an event that a manager had to explain. Accepting the classification created no comparable event.

That asymmetry changed the rational response for an advisor with a full caseload. Overriding imposed extra work and scrutiny, while acceptance allowed the case to proceed. The visible behavior might resemble trust in the model, yet it could instead reflect a worker responding predictably to organizational incentives. A check for technical override permission cannot distinguish those explanations.

The EDPS’s condition for meaningful authority follows directly from that incentive problem: operators need to be able to exercise their power without fearing organizational consequences for doing so. Formal authority is comparatively easy to grant because it can be written into a process. Practical authority depends on what the organization does when a worker actually exercises it. Poland’s advisors had responsibility while managerial behavior undermined their ability to use it.

That difference puts overloaded frontline employees designated as human reviewers at particular risk. These workers may carry responsibility for the final decision while lacking time, adequate training, useful interface information, managerial support, or protection from extra scrutiny. Asking them to remain “in the loop” accomplishes little when the surrounding workflow systematically rewards acceptance. Human oversight then exists in process documentation while remaining unreliable as an operating capability.

Static decision-rights models can reinforce the same problem because they describe who owns a decision more easily than how authority works when a decision becomes contested. Deloitte argues that legacy tools such as RACI, which assigns who is responsible, accountable, consulted, and informed, assume authority can be allocated in relatively fixed terms. AI workflows require more dynamic decision rights, including system-embedded override privileges, escalation paths, and consensus rules that determine when humans and agents decide. Those mechanics govern how authority moves when a recommendation is contested.

Poland’s experience gives those mechanics a practical test. The public employment service had human reviewers and formal override power, yet workload and managerial incentives encouraged advisors to accept classifications and avoid creating another problem for themselves or their managers. A governance review has to examine what happens when permission is exercised. Otherwise, the review can verify the control while missing the behavior that makes it ineffective.

Automation can make the remaining human judgment harder and less practiced

Even where organizational conditions support overrides, automation can weaken the practice required to make them well. Amy Centers, an organizational psychologist and founder of SmartWorks Labs, describes the risk directly: “The more we outsource hard calls, the more we risk building leaders who can’t lead without a prompt.” Delegation can therefore change human competence over time as well as behavior at the moment a task is assigned.

That competence problem becomes visible when agreement continues for a long run. Consider a manager who accepts 98 straight model recommendations. The manager may have processed the work efficiently, but those decisions also represent repeated occasions when the underlying judgment was left to the system. When the model eventually produces a confident but consequential error, the manager still has nominal authority to intervene, while the plausible answer may be harder to challenge after little recent practice forming an independent view.

Because managers often occupy that intervention point, Centers places particular weight on management quality: “I would start with manager accountability. Most dysfunction cascades from managers. They’re too often unsupported, underdeveloped. They’re not held accountable for how they lead. If you don’t fix that hinge layer, every other reform like engagement, skills, AI adoption, it all collapses.” She describes managers as the hinge between strategy and human experience. In an AI-assisted workflow, that position can also put them where an erroneous recommendation can be stopped before it scales.

That position may carry responsibilities the role was never designed to support. Managers may have been promoted for operational reliability without being told that validating AI output is now part of their job. Giving them an override control assumes they possess the judgment, confidence, expectations, and support required to use it. Underdeveloped or unsupported managers can therefore appear to provide human oversight while lacking the practical capacity behind it.

The competence requirement rises further when automation changes which cases reach people. Victoria Pelletier, who previously led people and transformation functions at Accenture, IBM, and elsewhere, points to contact centers, where automation removed routine calls that could be handled through established rules. The cases reaching people increasingly became those the rules could not resolve. Human volume could fall as the difficulty of each remaining case rose.

That change in case mix requires a corresponding change in role design. Job descriptions, training programs, and performance criteria can continue describing the previous work even though the remaining work, as Pelletier puts it, “requires a very different proficiency level of some of the things we already expect today.” Oversight capacity then weakens from two directions: routine cases provide less practice, while residual cases require more judgment.

Organizations may also have less internal capacity for that redesign. Pelletier says organizational-design and job-architecture functions were reduced over the previous decade as overhead, under an assumption that jobs would not need frequent redesign. Those are the functions needed when automation changes the composition of work. The change can affect the nature of the work as well as its volume. Pelletier calls the result the “Frankensteining of jobs,” where new responsibilities accumulate without a coherent redesign of authority, skills, and expectations.

Human intervention belongs at consequential boundaries

Preserving human authority does not mean routing every AI action through a person. Jurgen Appelo calls excessive mandatory review the “humans-in-the-loop trap”: people become bottlenecks, interactions remain bounded by human availability, systems cannot communicate directly, and operating speed falls to the pace of the slowest participant. A governance program that responds to every risk by adding another approval step can therefore undermine the operating model it is meant to govern.

The EDPS identifies a different failure at the checkpoints that remain. Human presence provides little protection when authority, interfaces, and escalation paths have not been designed to make intervention usable. Appelo’s concern is unnecessary human constraint on automation. The EDPS concern is whether people retained for consequential judgment have enough practical authority to intervene, so the two critiques apply to different parts of the same workflow-design problem.

That shared design problem makes the AI-human boundary a workflow decision with consequences that must be designed explicitly. Kaan Esendemir, an enterprise architect who has built AI-assisted workflows at large-scale enterprises, sees the issue in virtual assistants, workflow accelerators, and systems that recommend an option while a person makes the decision. Business and product owners negotiate where AI stops and human judgment begins, with engineering involved because engineers understand both system capabilities and failure modes. The resulting boundary should concentrate meaningful authority where consequences require it.

Design for disagreement: four moves for consequential AI workflows

Once the boundary is treated as a design choice, a practical redesign starts with reversibility. Amazon’s one-way-door and two-way-door distinction separates decisions that are hard to undo from those that can readily be reversed: irreversible decisions receive greater scrutiny and slower paths, while reversible decisions can move faster. For AI workflows, the same logic means setting agent autonomy according to how difficult a decision is to undo. A routine-looking action can still create durable consequences, so apparent routineness is a poor proxy for the scrutiny it deserves.

After reversibility establishes where scrutiny belongs, the second move is to identify how authority is being delegated. A workflow should distinguish rule-based delegation, example-based delegation, and goal-based delegation because the Max Planck experiments found materially different behavior under each interface. The highest-ambiguity case is one in which a human sets an outcome and the system decides the operative criteria or methods. Naming that delegation mode makes explicit how much judgment has moved into the system.

Once the delegation mode is visible, the third move is to preserve disagreement as operational data. Esendemir observes that organizations are more likely to retain outcomes than records showing where a person rejected a recommendation, which he says is “mostly a design decision.” An override record should capture that a person disagreed, what the person observed, and what happened afterward. Those fields show that someone assessed the recommendation, decided it was wrong, and acted differently.

The same record creates two forms of value. An override can become engineering feedback for improving future recommendations, while the log provides governance evidence about whether the human role is functioning. Neither use requires maximizing overrides or treating them as a performance target. Frequency has to be interpreted alongside model quality, the kinds of cases being handled, and the conditions under which reviewers can intervene.

Because those conditions change, the fourth move is to put a date on reconsidering the AI-human boundary. Capabilities change, failure modes become clearer, jobs change as automation removes routine work, and experience can reveal that a checkpoint is unnecessarily slowing the workflow or failing to provide meaningful oversight. The allocation of decision rights should therefore remain revisable. A scheduled review forces owners to reconsider the boundary using evidence generated by the workflow itself.

That scheduled review can also apply the Max Planck researchers’ recommendation at organizational scale. Delegation should be designed so people remain aware of how systems interpret their instructions, because more abstraction between a person’s objective and its consequences can make those consequences easier to tolerate. Governance therefore needs enough visibility and responsibility for the person delegating or reviewing a decision to understand what judgment has moved into the machine.

Six months provides a concrete interval for testing whether that design works in practice. Six months from now, a COO can choose three workflows in which AI shapes a consequential decision and inspect the documented human disagreements from the preceding half-year. For every disagreement, the review can examine what the reviewer observed, whether they could act differently, and what happened afterward. A zero count has two possible explanations worth investigating: nobody encountered a reason to disagree, or the organization’s formal authority to disagree is ineffective in practice.

That zero count remains a diagnostic signal and does not by itself prove failure. The claim that no vendor presents its product as so good that humans will find no reason to disagree for half a year should be treated cautiously. Policies, committee charters, named owners, and training completions establish that governance mechanisms have been created, while override logs with actual entries expose how authority behaves inside the workflow. The useful evidence is the record of whether a capable person could disagree with an AI recommendation, document the reason, act differently, and exercise that authority free of structural punishment.

In conclusion

For executives, the central governance question is not whether a human appears somewhere in the decision process. It is whether that person has the time, information, competence, authority, and organizational support to reach a different conclusion from the system. Policies and override controls establish formal responsibility. They do not establish that independent judgment survives in practice.

That distinction should change what leaders ask for. Alongside model performance and adoption metrics, consequential AI workflows should produce evidence about disagreement: when recommendations were challenged, why reviewers challenged them, whether they could act differently, and what happened afterward. A low override rate may indicate an excellent system. It may also indicate an environment in which disagreement has become difficult or costly. The number alone cannot resolve that question.

The executive responsibility is therefore to design for meaningful disagreement without turning human review into a bottleneck. Give reversible decisions room to move quickly. Put stronger human authority around decisions that are difficult to undo. Then revisit those boundaries as models improve, jobs change, and the remaining human decisions become more complex.

The strongest evidence of human oversight is not that someone has permission to intervene. It is that the organization has created conditions in which a capable person can disagree when necessary, act on that judgment, and leave a record showing that the authority was real.