In short: "There is always a human sign-off" tells a board almost nothing. Real human oversight of an AI system means the overseeing person can understand what the system is doing, monitor it for anomalies, and actually intervene with consequence - a standard the EU AI Act's Article 14 sets out concretely and that King V, the UNESCO Recommendation on AI Ethics, and the OECD AI Principles converge on from different directions. The oversight owed must be commensurate with risk: heavy for a system that screens job applicants or scores creditworthiness, light for one that suggests a support article. The quiet failure mode is automation bias - the well-documented tendency to over-trust a system that is usually right, until a zero-override rate looks like success when it is actually evidence that no one is checking anymore.
Picture a claims desk on a Tuesday morning. An analyst sits in front of a screen. An AI system scores each incoming claim and recommends approve or decline. The analyst has, on paper, the authority to overrule any recommendation at any time. In eight months, across more than 4,000 claims, she has overruled the system exactly zero times. Not once. When auditors ask why, the answer is honest and unsettling: "the system is usually right." There is a human in the loop. There is no human oversight. A board that confuses the two is exposed.
A person present is not the same as a person in control
A human in the loop is an architectural fact: someone occupies a step in the process. Human oversight is a capability: that person can understand what the system is doing, can tell when it is going wrong, and can actually intervene and stop it with consequence. The first is easy to install and easy to fake. The second is what governance is about.
A board told "there is always a human sign-off" has been told almost nothing. The sharper questions are: does that person have the information to judge the output? Do they have the time, the authority and the standing to overrule it? And when they do overrule it, does anything actually change? Oversight that cannot bite is not oversight. It is theatre, and theatre does not discharge a duty.
Why "commensurate with risk" is the whole phrase
King V frames the board's technology duty under Principle 10, and Recommended Practice 109(c)(ii) is explicit that there must be clear accountability for decisions, actions, outputs and outcomes - decisions, not designs. The board is accountable for what the system decides and does, not merely for how it was drawn up. And the oversight owed is not uniform: it is commensurate with risk. A system recommending a marketing email does not need the same scaffolding as a system screening job applicants or scoring creditworthiness. Applying heavy oversight to trivial systems wastes effort; applying light oversight to high-stakes systems is the failure that ends up in front of a regulator. The board's first question is never "is there a human?" It is "what is at stake here, and is the oversight proportionate to that stake?"
A concrete design standard: the EU AI Act's Article 14
The EU AI Act gives the clearest worked example of what proportionate oversight looks like in design - not a universal rule binding every organisation, but a concrete one that teaches well. Article 14 requires that high-risk AI systems be designed so they can be effectively overseen by natural persons: the overseeing person must be able to understand the system's capacities and limitations, monitor its operation for anomalies, and intervene, interrupt, or stop it through a reliable mechanism. That reframes oversight as an engineering requirement, not a personality trait. If a system does not surface the information a person needs to judge it, oversight was never possible, however conscientious the person at the desk.
Automation bias: the quiet failure mode
Return to the analyst who never said no. Article 14(4)(b) names the tendency directly: automation bias - the well-documented human tendency to over-trust an automated system's output, to treat "the machine said so" as sufficient reason, and to stop applying independent judgement. It is not laziness or incompetence. It is a predictable cognitive default that strengthens the more often the system is right, because each correct output trains the human to trust the next one. "We have trained our staff to stay alert" is not an answer, because alertness decays. The structural countermeasures are: surfacing the reasons behind an output so there is something to evaluate, tracking override rates and treating a zero-override rate as a red flag rather than a success, and rotating or auditing the oversight function so trust cannot silently calcify.
What a board insists on, concretely
Begin with the reasons behind a decision being visible to the person overseeing it - a score and a recommendation alone give an operator nothing to weigh. Insist on a genuine stop mechanism that can halt or reverse an action before harm, not a complaint form after the fact. Insist on override monitoring, because the override rate is one of the most honest signals a board can read: zero overrides across thousands of decisions does not mean the system is perfect. It almost always means the humans have stopped looking. And insist that all of this scales with the stakes - intensive, well-resourced, independent oversight for a high-risk system; proportionate, lighter-touch attention for a low-risk one. The board is not designing the controls. It is testing that management has designed controls a reasonable person would call proportionate, and demanding evidence rather than assurance.
Why every serious framework says the same thing
This duty is not a quirk of one jurisdiction's statute. King V, under Principle 10, places human accountability and oversight at the centre of technology governance. The EU AI Act's Article 14 turns the same idea into a design obligation for high-risk systems. The UNESCO Recommendation on the Ethics of Artificial Intelligence, adopted in November 2021, insists that final responsibility and accountability must always rest with people. The OECD AI Principles, under the accountability principle, place responsibility for a system's proper functioning on the actors who deploy it. Four instruments, four starting points, one destination - when independent frameworks converge this completely, a board should read it as a settled expectation, not an emerging one.
Reading the oversight story in a board pack
When management reports on an AI system, a board should listen for the oversight story - and it is usually missing. A pack stating "the system operated with 99.4% accuracy and a human reviewed all flagged cases" reports activity. It does not say whether that human could understand the outputs, whether they ever overruled anything, whether a stop mechanism exists, or whether the oversight is proportionate to what the system decides. The questions worth asking cost nothing: how many outputs were overridden, and if none, why is that good news? What does the overseer actually see when reviewing a case? What happens, mechanically, when they say no? Those questions are the difference between a board that oversees and a board that receives.
Test your own board's oversight, not just its policy
The interactive AI Governance Oversight Checklist scores your board against King V Principle 10's AI governance values, including human oversight and accountability - no email required to start. Enrolment in Celagenix® Academy unlocks the full learning unit this article is drawn from.
Take the free AI Governance Oversight Checklist →Frequently asked questions
A human in the loop is an architectural fact - a person occupies a step in the process. Human oversight is a capability - that person can understand what the system is doing, can tell when it is going wrong, and can actually intervene and stop it with consequence. A board told there is always a human sign-off has been told almost nothing until it confirms the second condition, not just the first.
It means the intensity of human oversight should scale with what an AI system's decision actually affects. A system recommending which support article to show does not need the same scaffolding as one screening job applicants or scoring creditworthiness. King V's Recommended Practice 109(c)(ii) requires clear accountability for a system's decisions and outcomes, and the level of oversight owed is proportionate to the stakes, not uniform across every system an organisation runs.
Article 14 requires that high-risk AI systems be designed so they can be effectively overseen by natural persons: the overseeing person must be able to understand the system's capacities and limitations, monitor its operation for anomalies, and intervene, interrupt or stop it through a reliable mechanism. This reframes oversight as a design requirement rather than something that depends solely on the diligence of the person at the desk.
Automation bias is the well-documented tendency to over-trust an automated system's output and stop applying independent judgement, named directly in the EU AI Act at Article 14(4)(b). It grows stronger the more often a system is correct, because each right answer trains the human to trust the next one. A zero-override rate across thousands of decisions almost never means the system is perfect - it usually means the human overseer has stopped genuinely evaluating the outputs.
They converge on the same expectation from different directions rather than stating an identical rule. King V's Principle 10 places human accountability and oversight at the centre of technology governance; the EU AI Act's Article 14 turns it into a design obligation for high-risk systems; the UNESCO Recommendation on the Ethics of Artificial Intelligence (adopted November 2021) insists that final responsibility must rest with people; and the OECD AI Principles place responsibility for a system's proper functioning on the actors who deploy it.