“Human in the loop” has become the standard reassurance offered when an organisation adopts artificial intelligence (AI). Regulators now write it into law, and procurement documents ask for it. Yet the research on how people actually behave alongside automated systems suggests that a reviewer’s signature is not the same as meaningful control.
This guide explains what human oversight is meant to achieve, what the main frameworks require, why it so often fails, and what health and public institutions can do to make it work.
What human oversight is meant to do
Human oversight is the set of arrangements that allow people to understand, check, correct and, if necessary, stop an AI system while it is in use. Its purpose is not ceremonial. It exists so that errors are caught before they affect a patient, a benefit claimant or a student, and so that a named person remains accountable for the decision.
That distinction matters. An AI tool can produce a recommendation, but it cannot be held responsible for it. Earlier Praxis analysis of why accountability cannot be delegated to an algorithm makes the same point from the advisory side: responsibility stays with people and institutions.
What the main frameworks require
The EU AI Act
Article 14 of the EU Artificial Intelligence Act requires high-risk AI systems to be designed so that they can be “effectively overseen by natural persons” while in use. The people assigned to oversight must be able, as appropriate, to understand the system’s capacities and limitations, remain aware of the tendency to over-rely on its outputs (“automation bias”), interpret its outputs correctly, decide not to use or to override an output, and interrupt the system through a “stop” button or similar procedure.
For certain remote biometric identification systems, the Article goes further: no action may be taken on an identification unless it has been separately verified by at least two competent people, with exceptions in some law enforcement and migration contexts.
These obligations are not yet in force for most systems. Under the Digital Omnibus on AI, published in the Official Journal in July 2026 as Regulation (EU) 2026/1744, the rules for stand-alone high-risk systems now apply from 2 December 2027 and those for high-risk systems embedded in products from 2 August 2028. Institutions therefore have a window to design oversight properly rather than retrofit it.
The NIST AI Risk Management Framework
The US National Institute of Standards and Technology (NIST) AI Risk Management Framework, a voluntary framework first published in 2023, treats oversight as an organisational process rather than a single checkpoint. It asks organisations to define and differentiate “roles and responsibilities for human-AI configurations” (GOVERN 3.2), to define, assess and document processes for human oversight (MAP 3.5), and to assign responsibility for superseding, disengaging or deactivating systems that perform inconsistently with their intended use (MANAGE 2.4).
WHO guidance for health
In January 2024 the World Health Organization (WHO) issued guidance on large multi-modal models in health with more than 40 recommendations for governments, technology companies and health care providers. It warns specifically that such models can encourage automation bias among health professionals and patients, so that errors which would otherwise be caught go unnoticed. It also calls for independent post-release auditing and impact assessments when models are deployed at scale.
Why “a human in the loop” often fails
The frameworks converge on the same risk: people tend to defer to machines. Two controlled studies in medicine show how strong that tendency can be.
In a 2023 experiment published in the journal Radiology and reported by the Radiological Society of North America, 27 radiologists assessed mammograms with suggestions from an AI system. When the suggestion was correct, inexperienced readers were accurate in almost 80% of cases; when it was deliberately incorrect, their accuracy fell below 20%. Even very experienced radiologists, with more than 15 years of practice on average, dropped from 82% to 45.5%. The lead author, Thomas Dratsch, said it was “surprising” that highly experienced readers were also affected.
A randomised study published in JAMA in December 2023 tested 457 hospital clinicians on diagnostic vignettes. A standard AI model raised their accuracy modestly, by 2.9 percentage points from a 73.0% baseline. A systematically biased model lowered it by 11.3 points. Adding image-based explanations of the model’s reasoning did not significantly reduce the harm.
Both studies used simulated cases rather than live clinical practice, and real-world performance may differ. But they indicate that a reviewer who sees an AI suggestion before forming their own judgement is not an independent safeguard.
The case against relying on oversight alone
Some researchers argue that the problem is structural. In a 2022 analysis of 41 government policies requiring human oversight of algorithms, published in Computer Law & Security Review, Ben Green concluded that people often cannot perform the oversight functions these policies assume, and that such requirements can “provide a false sense of security” in adopting flawed tools. He proposes that agencies should instead justify each use of an algorithm, provide evidence that any proposed oversight actually works, and submit that justification to public review.
This critique does not mean oversight is pointless. It means oversight has to be designed, tested and resourced, like any other control.
Six practical steps for institutions
- Decide whether the task suits AI at all. Before designing oversight, document why the system is needed, what it will and will not do, and what happens if it fails. Where errors are severe and hard to detect, the strongest safeguard may be not to automate.
- Map the decision, not just the tool. Identify exactly where in the workflow the AI output appears and who acts on it. Oversight designed around the real process works better than oversight bolted on afterwards, a point developed in Praxis’s piece on designing digital systems around the workflow.
- Protect independent judgement. Where feasible, ask reviewers to record their own assessment before seeing the AI output, or reserve AI suggestions for a second read.
- Give reviewers time, authority and a stop button. Oversight fails when staff have quotas that make overriding impractical, or when no one is clearly authorised to suspend a system.
- Train for limitations, not just features. Staff need to know the conditions in which a system performs poorly. Italy’s new requirement for AI literacy across professions and health workers reflects a wider move to make this training an obligation.
- Measure whether oversight works. Track how often reviewers override outputs, audit a sample of accepted outputs, and test periodically with known errors. An override rate close to zero may signal deference rather than accuracy.
What remains uncertain
There is no agreed measure of “effective” oversight, and most experimental evidence comes from high-income clinical settings. How automation bias plays out in low-resource health systems, where staff shortages and workload pressures are greater, is less well studied. Guidance on oversight of general-purpose AI assistants, as distinct from narrow diagnostic tools, is still developing.
What is clear is that oversight is an institutional capability, built from governance, workflow design, training and monitoring, rather than a box to tick.
Praxis supports institutions in building that capability through ethical AI and technology services, practical training and advisory work on governance; to discuss a specific system, contact the team.
Praxis Editorial Team
The Praxis Editorial Team publishes Praxis Global Institute’s explainers, practical guides and analysis of new evidence and policy in global health, education and professional capability, ethical AI and technology, research and data, and institutional strengthening. Every article is researched from primary sources, cites its evidence and is published under the editorial direction of its editors.
Facing a similar challenge in your institution?
We work with teams on research, data, digital systems, and learning that hold up in practice.
