We are at a critical juncture in the fundamental transformation of healthcare. Every conference, policy paper and funding call now mentions Artificial Intelligence (AI) as the solution to halt the collapse of our healthcare system, facing systemic strain from workforce shortages, and escalating operational costs. Hospitals are investing in diagnostic algorithms, workflow automation and predictive analytics. But while we fixate on the implementation of AI, we risk falling into the most dangerous pitfall of this century: the Clinical Autonomy Paradox.
Most new and innovative AI systems being introduced in hospitals today operate at what can be called Clinical Autonomy Level 3 (CAL 3). At this level, the algorithm performs predefined tasks, like reading images or performing patient triage, while the physician remains the fallback user. If the system fails, generates a wrong response or misinterprets a result, the physician must both recognise the mistake as well as intervene accurately and immediately.
This setup sounds reassuring: there is always a human in the loop. However, this places an impossible demand on our healthcare professionals. By doing so, we degrade expensive and highly trained medical professionals into monitors of an AI system while holding them fully liable (under medical liability law) when they fail to make an accurate (split-second) intervention. No professional, however skilled, can maintain full situational awareness while overseeing semi-autonomous systems.
The paradox lies within the mismatch between control and accountability, where AI assumes part of the diagnostic or therapeutic process and the responsibility remains entirely with the human healthcare provider. The result is a fear of liability, a cognitive overload and subtile yet dangerous deskilling of professionals.
The European AI act clearly states that high-risk systems used in healthcare must always include meaningful human oversight. While the AI-act aims to protect patients, it automatically reinforces the Level-3 paradox by stating there must always be a human in the loop, but it does so without realising the burden put on the healthcare professionals. Especially when, in the near future, AI advances and the position of the clinician changes from ‘human in the loop’ to ‘human on the loop’ of a system too complex to comprehend. In such a setting, questions arise like: if diagnostic AI fails, who is to blame: the developer, the hospital or the physician?
From this dilemma, three different scenarios emerge. In the first one, healthcare providers, in fear of legal and reputational damage, will limit the use of AI to a mere advisory role (AI as a copilot). AI will remain a tool, innovation stalls, funding dries up, and the European health-tech sector loses ground to regions with more flexible liability laws.
The second scenario is one where hospitals pursue automation at all costs, resulting in a cold, inhuman efficiency drive where the physician becomes a 'checkbox ticker' for a 'black box' that no one understands. Ethical decision-making will erode under the pressure of production and revenue.
The third scenario, what we might call ‘the golden path’, is a future which is not only more sustainable, efficient and safe, but also fundamentally more human. It embraces automation not as a replacement but as a symbiotic intelligence to the healthcare provider. Physicians can focus on atypical and complex cases while the AI handles predefined tasks, routine and repetitive tasks. This scenario will require us to move past CAL-3 and towards Clinical Automation Level 4 or 5.
In CAL-4 (figh automation), the AI operates autonomously within a bounded domain, for instance interpreting chest x-rays for the presence of a pneumothorax, with performance validated through clinical trials. The physician remains accountable for the system selection and monitoring but is no longer required to intervene immediately when it fails.
At CAL-5 (full automation), the system performs all predefined tasks within a specific domain without human intervention. Examples of these systems can include systems like autonomous continuous monitoring of clinically stable patients or automated laboratory equipment.
Symbiosis in Clinical Autonomy Level 4/5
In the third scenario, we consciously and deliberately step past the CAL-3 phase, and we strive for CAL-4 and CAL-5 for clearly defined vital tasks. To do this responsibly, this scenario requires two parallel breakthroughs.
The first requirement is a technological breakthrough. Healthcare can not build trust in opaque systems. The 'black box’-systems must be opened. Physicians need systems that can not only provide validated answers but are transparent and can explain their reasoning in human-interpretable terms. A responsible transformation demands explainable AI. Why does the AI think this is a malignancy? Which data points did it weigh most heavily? Only then can the physician leap from 'passive monitor' to 'expert validator'.
The second breakthrough concerns AI law. As long as liability fully rests with the healthcare provider, no amount of transparency will resolve the CAL-3 paradox. A shift towards product liability, where the manufacturer of the AI product is responsible for a safe and certified system, is the true game-changer. This model is a for healthcare adapted version of the classification for self-driving cars by the Society of Automotive Engineers. Once a car operates at a level 4 or 5, the liability for a system failure rests largely with the manufacturer, provided the system was used as intended.
Translating this to healthcare means that when a certified CAL-4 or 5 AI system fails while used as intended, the liability claim lies largely with the developer or vendor of the system and not with the physician. The manufacturer must demonstrate the system can reach a ‘minimal risk condition’, as automatically stopping the car, in situations where human oversight is needed. In this situation, meaningful human oversight is redefined: not as 'immediate intervention', but as 'passive validation of the process’. The physician oversees the workflow without micro-managing the AI algorithm.
AI as a catalyst for re-humanising care
In this high or full autonomic world, the physicians are liberated from the repetitive, data-intensive routine task and not threatened by ‘deskilling' or being demoted to being a ‘checkbox ticker’, but are stimulated towards ‘up-skilling' where the role transforms into that of the 'system conductor' or 'expert validator'. They evaluate the output of the AI and overwrite it with clinical wisdom. It will require the clinician to become fluent in data science, ethics and systems thinking.
Most importantly, the new role will provide the clinician with more time. Time to listen and to invest in the patient. Time for the ethical dilemmas or complex psychosocial contexts. And time for the empathy and human touch that the algorithm will not be able to provide. In doing so AI does not become a threat but a catalyst for re-humanising care.
The Patient Gets the Best of Both Worlds
For the patient, this transformation offers the best of both worlds. They receive the analytical precision and speed of the best AI, combined with the holistic view, wisdom and compassion of the healthcare provider.
When AI explanations are available to both clinician and patient, informed consent gets real meaning. When patients can understand, in layman’s terms, why a certain treatment is recommended or why the option not to treat is advised, and the physician is provided with explanations in medical terms on why and how this treatment decision was made, a true, meaningful conversation can take place where communication skills, rather than technological skills, define the patient-doctor relationship.
The future of AI in healthcare is not a technological one; it’s a strategic one. We must recognise the CAL-3 paradox as the pitfall it is. We must demand explainable AI from manufacturers and work with policymakers to create flexible liability frameworks that make Clinical Autonomy L4/L5 possible. Moreover, education must adapt. Medical curricula should include modules on algorithmic reasoning, data science, digital ethic and system safety to ensure our future clinicians are not only prepared for sharing the workspace with AI but are able to shape it responsibly to enable the healthcare professionals to be fully human again.
Citation
1. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance); PE/24/2024/REV/1; OJ L, 2024/1689, 12.7.2024, ELI: http://data.europa.eu/eli/reg/2024/1689/oj
2. SAE International Recommended Practice, Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles, SAE Standard J3016_202104, Revised April 2021, Issued January 2014, https://doi.org/10.4271/J3016_202104.
