Transparency, Explainability and Trust in Health AI

Why Doctors & Patients Need to See Under the Hood.

In recent years, health systems have eagerly explored AI tools, from diagnostic algorithms to virtual “doctors” with promises of faster, cheaper, more accurate care. But every big win has reminded us of the many ways things can go wrong when trust is missing.  IBM’s Watson Health division, once a poster child for AI cancer care, infamously “spit out erroneous treatment advice,” according to leaked internal documents, with “multiple examples of unsafe and incorrect treatment recommendations”.  In the UK, Babylon Health’s AI-driven symptom checker (the “GP-at-Hand” service) was criticised when The Lancet observed in 2018 that there was “no evidence” it worked better than a doctor and indeed might perform “significantly worse”.  These cautionary tales, technology touted as miraculous, only to disappoint patients and clinicians, highlight a critical lesson: breakthrough algorithms alone are not enough.  Without deliberate measures for transparency, explainability and trust, even the most advanced tool can be rejected or misused. As one UK AI policy adviser memorably put it, “if we want people to trust this tech, then ethics, transparency and the founding values of the NHS have got to run through our AI policy like letters through a stick of rock.”

The stakes are high.  Trust is what allows a doctor to say “you need this medication” or a patient to accept a machine’s word.  Psychologists note that trust “increases the tolerance of uncertainty” and “reduces the perceived complexity” in care.  In other words, if clinicians and patients trust an AI, they will be more willing to use it even amid natural worries and unknowns.  Building that trust means first understanding the terms:

·       Transparency is the broad principle of openness about an AI system.  It means making clear what the system is, why it was built, how it was developed and tested, and what data it uses.  Transparency is often described as an “umbrella concept” that spans many practical measures, everything from sharing the code and documentation, to logging decisions (audit trails), to explaining how data were collected and processed.  For example, UK NHS guidance on data governance insists that datasets fed into an AI should contain “only the specific information needed,” with all identifying information removed “to protect confidentiality.”   These steps help assure users that nothing hidden or unexpected is happening behind the scenes.

·       Explainability is a related but narrower idea: the extent to which the AI’s reasoning can be explained in human terms.  Even a transparent system can be complex, explainability asks, “Can I interpret its results?”  In US Food and Drug Administration (FDA) guidance, explainability refers to “the degree to which the logic of the machine learning system can be explained in a way that is understandable.”   This often means providing clinicians with clear descriptions of why the AI made a recommendation or flag.  Explainability may involve visual cues, simple rules, or confidence metrics alongside an AI score, so a doctor can see the logic (for instance, highlighting which features of an image triggered a diagnosis).

·       Trust is the ultimate goal, the confidence that healthcare providers and patients place in the AI tool.  Trust isn’t a purely technical attribute; it is built on experience, relationships and proven reliability.  It grows when developers and vendors demonstrate openness and rigour, and when users see AI decisions regularly confirmed in practice.  As one literature review notes, trust in clinical AI is often conceptualised around both the technology’s capabilities and the broader human context.  That includes the AI’s accuracy, safety and fairness, plus factors like patient privacy, clinician oversight and whether the AI aligns with clinical values.  In short, transparency and explainability are necessary conditions for trust but not enough on their own.  Users may demand evidence, trials, audits or human reviews, to actually trust the system, especially for high-stakes decisions.

These distinctions clarify our challenge: health AI must not only perform well on tests, but also be proactively unveiled to its users and regulators.  A truly transparent development process (sharing data origins, method and performance metrics) and clear, interpretable explanations are key to earning the trust that drives real adoption.

Regulatory and Ethical Frameworks

International and national authorities have rapidly moved to set guidelines for health AI, often echoing these concepts of openness and trustworthiness.  In the US, the FDA, together with counterparts in Canada and the UK, issued “Guiding Principles for the Use of AI in Medical Devices” (2021), explicitly defining transparency and explainability.  The FDA guidance states that “transparency” means appropriately communicating information about an AI medical device’s intended use, development history, performance, and logic.  Explainability is acknowledged as part of that: it’s the degree to which one can articulate how the AI’s logic produces an output.  These principles urge a human-centred design approach, involving clinicians and patients early on to decide what information users need and understand.

In the UK, the NHS and regulators have similarly insisted on AI alignment with ethics and patient trust.  For example, NICE’s recent position statement on AI in evidence generation makes clear that trust must be built into AI workflows.  NICE emphasises that AI should augment rather than replace human judgment, keeping a capable, informed human in the loop, and that extensive validation is required whenever AI is used.  The guidance even warns that the use of opaque “black box” models can “introduce challenges for transparent reporting”, and recommends using explainability tools and layman descriptions to make results accessible.  Essentially, NICE is saying: if it’s not explainable, and if humans aren’t overseeing it, trust will suffer.

Across Europe, the new EU Artificial Intelligence Act (set to begin enforcement in 2025) categorises medical AI as high-risk.  High-risk AI systems will require rigorous quality management and risk assessments akin to existing medical device rules, including documented measures for safety and transparency.  Manufacturers must monitor performance in the real world and report incidents, a process meant to “maintain accountability and enhance trust” in AI healthcare solutions.  In practice, this could mean independent audits, a public registry of approved algorithms, or mandatory patient consent procedures when AI is used.  The EU stance, like global WHO guidance on AI ethics, is clear: accountability, privacy and fairness must underpin AI deployments in health, or public confidence will erode.

Regulators in emerging markets are watching these developments closely.  Some countries are adopting similar principles (for instance, India’s draft national AI strategy emphasises fairness and transparency in healthcare AI), but many low- and middle-income settings still lack detailed AI rules.  Experts warn this uneven landscape could lead to inconsistent safeguards.  Indeed, analysts note that even within the EU, “varying levels of resources” among countries risk fragmenting oversight.  That makes it all the more urgent for developers to meet the highest global standards on trust, since national rules may lag behind the technology.

Trust on the Ground: Successes and Stumbles

What do real-world experiences tell us about trust in practice?  Some high-profile case studies show how easily confidence can vanish, or be gained, based on transparency and accountability.

Erosion of trust.  IBM Watson for Oncology has become a case study in overselling AI.  Early papers and marketing touted it as a revolution, but STAT News obtained internal slides showing the system often generated “unsafe and incorrect” treatment options.  Clinicians who tried Watson on real patients found its cancer therapy suggestions alarmingly poor.  When such flaws came to light, Watson’s reputation cratered and many hospitals pulled back from the technology.  Similarly, Babylon Health’s story in the UK ended badly.  The company repeatedly hyped its AI as outperforming doctors, yet independent analysts found no peer-reviewed evidence of safe, superior performance.  One NHS consultant even publicly recorded misdiagnoses by the chatbot.  The contrast between Babylon’s claims and its opaque “Excel-flowchart” reality fuelled public and professional scepticism.  By mid-2023, Babylon had collapsed financially, a casualty of lost trust as much as financial mismanagement.

Another cautionary example came from an algorithm used to flag high-risk patients for extra care.  Researchers Obermeyer et al. (2019) showed that a widely used “risk score” systematically under-identified Black patients, mistakenly treating them as healthier than they really were.  The flaw wasn’t a technical bug so much as a data bias: the model used healthcare costs as a proxy for illness, but Black patients often incur lower recorded costs due to unequal access to care.  The result was that Black patients with serious needs were overlooked.  This bias was discovered by careful auditing, an act of transparency, and led many health systems to reconsider that AI tool.  It’s a stark reminder that trust can only survive if models are tested for fairness: left unchecked, bias destroys confidence and worsens disparities.

Building trust by design.  On the positive side, there are examples where transparency and validation have paved the way to clinician acceptance.  The IDx-DR system for diabetic eye disease is one success story: it became the first FDA-authorised autonomous AI for a screening test (approved in 2018).  The company behind it published trial results showing high sensitivity and specificity at multiple clinics, and it was marketed as a direct aid to patients (bypassing need for an ophthalmologist for initial screening).  Although a recent U.S. study found that adoption of AI eye screening remains “low”, less than 5% of diabetic patients got AI-based retinopathy checks in practice, the technology has demonstrated that with proper validation and FDA review, doctors will at least give such tools a try.  Another example is AI-powered stroke detection software like Viz.ai, which received early FDA clearance and is now integrated into many hospital workflows; its clear use-case (alerting specialists faster than manual review) and regulatory backing helped gain clinician trust over time.

In global settings, some pilot projects have also shown promise when local doctors are deeply involved.  For instance, AI triage tools for X-ray interpretation in rural clinics (trained on local data) have achieved better uptake when frontline staff participated in building and testing the model.  The emerging lesson is clear: where clinicians see the results for themselves, and where decision rules are open to inspection, they are more willing to rely on the AI’s output.

Clinicians and Patients: Human-AI Collaboration 

Central to building confidence in AI is the human, computer partnership.  Even the FDA explicitly calls for “human-centred design” of medical AI.  That means involving nurses, doctors and patients early: asking what information they need to trust a tool, iterating interfaces with user feedback, and educating staff on the AI’s strengths and limitations.  For example, if an AI generates a diagnostic probability, the interface might include a simple explanation (e.g. “model focuses on this finding in the image”) or an easy way to drill down into the result.  Conversely, black-box outputs with no rationale will often be ignored.

Several studies have shown that clinicians’ trust depends on how AI is presented.  In some trials, doctors who were given visual or textual explanations alongside AI predictions were more likely to incorporate the AI correctly in decision-making.  When explanations were lacking or misleading, doctors either over-relied on the AI (leading to error) or distrusted it entirely.  Hence, best practice now recommends “informed human-in-the-loop” workflows: the AI flags a case, the doctor reviews the same data plus the AI’s reasoning, and ultimately the human takes responsibility.  This augments, rather than replaces, the clinician, a principle NICE spells out explicitly.

Data governance also intersects with trust.  Patients naturally want assurance that their personal health data aren’t being misused.  Good practice, and indeed many regulators, require strict data handling: anonymising patient records, securing consent, and being transparent about data use.  For instance, NHS digital guidance for AI emphasises using only minimal necessary data and stripping identifiers to “protect confidentiality”.  Demonstrating such care for privacy helps earn patient trust, which in turn enables broader data-sharing and better models.  Likewise, ongoing monitoring of AI systems is crucial: health organisations must track performance metrics continuously to catch drifts or rare failures.  Transparent reporting (for example, publishing the algorithm version and dataset statistics) makes it possible for outsiders to spot issues and foster trust.

Finally, it is vital that clinicians feel empowered to question and report AI outputs.  Regulatory bodies in the UK and US urge developers to include clear guidance on when the AI should not be used, and to establish feedback loops.  An environment where a nurse or doctor can say “This recommendation seems off”, and have that concern investigated and fed back into model improvements, builds trust more than a sealed box that only the vendor oversees.  In some pilot programs, multidisciplinary “AI ethics committees” review cases where clinicians disagree with the AI, echoing the review boards used for drugs and devices.  These practices recognise that trust is maintained not by pretending AI is infallible, but by openly acknowledging limitations and having processes to address them.

Designing Trust from Day One

What are the emerging “best practices” for building trustworthiness into an AI product from the very start?  The field is still coalescing around standards, but some themes recur in recent guidance and research:

·       Document everything.  Maintain thorough documentation of data sources, model choices, training and validation procedures.  Inspired by disciplines like medicine, several frameworks (e.g. model cards or datasheets for datasets) have been proposed so that product teams can describe in standardised language the scope and limits of their algorithms.  The idea is to create a “paper trail” akin to a drug trial: regulators and end-users can inspect it to verify claims.  For example, NICE advises publishing any identified risks (such as potential bias) and the steps taken to mitigate them.

·       Use rigorous checklists and frameworks.  Healthcare teams are already used to checklists for safety; similarly, AI development can follow published protocols.  Leading organisations recommend following guidelines like CONSORT-AI and SPIRIT-AI (for clinical trials involving AI), TRIPOD-AI (for transparent reporting of prediction models), and new Algorithmic Transparency Standards.  NICE explicitly mentions using the PALISADE and TRIPOD+AI checklists to ensure comprehensive risk and performance reporting.  These tools force developers to think about transparency at each stage and to communicate it clearly.

·       Plan for explainability.  Rather than scrambling to add explanations after the model is built, embed explainability in the design.  Choose algorithms that are naturally interpretable when possible (such as decision trees or attention-based neural nets for images) and engineer the user interface to highlight the AI’s reasoning.  Usability testing with clinicians, showing them mock AI outputs and refining the presentation until it’s intuitive, is a growing recommendation.

·       Engage stakeholders early.  Beyond clinicians, include patients, ethicists and legal experts from the beginning.  Involve regulatory affairs specialists so that product documentation meets evolving requirements (NHS and FDA both stress early regulatory engagement).  Some companies now bring ethicists into core teams to foresee non-technical risks (for example, explaining to users when a model might fail).  Frontline doctors and nurses should have a channel to voice concerns and see those addressed, the NHS guidance even suggests empowering clinical champions who can “translate” AI decisions for colleagues.

·       Pilot and validate in the target setting.  Performance on a lab dataset is not enough for trust.  The algorithm must be proven on real-world data, ideally across the range of populations and equipment where it will be used.  Successful deployments have used stepwise rollouts: first test the AI offline, then allow shadow usage (where humans can overrule it), and only gradually integrate it into actual care once accuracy is confirmed.  Transparency here means publishing validation results (peer-reviewed if possible) and openly acknowledging any limitations in subgroups.  It also means planning post-market surveillance: collecting real-time feedback and regularly retraining the model if needed, all while logging changes.

These best practices aren’t a guarantee, but they are ingredients of trust-building that appear in multiple health systems.  Notably, mature systems and startups alike are forming or adopting industry-wide standards.  For example, the UK’s NHS AI Lab has funded an Algorithmic Transparency Standard and tools like the PALISADE checklist to make sure new AI tools can be compared in a common way.  Hospitals in Singapore, the US and Europe are beginning to demand vendor transparency reports showing how an AI was tested on different ethnic or age groups, to ensure the tool is fair for their patients.

Towards a Trusted AI Future 

The story of AI in healthcare is still unfolding, and trust is the thread that will determine which tools become fixtures of care, and which fade into obsolescence or scandal.  For developers and product teams, the message is clear: technical innovation must go hand-in-hand with open, ethical design.  No matter how sophisticated the algorithm, without interpretability, documentation and human oversight, adoption will stall.  Conversely, an explainable AI that is slower or simpler but fully vetted may achieve far greater impact by earning clinicians’ confidence.

Actionable steps for healthtech teams include: start transparency reporting now (even before deployment); engage with local regulators early (the FDA and NICE both encourage pre-submission talks); and build multidisciplinary review processes (data scientists, clinicians and ethicists together).  After launch, monitor the AI’s performance and safety signals rigorously, and publicly share results.  All these practices not only reassure users, but are increasingly expected by regulations in the US, EU, UK and beyond.

Looking forward, several open questions remain.  How can we quantify trust in a way that guides design and certification?  What are the best ways to communicate uncertainty or potential biases to non-expert users?  As models like large language models enter medicine, how will existing transparency rules adapt?  And crucially, how will emerging health systems, with different patient mixes and regulatory environments, ensure AI benefits without reproducing past mistakes?

Trust isn’t automatic; it’s earned.  By keeping transparency, explainability and human oversight at the core of AI development, we can help health systems worldwide make the most of these powerful tools.  The alternative, backsliding into opaque black boxes and disillusionment, would betray the very patients and providers AI is meant to serve.

Sources: Authoritative reports from the FDA, NICE, WHO and EU agencies; recent peer-reviewed studies and reviews on health AI; and reputable news investigations of real-world AI deployments.