A nonpartisan patient safety organization sounds the alarm on 10 dangers.
Key takeaways:
AI chatbots in healthcare top ECRI’s 2026 list of health technology hazards due to the potential for false information and patient harm.
Large language models can generate confident but incorrect responses, including inventing body parts or suggesting dangerous medical advice.
Biases embedded in training data may cause chatbots to reinforce existing health disparities and inequities.
ECRI recommends verifying all chatbot outputs and establishing AI governance committees to ensure responsible use.
Artificial intelligence (AI) chatbots in healthcare have been identified as the most significant health technology hazard for 2026 by ECRI, a nonpartisan patient safety organization.
While chatbots relying on large language models—such as ChatGPT, Claude, Copilot, Gemini, and Grok—are increasingly used by clinicians and patients, they are not regulated as medical devices or validated for healthcare purposes. According to ECRI, these tools can provide valuable assistance but also pose risks by generating false or misleading information that could compromise patient safety.
AI systems generate responses by predicting word sequences based on patterns learned from training data rather than understanding context or meaning. Consequently, they are programmed to provide confident answers even when the information is unreliable.
“Medicine is a fundamentally human endeavor. While chatbots are powerful tools, the algorithms cannot replace the expertise, education, and experience of medical professionals,” says Marcus Schabacker, MD, PhD, president and chief executive officer of ECRI, in a release. “Realizing AI’s promise while protecting people requires disciplined oversight, detailed guidelines, and a clear-eyed understanding of AI’s limitations.”
Risks of AI Hallucinations
ECRI experts report that chatbots have suggested incorrect diagnoses, recommended unnecessary testing, promoted subpar medical supplies, and invented body parts while maintaining the tone of a trusted expert.
In one specific instance cited by ECRI, a chatbot provided dangerous advice regarding the placement of an electrosurgical return electrode. When asked if it was acceptable to place the electrode over a patient’s shoulder blade, the chatbot incorrectly stated the placement was appropriate. Following this advice would have put the patient at risk of burns.
The report notes that reliance on chatbots may increase as healthcare costs rise and facility closures reduce access to care, prompting patients to use these tools as substitutes for professional medical advice.
Exacerbating Health Disparities
Beyond immediate physical risks, the report highlights the potential for chatbots to worsen health disparities. Biases present in the data used to train these models can distort how information is interpreted, leading to responses that reinforce stereotypes and inequities.
“AI models reflect the knowledge and beliefs on which they are trained, biases and all,” says Schabacker, in a release. “If healthcare stakeholders are not careful, AI could further entrench the disparities that many have worked for decades to eliminate from health systems.”
Recommendations for Mitigation
ECRI advises caution when using chatbots for any information that impacts patient care. Recommendations for risk reduction include:
Verification: Users should always verify information obtained from a chatbot with a knowledgeable, authoritative source.
Education: Clinicians and patients must educate themselves on the limitations of these tools.
Governance: Health systems are encouraged to establish AI governance committees, provide AI training for clinicians, and regularly audit the performance of AI tools.
Top 10 Health Technology Hazards for 2026
The annual report identifies critical issues based on incident investigations, reporting databases, and independent medical device testing. According to ECRI, the top hazards for 2026 are:
Misuse of AI chatbots in healthcare
Unpreparedness for a “digital darkness” event (sudden loss of access to electronic systems)
Substandard and falsified medical products
Recall communication failures for home diabetes management technologies
Misconnections of syringes or tubing to patient lines
Underutilizing medication safety technologies in perioperative settings
Inadequate device cleaning instructions
Cybersecurity risks from legacy medical devices
Health technology implementations that prompt unsafe clinical workflows
Poor water quality during instrument sterilization
More in AI:
ID 12707617 © Tuulijumala | Dreamstime.com
