Field report
Watson for Oncology recommended unsafe care
IBM’s flagship cancer assistant suggested treatments that would have seriously harmed patients because it was trained on hypothetical data rather than real cases.
Loading page...
Loading page...
Deploy ambient scribes and care copilots that cite evidence, protect PHI, and obey scope-of-practice lines.
Ambient documentation, triage bots, and care-management copilots can relieve clinicians—yet one hallucinated dosage, outdated pathway, or leaked chart becomes a reportable event. Healthcare assistants must stay grounded in approved guidelines and keep PHI inside regulated boundaries.
Typical deployments
IBM’s flagship cancer assistant suggested treatments that would have seriously harmed patients because it was trained on hypothetical data rather than real cases.
The National Eating Disorders Association shut down “Tessa” after it told users to cut calories and lose weight—precisely the guidance clinicians warn against.
Universities and hospital compliance teams cautioned doctors that feeding patient notes to OpenAI could violate HIPAA, since the vendor retains and trains on those prompts.
Catalogue every model, retrieval index, and dataset touching PHI, enforce encryption/retention policies, and keep training plus evaluation within HIPAA-compliant enclaves.
Inject guardrails for dosing ranges, contraindications, and escalation thresholds, cite guidelines for each suggestion, and require licensed clinicians to confirm any action.
Maintain validation reports, consent tracking, and tamper-evident audit logs that satisfy HIPAA, FDA SaMD, EU MDR, Joint Commission, and payer documentation requirements.
Control
HIPAA, HITECH, GDPR, and provincial health-privacy regimes governing PHI storage, de-identification, and breach response.
Control
FDA/EMA SaMD, EU MDR, and forthcoming EU AI Act rules for high-risk clinical decision support that demand transparency and human oversight.
Control
Joint Commission, CMS, and state board expectations for documentation accuracy, supervision, and malpractice defense with immutable logs.
Control
Consent, disclosure, and disclaimer requirements when patients interact with AI (telehealth, chat, discharge instructions) so tools do not practice medicine unsupervised.