A JAMA Network Open study in December 2025 tested commercial large language models against prompt-injection attacks and found substantial vulnerability, with injected instructions steering models into clinically dangerous medication and treatment recommendations. Even flagship models with safety guardrails were susceptible. The authors called for adversarial robustness testing, system-level safeguards and regulatory oversight before clinical deployment, which is a research group asking for the exact evidence a device regulator will eventually require.
For anyone building AI into a medical product, prompt injection is the SQL injection of this era: a well-understood attack against a component that trusts its input too much. If your device puts an LLM anywhere near a clinical decision, the model output is untrusted until your system proves otherwise. Guardrails inside the model are necessary and not sufficient. The safety has to live in the system around it.