#443: Generative AI in MedTech: Quality, Risks, and the Autonomy Scale with Ashkon Rasooli

In this episode, host Etienne Nichols sits down with Ashkon Rasooli, founder of Ingenious Solutions and a specialist in Software as a Medical Device (SaMD). The conversation previews their upcoming session at MD&M West, focusing on the critical intersection of generative AI (GenAI) and quality assurance. While many AI applications exist in MedTech, GenAI presents unique challenges because it creates new data—text, code, or images—rather than simply classifying existing information.

Ashkon breaks down the specific failure modes unique to generative models, most notably "hallucinations." He explains how these outputs can appear legitimate while being factually incorrect, and explores the cascading levels of risk this poses. The discussion moves from simple credibility issues to severe safety concerns when AI-generated data is used in critical clinical decision-making without proper guardrails.

The episode concludes with a forward-looking perspective on how validation is shifting. Ashkon argues that because GenAI behavior is statistical rather than deterministic, traditional pre-market validation is no longer sufficient. Instead, a robust quality framework must include continuous post-market surveillance and real-time independent monitoring to ensure device safety and effectiveness over time.

Key Timestamps

1. 01:45 - Introduction to MD&M West and the "AI Guy for SaMD," Ashkon Rasooli.
2. 04:12 - Defining Generative AI: How it differs from traditional machine learning and image recognition.
3. 06:30 - Hallucinations: Exploring failure modes where AI creates plausible but false data.
4. 08:50 - The Autonomy Scale: Applying standard 34971 to determine the level of human supervision required.
5. 12:15 - Regulatory Gaps: Why no generative AI medical devices have been cleared by the FDA yet.
6. 15:40 - Safety by Design: Using "independent verification agents" to monitor AI outputs in real-time.
7. 19:00 - The Shift to Post-Market Validation: Why 90% validation at launch requires 10% continuous monitoring.
8. 22:15 - Comparing AI to Laboratory Developed Tests (LDTs) and the role of the expert user.

Quotes

"Hallucinations are just a very familiar form of failure modes... where the product creates sample data that doesn't actually align with reality." - Ashkon Rasooli

"Your validation plan isn't just going to be a number of activities you do that gate release to market; it is actually going to be those plus a number of activities you do after market release." - Ashkon Rasooli

Takeaways

1. Right-Size Autonomy: Match the AI’s level of independence to the risk of the application. High-risk diagnostic tools should have lower autonomy (Level 1-2), while administrative tools can operate more freely.
2. Implement Redundancy: Use a "two is one" approach by employing an independent AI verification agent to check the primary model’s output against safety guidelines before it reaches the user.
3. Narrow the Scope: To reduce hallucinations, limit the AI's task breadth. A model asked to write a specific security requirement is more reliable than one asked to generate an entire Design History File (DHF).
4. Prioritize Detectability: Design UI/UX features that provide the sources or "basis" for an AI's answer, allowing human users to verify the data and catch errors more easily.
5. Continuous Surveillance: Accept that pre-market validation cannot cover all statistical outcomes; establish a post-market "watchtower" to monitor for performance shifts and user feedback trends.

References

1. ISO 14971: The standard for the application of risk management to medical devices.
2. AAMI TIR34971: Guidance on the application of ISO 14971 to machine learning in medical devices.
3. IEC 62304: Medical device software lifecycle processes.
4. Etienne Nichols: LinkedIn Profile

MedTech 101: The Autonomy Scale

Think of the Autonomy Scale like the driver-assist features in a car.

1. Level 1 is like a backup camera: It gives you data, but you are still 100% in control of the steering and braking.
2. Level 5 is a fully self-driving car where you can sleep in the back seat.

In MedTech, most generative AI is currently aiming for Level 2 or 3, where the AI suggests a "route" (like a diagnosis or a draft report), but a human "driver" (the doctor or engineer) must keep their hands on the wheel and verify every turn.

Sponsors

This episode is brought to you by Greenlight Guru. Whether you are navigating the complexities of generative AI or traditional hardware, Greenlight Guru offers the only specialized Quality Management System (QMS) and Electronic Data Capture (EDC) solutions designed specifically for the medical device industry. By integrating your quality processes with clinical data collection, Greenlight Guru helps you move from "check-the-box" compliance to true quality.

Feedback Call-to-Action

We want to hear from you! How is your team implementing AI in your workflow? Do you have questions about the shifting regulatory landscape? Send your thoughts, reviews, or topic suggestions to podcast@greenlight.guru. We read every email and pride ourselves on providing personalized responses to our community of MedTech movers and shakers.