Former FDA Officials Warn Industry Against Overreliance on AI

Medical device manufacturers increasingly using artificial intelligence (AI) for quality assurance and regulatory functions must maintain human oversight of those systems, according to former senior FDA officials who say the agency is also beginning to incorporate AI into its inspection and postmarket surveillance activities.

Speaking on the 10/26 FDA Watch podcast, Phil Desjardins, a partner at Arnold & Porter and former CDRH associate director for policy, said the industry's use of AI is expanding beyond AI-enabled products to internal operations traditionally handled by trained quality, regulatory and research personnel.

Desjardins said FDA has already issued Warning Letters involving companies that relied too heavily on AI in their quality systems.

"We've seen at least a couple of warning letters from FDA already where companies have become over-reliant on AI, particularly with regards to their quality systems," he said.

The concern, he explained, is that manufacturers may delegate monitoring and decision-making responsibilities to AI systems without maintaining adequate human supervision. Desjardins said AI can identify emerging trends and potential quality problems, but manufacturers cannot simply rely on the technology to determine when intervention is necessary.

"It's fine to have AI take a look and sort of identify, hey, we're seeing trends, we're seeing signals over here, but you still have to have some type of human intervention, double-checking the work that's being done, not simply relying on the outputs of artificial intelligence," he said.

Desjardins did not identify the Warning Letters or companies involved, but his concerns echo those raised in an earlier analysis by former FDA Office of Enforcement director David Elder.

In a guest column published on Pharmaceutical Online, Elder, now a principal at Eliquent Life Sciences, noted that FDA has already begun signaling increased scrutiny of industry AI applications. One example he cited was an April Warning Letter involving what the agency described as inappropriate use of artificial intelligence in pharmaceutical manufacturing (see earlier story).

Separately, law firm Morgan Lewis recently issued a client alert (see story) emphasizing that the Warning Letter underscored a fundamental regulatory principle: companies remain fully accountable for compliance, regardless of whether AI tools are involved.

"This Warning Letter sends an unambiguous message that reliance on AI is not a defense against regulatory violations," the Morgan Lewis analysis noted, adding that FDA expects human oversight to remain central to regulated activities.

FDA's Use of AI in Inspections
Sean Boyd, former CDRH Office of Regulatory Programs director, who now serves as corporate vice president at Quality Hub, said FDA's own increasing use of AI is another development manufacturers should monitor. Boyd said he is hearing that FDA investigators are using AI-enabled tools to help guide questions during inspections, determine the direction of inspections and evaluate information provided by manufacturers.

He also cited the potential use of AI to assist FDA personnel in analyzing adverse event reports and postmarket surveillance data.

Boyd suggested that manufacturers consider how FDA's adoption of these technologies could affect the information they submit to the agency and the documentation they present during inspections.

Both former officials emphasized that AI should supplement rather than replace human judgment in regulatory and quality-related activities.

Rusty Segan, former CDRH Office of Product Evaluation and Quality director and now CEO of Medical and Scientific Advisors, said regulatory oversight of AI-enabled devices should depend heavily on the technology's intended use, degree of autonomy and potential consequences of erroneous outputs.

For example, he said, an AI system providing insulin-dosing recommendations could present substantially greater risks than one providing general wellness information. Segan said FDA's evolving approach should focus on the clinical impact of AI, its potential benefits and harms, and the level of regulatory oversight appropriate to those risks.

The discussion followed AdvaMed's annual MedTech conference, where AI regulation was among the major policy issues addressed by FDA officials and industry representatives.

Desjardins said FDA is continuing to evaluate more than 500 comments on a recently published AI policy framework, while also considering incremental changes to existing policies.

Segan predicted that the agency's work could lead to additional guidance or draft guidance in 2027.

The three former officials agreed that AI offers opportunities to improve medical device development, manufacturing and regulatory oversight, but said those benefits depend on appropriate safeguards and continued human accountability.