Let FDA Set AI Postmarket Standards: Califf
FDA commissioner Robert Califf says it would be helpful if Congress gave the agency authority to set standards in the postmarket phase of artificial intelligence (AI) usage. Califf was interviewed on setting guardrails for AI in health care by JAMA+ AI editor-in-chief Roy Perlis in a JAMA article.
“If you go back even 10 years ago, as algorithms were more and more coming in,” Califf explained, “you had guidances from FDA saying the premarket phase is important, but this postmarket phase of monitoring the model, what the algorithm is doing, is critical. So more authority to make that happen. Not to do it, but to say if you’re putting an algorithm out there, you should be aware that you could be audited for whether your algorithm is doing what it says.”
Throughout the interview, Califf stressed the need for mutual accountability between regulated industries and the agency, noting that FDA will never have the resources to monitor everything that happens in the postmarket AI environment.
“We’re very dependent on the industries following the guidances, regulations, and laws that are in place,” he said…. “Think of an AI algorithm like an ICU patient being monitored as opposed to drugs and devices in the old-fashioned way. You make a drug, you do a clinical trial, it’s got to be the same drug for its lifetime. That’s not the case with AI…. AI, as it’s embedded in a device or AI as a decision-support algorithm, it’s out there in the market and we have a joint responsibility in the postmarket phase to take a step up in terms of our assessment capabilities. And this is I think the biggest area where FDA has a role to play because there are probably algorithms out there now that haven’t been through FDA being used by health systems where no one knows what the drift is in terms of whether they’re accurate, getting better or worse over time.”
He said FDA can’t do that work on its own and will need clinical health systems and professional societies to get very involved in self-regulation.
“I think so much of the power lies with health systems and the clinicians who work in them to demand that when AI is used, that’s it’s used intelligently,” Califf said …. “If you say you are predicting sepsis, how often are you accurate? What’s the sensitivity, specificity, and the ability to discriminate sepsis from non-sepsis patients in your prediction? That’s got to be continuously looked at to make sure you’re doing it right.”