More Comprehensive AI Regulation Needed: Report

AI researchers in the U.S. and Canada are calling on FDA to foster an agile, transparent, and ethics-driven artificial intelligence (AI) oversight system that can balance innovation with patient safety, ensuring that AI-driven medical technologies enhance, rather than compromise, healthcare outcomes. Writing in the open-source PLOS Digital Health, the researchers cite “significant weaknesses” in FDA’s oversight to date.

“A truly comprehensive regulatory framework for AI in healthcare must integrate ethical, technical, societal, and practical considerations to ensure responsible and effective oversight,” the authors write. “Establishment of a specialized regulatory body equipped with both legal authority and technical expertise to oversee AI development and deployment would address this need.”

They say the FDA regulatory framework would benefit from actively involving patients in the decision-making process as they are the ultimate end-users of AI technologies in health, making their insights crucial to ensuring these products meet their needs and uphold safety standards. “While FDA claims to incorporate public engagement into its regulatory framework,” the report says, “its current approach often feels performative, primarily soliciting feedback from individuals with the time, resources, and knowledge to participate…. True public engagement must be more expansive, inclusive, and intentional, ensuring that the perspectives of those most affected by these technologies, including through trusted patient and community advocates, have a voice in how these technologies are evaluated and implemented.”

The report recommends a comprehensive framework that includes:

• mandating extensive post-marketing monitoring and requiring updates to approvals if significant drift or performance degradation is detected;
• ensuring AI developers disclose the composition of their training data and the methodologies behind their models for full transparency;
• requiring evaluations of how AI tools perform across different demographic groups to mitigate biases;
• involving a diverse range of stakeholders, including manufacturers, healthcare providers, patients, and regulatory bodies, in the development of AI regulations to ensure comprehensive and effective oversight; and
• community sourcing the regulation of health AI by actively including patients and their caregivers, especially those from historically underrepresented groups such as ethnic minorities, women and gender minorities, individuals from lower socioeconomic strata, people with disabilities, elderly individuals, and rural or underserved communities.

“The key to effective regulation,” the report concludes, “is fostering a system that is constantly learning and consistently engaging with those most likely to be harmed by these technologies. Collectively embedding these principles within FDA regulations will promote responsible and effective use of healthcare AI.”