Many AI/ML Framework Comments Lack Evidence: Study
University of Oxford researchers say their review of comments on a proposed FDA regulatory framework for modifications to artificial intelligence and machine learning-based software as a medical device found that 63% of the comments came from parties with financial ties to industry and 86% of comments did not cite any scientific evidence. The results of their study were published in BMJ Open.
“We found no evidence that submitters without a financial interest were more likely to cite scientific evidence or contribute more comprehensive comments,” the researchers say. “… There were far fewer academic submissions than those from industry.”
The article says the researchers recommend that FDA request a statement of interests in comments to help further determine the extent of conflicts of interest in future public consultations. They also say the agency could reject anonymous comments unless there is a clear reason that anonymity would improve comment quality. And they call on FDA to encourage more academic participation and to call for the use of scientific evidence in public comments.
The report’s lead author, postdoctoral scientist James Smith, said in a university statement that the process for development regulations “is, roughly, to get feedback from the public on its initial proposal, make changes and draft regulations or guidance, get more feedback, and eventually finalize. Anyone can comment but at present there is no requirement, or even recommendation, to disclose any conflicts of interest. Also, FDA states that it looks for ‘good science’ in comments but it is not a requirement to incorporate it. Our goal was to look at the extent and disclosure of financial ties to industry and the use of scientific evidence.”