FDA’s Elsa ‘Hallucinates Confidently’: Report
Former and current FDA employees who have used the agency’s new internal generative artificial intelligence (AI) tool Elsa report that it can’t be trusted. An online Gizmodo post says three employees told CNN that Elsa makes up nonexistent studies, something commonly referred to in AI as “hallucinating.” The employees said Elsa also misrepresents research.
“Anything that you don’t have time to doublecheck is unreliable,” one employee told CNN. “It hallucinates confidently.”
Gizmodo says that people who insist that AI saves them time often are fooling themselves, citing a recent study of programmers showing that tasks took 20% longer with AI, even among people who were convinced they were more efficient.
Noting that FDA commissioner Martin Makary introduced Elsa by saying it came online ahead of schedule and under budget, Gizmodo says, “It seems like you get what you pay for (Elsa reportedly cost only $12,000 in its first week). If you don’t care about the accuracy of your work, Elsa sounds like a great tool for allowing you to get slop out the door faster, generating garbage studies that could potentially have real consequences for public health in the U.S.”
The story says the agency employees who talked to CNN said they tested Elsa by asking basic questions like how many drugs of a certain class have been approved for children. “Elsa confidently gave wrong answers,” it says, “and while it apparently apologized when it was corrected, a robot being sorry doesn’t really fix anything.”
The news network said that if an FDA employee asks Elsa for a one-paragraph summary of a 20-page paper on a new drug, there is no easy way to know if the summary is accurate. And even if the summary is accurate, CNN says there could be something in the 20-page paper that would be a red flag for any human with expertise. “The only way to know for sure if something was missed or if the summary is accurate is to actually read the report,” CNN says.
Gizmodo reports that FDA did not respond to emailed questions about what it is doing to address Elsa’s fake study issue. Makary reportedly told CNN that Elsa could “potentially hallucinate,” but said that’s no different from other large language models and generative AI.
“He’s not wrong on that,” Gizmodo concludes. “The problem is that AI is not fit for purpose when it’s consistently just making things up. But that won’t stop folks from continuing to believe that AI is somehow magic.”