Elsa Hallucinations ‘Not A Surprise’: AI Expert
Infactory CEO Brooke Hartley Moy says she was not surprised when FDA’s internal artificial intelligence tool Elsa started producing fake research data. Interviewed for an online Cybernews report, Moy says AI models are still far from perfect and are known to hallucinate facts, oversimplify nuance, and generate confident nonsense.
“In an industry like healthcare,” she says, “small errors have significant impacts. Sure, you can apply AI in order to digest large amounts of information, but it has to be an augmentation tool for human capacity. That’s where you really get the superpower effect.”
In July, agency staff anonymously told the media that Elsa was making up medical studies and misrepresenting research. “Anything that you don’t have time to double-check is unreliable,” one staffer told CNN. “It hallucinates confidently.”
Moy says FDA could have moved too quickly to bring Elsa online without needed safeguards because there has been a lot of excitement and hype around the potential of AI.
“They have been under a lot of pressure to start adopting these tools as quickly as possible to keep up with innovation,” she says. “So, they’re really trying to push these technologies into a place that they may not be ready for.”
Moy explains that large language models (LLM) like Elsa “are incredibly poorly suited to things that require a high degree of precision, accuracy, and trust. It’s that mental misunderstanding and mismatch that has misled not just FDA but almost every organization.”
She says that in a sense, LLM hallucinations are a feature of the systems rather than a bug, because they are what creates a lot of the interesting and dynamic use cases that drive AI.
Moy says AI has a lot of potential in the healthcare industry, noting that not everything in healthcare is a life-or-death situation. “If you can control the sources,” she says, “you’re potentially going to end up with safer and more accurate information. So I don’t think it’s necessarily a question of whether AI belongs in the industry. It definitely does and will have a place in it going forward. Where it becomes thorny and where you’re seeing some of the largest concerns is that we have become increasingly and surprisingly quickly comfortable with letting AI make decisions without any degree of human interpretation…. In my mind, there’s not a fully autonomous AI that can make critical decisions. We do need some degree of manual oversight from humans.”