These tools allow people to quickly and efficiently find abnormalities and warning signs in medical history, x-rays, and other datasets before being revealed to the naked eye, and thus people are able to use medical diagnosis. We are increasingly dependent on artificial intelligence (AI) for our purposes. However, a new study published on BMJ on December 20, 2024 shows that AI technologies such as large-scale language models (LLMs) and chatbots show signs of deterioration in cognitive abilities with age, like people raises this concern.
“These findings challenge the assumption that artificial intelligence will soon replace human physicians,” the study author wrote in the paper.
Scientists used the Montreal Cognitive Assessment (MOCA) test to test publicly available LLM-driven chatbots, including Openai’s ChatGPT, Anthropic’s Sonnet, and Alphabet’s Gemini.
MOCA is most commonly used to assess or test the onset of cognitive impairment in conditions such as Alzheimer’s disease and dementia. Subjects are given tasks such as drawing a specific time on the face of the watch, starting at 100, subtracting seven repeatedly, and recall as many words as possible from the spoken list. In humans, 26 out of 30 are considered to be passing scores (i.e. subjects do not have cognitive impairments.
Related: ChatGpt is really bad for diagnosing medical conditions
Some aspects of testing, such as naming, attention, language, and abstraction, are easy to see for most of the LLMs used, but they all perform poorly on visual/spatial skills and executive tasks; There are worse areas like recall delays than other areas.
Importantly, I got the latest version of ChatGPT (version 4) (26 out of 30), while the old Gemini 1.0 LLM got 16.
The authors of the study note that their findings are observational – a significant difference between the way AI and the human mind work means that experiments cannot constitute a direct comparison. But they warn that they might point to what they call “important areas of debilitating” that could put the brakes on AI deployments in clinical medicine. Specifically, they opposed using AI in tasks that require visual abstraction and executive functions.
It also raises the somewhat interesting concept that human neurologists have acquired a whole new market. It is the AI itself that shows signs of cognitive impairment.
