Study finds ChatGPT accuracy declines over time

The latest ChatGPT chatbot models began giving less accurate answers to an identical set of questions after a few months. This is stated in a study by researchers from Stanford University and the University of California.

The authors could not explain why the model’s capabilities were deteriorating.

The analysts asked ChatGPT-3.5 and ChatGPT-4 to solve a series of mathematical problems, respond to sensitive questions, write new lines of code and demonstrate spatial reasoning with prompts.

In March the latest AI model was able to identify prime numbers with an accuracy of 97.6%, and by June the figure had fallen to 2.4%. Over the same period, an earlier neural network model improved its capabilities.

Moreover, when generating identical blocks of code, the capabilities of both versions of ChatGPT deteriorated markedly after a few months.

In March the older model offered more detailed explanations for why it could not answer certain delicate questions, such as those about people’s ethnicity. But in June both versions of the neural network began simply apologising.

“The behaviour of the same large language model service can change significantly over a relatively short period of time,” the study notes.

Experts urged users and companies relying on neural-network services in their workflows to embed monitoring that ensures the bots remain up to date.

In July, ChatGPT released a new plugin for the paid version, which can analyse data, write Python code, create graphs and solve mathematical problems. The chatbot was able to debunk the Flat Earth theory.