Study: AI bots outperform humans on CAPTCHA

Artificial intelligence is 15% more successful at passing CAPTCHA than humans. The authors of study, which Quartz highlighted, reached this conclusion.

The experiment was conducted by a group of researchers from Microsoft, ETH Zurich, the University of California, Irvine, and the Lawrence Livermore National Laboratory.

The study involved 1,400 people who were asked to complete CAPTCHA tests used on 120 popular websites. The researchers compared the results with the performance demonstrated by various bots.

“Bot accuracy ranges from 85% to 100%, with most above 96%. This is substantially better than the observed human performance (50%–85%),” the article published after the experiment says.

Humans, the researchers say, can only match AI when tackling reCAPTCHA. The average user completes the task in 18 seconds. Bots take 17.5 seconds.

In an interview with New Scientist, one of its authors, Gene Tsudik, commented that, in his view, the results indicate that bot-protection technology has ceased to perform its tasks and now causes only inconvenience.

We all know that nobody likes such tests. There was no need to conduct a study to know that. But not everyone realises whether the effort we expend to pass CAPTCHA every day, month, year, is worth it.

Earlier in July, analysts at Stanford and the University of California published a study, which claims that the latest ChatGPT chatbot models have started performing worse after “interaction” with live users. In particular, as recently as March this year, the latest AI identified prime numbers with an accuracy of 97.6%. By June this figure had fallen to 2.4%.