AI Hits a Ceiling: Startups Seek New Paths for Scaling

OpenAI’s forthcoming AI model is expected to show less performance improvement compared to its predecessors, according to The Information, citing sources. 

The publication reports that Orion reached the level of GPT-4 after completing 20% of its training. This suggests that the performance gain of GPT-5 over GPT-4 will be less than that from GPT-3 to GPT-4. 

“Orion is not better than its predecessor in solving certain tasks. It performs well in language tasks but does not surpass previous models in coding,” startup employees told the publication. 

The most noticeable improvements in neural networks typically occur in the early stages of training. Progress slows in the subsequent period. Thus, the remaining 80% of the time is unlikely to yield significant performance gains, sources told The Information.

AI Hits a Ceiling

OpenAI’s less than optimistic results point to a more fundamental issue facing the entire industry: the depletion of high-quality training data. 

A study published in June by several experts claims that AI companies will exhaust all publicly available text materials between 2026 and 2032. This will become a critical point for traditional approaches to artificial intelligence development. 

“Our results show that current LLM development trends cannot be sustained solely through traditional data scaling,” the authors assert. 

The study emphasizes the need to develop alternative approaches to improving neural networks, such as generating synthetic data or using proprietary information. 

The Information noted that the current strategy of training LLMs on publicly available text data from websites, books, and other sources has reached a point of diminishing returns, as “developers have squeezed all they can from this type of information.” 

A Solution Exists

OpenAI and other players are radically changing their approaches to AI development. 

“Amid the slowdown in GPT improvements, the industry seems to be shifting focus from scaling during training to optimizing models after their initial training. This approach could lead to the formation of new scaling laws,” reports The Information.

To achieve continuous improvement, OpenAI is dividing model development into two distinct directions:

• The O series focuses on reasoning capabilities. These models operate with significantly higher computational intensity and are designed to solve complex tasks. The computational requirements are significant: operational costs are six times higher compared to current models. However, the enhanced reasoning capabilities justify the increased costs for specific applications requiring analytical processing;
• In parallel, the GPT series is being developed, aimed at general communication tasks. The model uses a broader knowledge base.

During an AMA session, OpenAI’s product director Kevin Weil noted that there are plans to merge both developments in the future. 

The Risks of Synthetic Data

The approach to solving the data shortage problem through artificial creation may pose a risk to information quality, according to a study by experts from various UK universities.

They argue that such a solution could ultimately disconnect AI from reality and lead to a “model collapse.” The problem lies in the neural network using unreliable data to form the training set for the next generation of artificial intelligence. 

To address the issue, OpenAI is developing filtering mechanisms to maintain information quality, integrating various verification methods to separate high-quality content from potentially problematic material. 

Post-training optimization is another relevant approach. Researchers are developing methods to enhance neural network performance after the initial tuning phase, without relying solely on expanding the information set. 

Previously, media reported on OpenAI’s plans to launch the next advanced AI model, codenamed Orion, by December. Later, company head Sam Altman denied this information. 

Approaches of Other Companies

Several scientists, researchers, and investors told Reuters that the methods underlying the recently introduced AI model o1 “could change the arms race” in artificial intelligence. 

In September, OpenAI introduced the large language model o1, trained using reinforcement learning to perform complex reasoning. The neural network can think—it is capable of creating a long internal chain of thoughts during question analysis, the company stated.

Co-founder of AI startups Safe Superintelligence (SSI) and OpenAI Ilya Sutskever noted that the results of training with large volumes of unlabeled data “have peaked.” 

“The 2010s were the age of scaling, and now we are back in times of wonders and discoveries. Everyone is looking for something new,” he noted. 

Sutskever declined to share details of his new company SSI’s work, noting only the existence of an alternative approach to expanding the scale of pre-training. 

Reuters sources noted that researchers from major AI labs face delays and unsatisfactory results in their quest to create a large language model surpassing OpenAI’s GPT-4, released nearly two years ago.

They are trying to apply techniques to improve neural networks during the so-called “inference” phase. For example, instead of providing one answer, AI first generates several options and selects the best one. 

In October, media reported on OpenAI’s work on its own AI chip.