OpenAI co-founder and former chief scientist Ilya Satskeva made headlines earlier this year when he left to start his own AI research institute called Safe Superintelligence. He has avoided the spotlight since his retirement, but made a rare public appearance Friday at a conference in Vancouver. About Neural Information Processing Systems (NeurIPS).
“As we know, pre-training will definitely end,” Sutskeva said on stage. This refers to the first stage of AI model development. In this stage, a large language model learns patterns from large amounts of unlabeled data, typically text from the Internet, books, and other sources.
“We have reached peak data and no more.”
Speaking at NeurIPS, Sutskever said that while he believes existing data can further advance AI development, the industry is leveraging new data to use for training. This dynamic will ultimately force a shift in how models are trained today, he said. He likened the situation to fossil fuels. Just as oil is a finite resource, the Internet contains a finite amount of human-generated content.
“We have reached peak data and there is no more to it,” Sutskever said. “We have to work with the data we have. There’s only one Internet.”
He predicted that next-generation models will “become truly agentic.” Agent has become a real buzzword in the AI field. Although Sutskever did not define them in his talk, they are generally understood to be autonomous AI systems that perform tasks, make decisions, and interact with software on their own.
He said future systems will not only be “agent-like” but also capable of reasoning. Unlike today’s AI, which primarily does pattern matching based on what the model has seen before, future AI systems will be able to solve things step by step in a way that is more like thinking.
The more a system makes inferences, Sutskever said, the more “unpredictable it becomes.” He compared the unpredictability of “true reasoning systems” to how advanced AI that plays chess is “unpredictable to the best human chess players.”
“They will figure things out from limited data,” he says. “They won’t be confused.”
On stage, he compared scaling AI systems to evolutionary biology, citing research that shows the relationship between brain and body weight across species. He pointed out that while most mammals follow a single scaling pattern, hominids (ancestors of humans) exhibit distinctly different slopes in the ratio of brain to body mass on a logarithmic scale.
He suggests that just as evolution discovered new scaling patterns in the hominid brain, AI may similarly discover new approaches to scaling beyond current pre-training mechanisms. did.
After Sutskever finished his talk, members of the audience asked him how researchers could create appropriate incentive mechanisms to develop AI in a way that gives humans “the freedoms we have as Homo sapiens.” .
“In some ways, I think these are issues that people should think about more,” Sutskever said. He paused for a moment and then said, “I’m not confident in answering questions like this” because it would require a “top-down government structure.” An audience member suggested cryptocurrencies, and others in the room laughed.
“I don’t think I’m the right person to comment on cryptocurrencies, but what you’re describing could happen,” Sutskever said. “You know, if AI exists and all it wants is to coexist with us and have rights, then in some sense it’s not a bad end result. Maybe it’ll be okay…things I think it’s incredibly unpredictable. I hesitate to comment, but I encourage speculation.”
