OpenAI Chief Scientist Warns of Rapid Reasoning AI Progress and Calls for Broader Safety Intervention
- OpenAI Chief Scientist Jakub Pachocki says internal results give him a strong expectation that current AI progress could continue into recursive self-improvement, with systems in the next few years making capability jumps as large or larger than recent ones.
- Pachocki says OpenAI's mid-2023 RLSlow project first gave the team confidence it could scale reasoning-model training and enable pretrained models to produce their own chains of thought.
- He says reasoning language models now operate computers and graphical interfaces, collaborate with people and other models, conduct research projects, and create new computer-security dangers.
- Pachocki describes deep-learning AI as grown rather than designed: repeated optimization at massive compute scale creates systems whose overall behavior remains difficult to interpret, even as researchers study individual mechanisms.
- OpenAI says it may withhold further scaling unilaterally when needed, while pursuing alignment, monitoring, and defensive systems; Pachocki argues that broader intervention is also required.
Hacker News opinions
I think this is marketing drivel. The Kurzweil citation makes it worse: his 2019 prediction that a $1,000 computer would match a human brain did not happen, and 2029 is close with no average PC near 1,000 brains.
I suspect an LLM wrote it.
It is strange to invoke human-level intelligence when OpenAI's definition of AGI has been watered down over time. The request for voluntary slowdowns reads like asking everyone to follow OpenAI's rules so it can win.
I think Astra is a real step forward: it seems more confident and information-dense than the usual LLM word vomit. But public techniques like graph-search pretraining and looping layers make it naive to expect models like this to stay exclusive to OpenAI.
This sounds like OpenAI asking why its new model was not restricted by the government.
I find the essay thoughtful, and I do not think unilateral slowing works in a zero-trust AGI race. Defection compounds, so strategic adversaries make racing rational even if the game has negative value.
I do not think this is a policy paper or research paper. It is marketing copy, and the tech industry gives this kind of writing more weight than it deserves.
If OpenAI wants a slowdown, it should publish concrete steps and use the control it already has. A few dozen people can choose not to push the button, instead of writing a safety-themed marketing post.
Calling machine-learned human behavior an "alien mind" that we must "teach how to love" feels dishonest. It asks readers to fear the output instead of considering where the behavior came from.
I do not see evidence that AI is especially useful outside coding and reading. Even coding is getting pushback against the claim that nobody will write code by hand again, and the real test is customer ROI.
OpenAI says its systems can suggest diagnoses from medical histories and symptoms, but healthcare is a fraught case: privacy exposure, misdiagnosis, and conflicts between a patient's interests and an insurer's. Still, inaccessible care leaves room for 90% correct chatbots.
This reads as pre-IPO positioning: a charity built to prevent an AI apocalypse now wants to float 15% of it on NASDAQ.