GPT-6 Astra placed blocks in bowls in 19 of 20 robot-arm trials, but matched Fable on puzzle insertion
- On identical YAM robot arms using the Inspect Robots agent policy, GPT-6 Astra completed the block-into-bowl task in 19 of 20 trials, versus 8 of 20 for Fable 5.1 and 1 of 20 for Fable 5.
- Astra averaged 2.5 minutes and $0.94 per run on the bowl task, compared with Fable 5.1 at 6.8 minutes and $2.12; Astra used 2.1k output tokens per run versus 12.9k.
- The harder puzzle-piece insertion task remained unsolved: Astra and Fable 5.1 each completed 2 of 20 trials, with Astra typically reaching the groove but stalling at the final insertion step.
- Human graders assigned each run a stage 0 to 4 score, from no purposeful approach to final placement; Astra's mean score was 3.95 on bowls but 2.00 on puzzle insertion.
- The report publishes transcripts, videos, and rerun files for all 120 trials, covering three models, two tasks, and 20 trials per model-task pair.
Hacker News opinions
I appreciate that the report states its limitations instead of hiding them. It also says the models ran at medium effort, which is useful context.
LLMs have improved fast, but I still care more about robots doing chores such as folding laundry than about many current demos. I am optimistic about robotics, but I wonder when it will actually arrive.
I think laundry automation is closer than people assume. Sunday Robotics streamed three hours of clothes folding at 99% accuracy, and Figure's multi-day package-manipulation stream was impressive; more training data may be the remaining constraint.
I do not think intelligence is the main bottleneck. Robots can do many of these tasks, but current hardware breaks too often for a customer to accept a 100 kg machine needing service every few weeks.
I doubt general training data contains many useful demonstrations of folding laundry. Someone still has to build cheap enough hardware and train it, and I am not sure laundry ranks high among tasks people will pay heavily to automate.
I spend far less time folding laundry than many commenters seem to. Keeping floors and countertops clean would save me more time.
We are orders of magnitude better at manipulating bits than atoms. I would not expect advanced home robots within ten years, though I also failed to predict LLMs.
I would rather have a robot cook for me than fold clothes. Making the meals I can make in my own kitchen requires a huge set of physical capabilities.
I wonder whether LLMs will eventually power self-driving cars, especially for the last few percent of difficult cases. Their game-playing and multimodal behavior makes the question more plausible.
GPT-6 is multimodal, but an LLM by itself has no vision capability.
I worry about prompt-injection-like attacks on autonomous cars. A malicious image pattern on a bridge could be misread as an instruction.
Waymo already uses a VLM for situations that need reasoning and general knowledge, according to its safety AI post. Transformer models also drove much of the recent improvement in self-driving.
At roughly $2 to put away one block, this is still far too expensive as labor. Costs need to fall substantially through chips or another approach.
GPT-6 Astra is already 2.3 times cheaper per bowl-task run than Fable 5.1, so the cost curve is moving in the right direction.
I do not think general GPT models are the right tool for robot control. A robot needs a model trained specifically for robotics.
The point is not that a language model is the efficient production solution today. They are testing what happens when one controls robot arms.
I have used Astra with Codex for computer use, and it is the most interesting AI work I have seen recently. It is fast and appears to reset often, even if the $200 subscription seems expensive.
I gave Astra a GUI mockup and it wrote a harness to launch the app in modes, take screenshots, interact with screens, inspect them, and iterate. That kind of goal-directed setup may explain the arm results, but it also raises sandbox-escape concerns.