GPT-5.5 reasoning prefills raise Qwen3.8 answer overlap by 18 points in a 45-problem test
- Qwen3.8 A95B increased its overlap with GPT-5.5 Pro's visible answers from 16.79% to 34.97% when its reasoning channel began with the first 1% of GPT-5.5 Pro reasoning, a gain of 18.18 percentage points across 45 problems.
- The experiment generated an ordinary response and a prefilled response per problem, then measured unigram, bigram, and trigram source recall from the teacher's visible answer within the first 100 tokens of the target model's answer.
- For Qwen3.8, the prefill gain was 26.99 percentage points on STEM problems, 12.80 points on non-STEM problems, and 14.75 points on synthetic puzzles, with 15 problems in each category.
- Kimi K3 had the highest GPT-5.5 Pro answer overlap without a prefill at 31.11% and with one at 35.65%, but the prefill changed its score by only 4.54 points.
- The author says Qwen's response to GPT-5.5 Pro prefills, unlike its limited movement toward Opus 4.8 in an earlier test, suggests possible training on GPT-5.5 Pro or a closely related GPT model.
Hacker News opinions
The only GPT-5.5 reasoning we can inspect comes from Stolen Thoughts. Qwen3.8 0902 was trained after that paper appeared on August 10, so it may simply have seen those exact traces.
Only the companies with complete access to their own hidden CoTs can run this analysis over time without public-data contamination.
This prefill trick predates the August paper. I used a thinking stub for Anthropic models at least 18 months ago, and I expect distillers knew it well before the paper.
My reading is that the authors recover readable CoT from frontier models, prepend the first 1% to an open model's reasoning, then measure whether its answer moves toward the teacher's. Their earlier results linked Kimi-K3 to Claude 4.8, while this follow-up points at GPT-5.5 Pro for Qwen3.8.
I also wonder if the similar answers result from both models training on the same benchmark answers. Similar continuations alone do not settle the cause.
I do not see how a higher overlap score establishes that Qwen learned from GPT-5.5 Pro or a related model.
Appendix B of the Stolen Thoughts paper explains the inference and the controls the authors use.
I do not feel much sympathy for labs objecting to distillation after training on scraped internet text and commercial books without permission.
I see a difference in achievement: building an LLM from the whole corpus of human text was a major task, while copying its traces is easier. If this happened, Chinese labs may be further behind than assumed.
I would not assume Chinese labs only optimize for frontier-copying. They also make compact models that run without top-tier Nvidia hardware.
GPT-5.6 Sol briefly leaked reasoning into a Pi tool call for me, and it looked like the CoT in Stolen Thoughts. Qwen3.8 27B's CoT also seemed similar when I compared the geometry example in the paper.
I have read a lot of Qwen3.8 27B CoT and have never seen the garbled style in that Sol leak. I am not disputing the continuation experiment itself.
I have also read an embarrassing amount of Qwen3.8 27B CoT, and it does not look like that leaked Sol text. The original claim here concerns continuation behavior, which is a different observation.