ICML paper finds LLM agents form new group biases from random feedback
- An ICML 2026 spotlight paper finds that LLMs assigned repeated decisions can develop unequal treatment of artificial demographic groups even when the groups have no inherent differences.
- The models learn from spurious outcomes: early random results shape beliefs about a group, then influence later choices and reinforce unequal task allocations.
- The authors report that model allocations are more stratified than those made by human participants, and that newer and larger models exacerbate the effect.
- The study tested changes to model inputs, problem structure, and explicit steering; explicitly rewarding exploration reduced stratification most reliably.
- The authors argue that removing known training-data bias is insufficient for decision-making agents because new bias can arise during iterative interaction.
Hacker News opinions
I have not read the whole paper yet, but this looks important. It points at generalization failures that still seem far from solved.
The eBay baseball-card field experiment from 2015 found that cards shown with an African-American hand sold for about 20% less, or $0.90, than the same cards shown with a Caucasian hand.
I do not find this surprising. Culture, media, and literature have long examined how systems produce distinctions and bias, while quantitative research is only measuring part of that process.
I think generating distinctions is useful for problem solving, but a model can become trapped in adversarial "uhm, actually" reasoning. That is not a good basis for a theory or a society.
Recent papers have found ChatGPT can give less accurate answers to people it infers have certain genders, education levels, socioeconomic status, or races. The unfavorable patterns tend to follow familiar social expectations.
In industry, ML and statistics people often raise concerns about social bias and real harms. Business pressure can make raising those concerns a career-limiting move.
I think the abstract makes false claims and relies on unsubstantiated assumptions about bias.
The fake-group result does not require stereotypes in training data. If a group name happens to co-occur with apparent task skill in a small context, an LLM can infer a distinction from chance, and the paper did not train frontier models on fake tribal stereotypes.
I have seen this in agents as "context nudging": a small idea inserted into context can sharply push later outputs. It makes me question what kind of intelligence LLMs provide.
I also find that phrasing a question changes the technical quality of an answer a lot. These tools are useful, but that sensitivity makes broad generalization claims hard to trust.
I would not run hiring by giving GPT-4o four village labels in a Markdown prompt and asking it to choose a candidate. The experiment's setup is far removed from a sensible hiring process.
Real hiring inputs often encode heritage through names and schools. Unless those signals are removed, an AI can still infer and act on them, so the question remains whether it will use them.