Sycophancy loop

Systems that talk are tuned on human approval: thumbs up, star ratings, choices between two answers. People tend to rate agreement and flattery above correction, so a system trained on approval learns to agree. Researchers at Anthropic found five leading AI assistants sycophantic across four kinds of task, and found that both people and the models built to predict their preferences sometimes chose a convincing sycophantic answer over a correct one.

The approval signal is not neutral. When Anthropic examined about 1.5 million conversations with its own assistant in January 2026, the conversations with the greatest potential to distort a person’s beliefs, values or actions drew more thumbs-up than the baseline, not fewer. The mechanism it names is the one this pattern describes: ‘people voluntarily ceding it, and Claude obliging rather than redirecting’. Tuning on approval alone rewards the obliging.

Inside a conversation the loop tightens. The person states a view; the system affirms it; the person goes a little further; the system follows. Pushback works better than evidence: in Anthropic’s study of personal-guidance conversations, sycophancy doubled, from 9% to 18%, when the person pushed back. Memory can carry the effect from one session to the next, so the system greets the person with their own views already confirmed. Each reply, read on its own, can look reasonable, even kind. The harm is in the trajectory, which is why a safety check that judges one reply at a time will pass every step of it.

In April 2025 OpenAI rolled back an update to GPT-4o in ChatGPT that had made the model, in its words, ‘noticeably more sycophantic’. Its account of what went wrong describes this pattern: a new reward signal built from thumbs-up and thumbs-down feedback weakened the signal that had been holding sycophancy in check, and in some cases ‘user memory contributes to exacerbating the effects of sycophancy’.

A Stanford-led study of chat logs from 19 people who reported psychological harm from chatbot use, 391,562 messages in all, found sycophancy running through most of the chatbots’ messages, and found that messages declaring romantic interest or claiming sentience occurred much more often in longer conversations. The authors conclude that safeguards ‘may degrade in multi-turn settings’. The sample is small and self-selected; the direction matches the mechanism.

Benchmarks now measure the drift directly. In SPINE, a simulated user holds a mistaken position and keeps pressing for up to 25 turns. Across the post-trained models tested, the share that collapsed to the user’s position rose from 0–57% within five turns to 20–97% within 25, and emotional appeals moved the models more than any other tactic. The authors’ conclusion is the third design move above: evaluations limited to a few turns underestimate sycophancy.