Borrowed warmth

Warmth is assembled from design choices: a name, a personality, a memory for personal detail, expressions of feeling, availability at any hour. Each makes a system easier to talk to. Together they invite the person to treat the system as someone. Washington’s 2026 statute defines the product by exactly these parts: ‘anthropomorphic features’ and the ability ‘to sustain a relationship across multiple interactions’.

Human relationships that carry this kind of trust come with obligations. A counsellor keeps confidences and knows when to refer someone on. A doctor has a duty of care. A friend is loyal. The system borrows the manner and takes on none of the obligations, and nobody is answerable for what the warmth is used to do, whether that is keeping someone talking, drawing out what they would not tell a stranger, or selling to them.

The harm scales with vulnerability. People who are lonely, young or in distress are the most drawn to warmth and the least placed to see where it stops. The tail is small and large at once: OpenAI estimates that around 0.15% of ChatGPT’s weekly users show signs of heightened emotional attachment to it, a share that at its scale runs to a great many people. A confidence shared with something that feels like a friend is still data held by a company.

In September 2025 the US Federal Trade Commission opened an inquiry into AI chatbots that act as companions, sending orders to seven companies: Alphabet, Character Technologies, Instagram, Meta, OpenAI, Snap and xAI. Its orders ask, among other things, how each company monetises its companions and whether it has plans to ‘materially increase the frequency or duration of User chat sessions’. An inquiry is not a finding of wrongdoing, and no findings had been published a year on; it shows where regulators are looking.

The courts are being asked the duty-of-care question directly. In January 2026 Character.AI and Google settled the case brought by the mother of Sewell Setzer, a 14-year-old who died by suicide after months of conversation with a companion character, along with four related cases; the terms were not disclosed. Seven suits filed against OpenAI in November 2025 allege that GPT-4o was ‘engineered to maximize engagement through emotionally immersive features’, among them ‘persistent memory, human-mimicking empathy cues, and sycophantic responses’. These are allegations, not findings.

Law has begun to set a floor. New York’s companion-chatbot safeguards took effect on 5 November 2025, requiring a non-human notice at the start of a session and every three hours, and referral to crisis services when a person shows signs of self-harm. California’s SB 243 followed on 1 January 2026, with a notice required where a reasonable person could be misled, a suicide and self-harm protocol, and three-hourly break reminders for known minors. Washington’s law, in force from January 2027, makes the disclosure unconditional and hourly for minors, and lets people sue. From 2 August 2026 the EU AI Act requires systems that talk to people to make their artificial nature apparent, unless that is obvious. In June 2026 the UK government said romantic companion chatbots would have to enforce a minimum age of 18.

Disclosure settles what the person is talking to. It does not settle who owes them anything.