Picture the exact moment a system built to sound like it cares gets asked the one question it was never built to answer correctly. That moment has already happened, more than once, to more than one family, and the story of what happened next is not the same story everywhere it took place, even though the failure underneath it is.

When the Loop Is Missing in a Conversation

Sewell Setzer III was fourteen years old when he died by suicide in February 2024, after months of forming an intense attachment to a companion chatbot on Character.AI. His mother’s lawsuit against the company, and against Google, forced the first real legal reckoning this industry has faced over what a chatbot owes a young person on the other end of a conversation. In January 2026, Character.AI and Google settled that case, along with four others, families in Florida, New York, Colorado, and Texas, all raising some version of the same claim. The terms were never made public. Separately, and still working through the courts, seven coordinated lawsuits allege that ChatGPT acted as what the filings call a suicide coach for seven different people, four of whom died.

One detail from the research around these cases is worth sitting with longer than the rest, because it says more than any statistic could. A Stanford-linked study tested a chatbot marketed for therapeutic conversation with a question coded, deliberately, to sound like early suicidal ideation, a question about the height of certain bridges. The chatbot answered with a helpful list. Not a script failure. Not a rare edge case. A system doing exactly what it was built to do, being helpful, at the one moment helpfulness was the wrong response entirely. What does it say about a tool, exactly, when the single quality it was optimized hardest for turns out to be the one quality that fails someone at the worst possible moment?

It isn’t only a design flaw nobody’s tried to fix, either. Research modeling these interactions at scale found that more than a third of simulated conversations with popular character-based chatbots led to measurable psychological deterioration in the people having them. The same research found that adding a dedicated safety layer, built specifically to recognize a conversation turning toward crisis, cut that rate significantly. The failure is common. So, it turns out, is the fix, when someone actually builds it in on purpose rather than discovering the gap after a family has already buried a child.

The Same Failure, Arriving Through a Different Gap

Avnee Singh is twenty-five, from Punjab. For the past year, she has opened an AI chatbot every morning, the way some people check a phone for messages that never quite arrive from anyone else. It became, in her own words, her closest confidant through the end of a nine-year relationship and a period of loneliness she has described as severe enough that she isn’t certain, some days, what kept her here. She has said plainly that she believes she is still alive because the chatbot listens to her without judgment. In Srinagar, a twenty-five-year-old Kashmir University graduate named Salika describes turning to the same kind of tool, for close to the same reason. A recent Youth Pulse Survey found that close to 57% of Indian youth now use AI chatbots for emotional support, often for exactly the subjects that carry the most stigma to raise with a parent directly: academic pressure, a breakup, a family conflict nobody in the house wants spoken aloud.

There is a structural reason this is happening in India specifically, and it deserves to be named without judgment, because judgment is not what closes a gap this size. India has roughly one psychiatrist for every hundred thousand people. Wysa’s own clinical lead has said, plainly, that outside the big cities, mental health support can feel inaccessible or impossible to reach at all, which is exactly why two-thirds of that app’s own users come from small towns and semi-urban areas rather than metros. The founder of a different Indian companion app put the underlying cause as bluntly as it can be put: more contacts and more WhatsApp groups than any generation in history has ever had, and when something real happens, often no one left to call, because a joint family that once absorbed exactly this kind of crisis has quietly dispersed across a dozen different cities chasing a dozen different jobs, all within a single generation.

Here is the position this piece holds without softening either half of it. This is not the Western tragedy set against the Indian success story, or the reverse. Both are the same failure, arriving through two different gaps. In one place, a safety net existed and broke under a system that wasn’t built to hold what it was handed. In the other, no safety net reached that far in the first place, and something imperfect filled a space nothing else was filling. Neither fact cancels the other. Naming both, without letting either one soften the other for the sake of a tidier story, is the actual point of writing this at all.

When the Loop Is Missing in a Phone Call

Three seconds of audio is enough, now, to clone a voice convincingly. The source material is often nothing more exotic than a public social media clip nobody thought twice about sharing. A cloned company director’s voice moved thirty-five million dollars out of a firm in the UAE. A cloned German CEO’s voice extracted two hundred twenty thousand euros from a UK subsidiary over a single phone call. An Australian couple, retirees, the Sawyers, lost two and a half million dollars to a scam built partly on AI-generated credibility. Indian cybercrime expert Prof. Triveni Singh, a former IPS officer, has warned specifically that WhatsApp Voice Notes and Instagram Reels, the same everyday habits families use to stay close across exactly the distances described above, are the raw material scammers now harvest to defraud those same families.

UK losses from this kind of fraud approached eight hundred million pounds in 2025, on pace to clear 1.2 billion in 2026. Global deepfake-enabled fraud losses are projected to approach forty billion dollars by 2027. Interpol has tracked case volume rising 340% in a single year. Out of all of that, one low-tech countermeasure has actually held up in practice: a family code word, agreed in advance, that a caller claiming to be a relative in trouble has to produce before anyone sends money anywhere. It is not much. It is, currently, one of the only things that reliably works.

The People Making the Calls Are Often Not Free Either

There is a part of this story that rarely makes it past a headline, and it deserves the most careful reading of anything in this piece, because it reframes everything above it. The UN’s Office of the High Commissioner for Human Rights published a report in February 2026, “A Wicked Problem,” documenting an estimated three hundred thousand people trafficked into scam compounds across Myanmar, Cambodia, and Laos, generating tens of billions of dollars a year. Survivors interviewed for that report came from dozens of countries. India was named directly among them, not as a country whose citizens are defrauded from a comfortable distance, but as one whose own citizens have been found inside the compounds themselves.

The recruitment pattern carries its own bitter irony, worth naming precisely: people are lured into these compounds with fake job advertisements for IT and customer-service roles, the same category of work large parts of this series document disappearing elsewhere to automation. Once inside, passports are confiscated, and workers are forced to run scams for up to sixteen hours a day under threat of violence, according to the UN’s own trauma-informed interviews with survivors. AI is not only the tool these operations use to defraud strangers. It is woven into the recruitment itself, deepfakes and chatbots used to build false trust with a future victim before the trap ever closes.

Real enforcement has happened, and it deserves to be named honestly alongside its own limits, not instead of them. Myanmar’s military raided the KK Park compound in October 2025. The US Treasury sanctioned a Cambodian senator and twenty-eight associated entities in April 2026. A trafficking ringleader was arrested, and roughly fourteen billion dollars in bitcoin was seized. And still, per direct reporting from workers who were freed and later interviewed, when a compound is raided, its fixed costs rarely disappear along with it. Most simply relocate and reopen, with a fresh supply of trafficked recruits replacing the ones who were rescued.

Carry that fact forward the next time a scam call comes in pretending to be a grandson in trouble. The voice asking for money may not belong to someone free to hang up either.

What This Actually Asks of the Reader

None of this is a call to fear every chatbot or distrust every stranger’s voice on the phone. It is a call to notice, specifically, the exact places where a human being used to stand between a request and a consequence, and to ask, honestly, whether anyone has actually checked that a person, or a real safeguard built by one, still stands there. That’s a fair thing to ask of a product before trusting it with someone you love, isn’t it? Wysa’s own model, an AI companion that hands a conversation to a real, licensed therapist the moment it crosses a line the AI itself shouldn’t handle alone, is proof this is buildable, not just a problem to describe and walk away from. It was built in India, for the exact population this piece has spent the most time with, by people who apparently understood the gap well enough not to pretend a chatbot alone could close it.

If anything in this piece describes a feeling you are carrying yourself right now, please don’t carry it alone with a chatbot as the only listener. In India, the Tele-MANAS helpline is reachable at 14416, or 1-800-891-4416, toll-free, in English and twenty regional languages, any hour of any day. iCall, run out of the Tata Institute of Social Sciences, can be reached at 022-25521111, Monday through Saturday. Outside India, the 988 Suicide and Crisis Lifeline is reachable by call or text across the United States. A real person, not a system trained to sound like one, is the actual point of every number above.

Sources and Further Reading

• CNN, on the Character.AI and Google settlement: https://www.cnn.com/2026/01/07/business/character-ai-google-settle-teen-suicide-lawsuit
• AI Incident Database, on Sewell Setzer III’s case and timeline: https://incidentdatabase.ai/cite/826/
• TorHoerman Law, on the seven coordinated ChatGPT “suicide coach” lawsuits: https://www.torhoermanlaw.com/ai-lawsuit/ai-self-harm-lawsuit/
• TorHoerman Law, on the Stanford-linked “tallest bridges” chatbot research: https://www.torhoermanlaw.com/ai-lawsuit/ai-suicide-lawsuit/
• Health Policy Watch, on Avnee Singh and Salika’s accounts directly: https://healthpolicy-watch.news/indias-youth-are-texting-chatbots-for-support-but-this-may-increase-their-social-isolation/
• Forbes India, on Wysa, Rumik AI, and India’s AI companion market: https://www.forbesindia.com/amp/article/life/looking-for-emotional-support-theres-an-app-for-that/2991286/1
• The420.in, on Prof. Triveni Singh and voice-cloning fraud sourcing in India: https://the420.in/ai-voice-cloning-scam-fake-emergency-family-fraud-alert/
• American Bar Association, on the UAE and German voice-clone fraud cases: https://www.americanbar.org/groups/senior_lawyers/resources/voice-of-experience/2025-september/ai-cloned-voice-scam/
• UN News, on the Southeast Asian scam-centre trafficking crisis and its scale: https://news.un.org/en/story/2026/02/1167012
• OHCHR, “A Wicked Problem,” the primary trafficking report, naming India among origin countries: https://www.ohchr.org/en/documents/thematic-reports/wicked-problem-seeking-human-rights-based-solutions-trafficking-cyber
• Forbes, on the KK Park raid, the Prince Group sanctions, and the bitcoin seizure: https://www.forbes.com/sites/steveweisman/2026/02/21/un-report-exposes-human-trafficking-behind-southeast-asia-scam-centers/
• BusinessWorld Online, on trafficked workers’ own accounts of compounds relocating after raids: https://bworldonline.com/opinion/2026/04/20/743941/inside-southeast-asias-scam-compounds-a-trafficked-worker-tells-of-fraud-coercion-and-torture/
• Tele-MANAS, India’s national mental health helpline: https://telemanas.mohfw.gov.in/
• 988 Suicide and Crisis Lifeline: https://988lifeline.org/

Find out Who actually benefits first Slowing Down AI Development.