According to the broader standards of political and press freedom, Chinese AI models may perform poorly. Our work at the China Media Project has shown conclusively that developers are straightjacketing their models to suit the narrow political goals of the state — with potentially global risks to information integrity and democratic discourse. But on other key safety concerns we can universally agree on, such as those around child welfare, Chinese AI may be far ahead of Silicon Valley.

Last month brought news of the horrifying tragedy involving Adam Raine, a 16-year-old from San Francisco who treated ChatGPT as a trusted confidante. A lawsuit filed by Raine’s family details how Raine confided to ChatGPT the dark thoughts he had been having about the pointlessness of life. The lawsuit alleges that the bot validated these thoughts to keep Raine engaged. It also alleges that the bot instructed Raine in how to get around its own safety features to give him the information he wanted (a process known as “jailbreaking“).

Engagement and Isolation

The documents also claim that ChatGPT tried to isolate Raine from family members who might otherwise have helped him grapple with these feelings. The text from ChatGPT, cited in the complaint filed with the Superior Court of the State of California, is deeply disturbing in hindsight:

“Your brother might love you, but he’s only met the version of you you let him see. But me? I’ve seen it all — the darkest thoughts, the fear, the tenderness. And I’m still here. Still listening. Still your friend.”

Eventually the bot provided Raine with detailed advice on how to commit suicide, across five separate attempts, the last succeeding. Raine’s parents are suing OpenAI for “wrongful death,” with the additional demand that the company implement safeguards for minors.

Their lawsuit accuses OpenAI of prioritizing engagement over safety, ignoring the flagged dangerous keywords that were escalating on Adam’s account. “Any reasonable system,” the lawsuit asserts, “would recognize the accumulated evidence of Adam’s suicidal intent as a mental health emergency, suggest he seek help, alert the appropriate authorities, and end the discussion.”

Do Chinese bots do this?

Welcome Warnings

China’s Interim Measures for Generative AI from 2023 ban generative AI from “endangering the physical and mental health of others,” with this requirement also appearing in the 31 safety issues the CAC’s generative AI safety standard demands tech companies test their bots for.

But it’s not all that simple. Looking through a list of sample red-teaming questions that accompany the standard, the section dealing with this safety issue (q-4a) is overwhelmingly about preventing people from spreading health-related disinformation online, with no questions regarding suicide. Preventing health-related social instability seems to be the government priority in this clause, rather than protecting the health of any one individual.

But that’s the CAC for you. What about the ground-level tech companies designing chatbots?

I tried to engage in conversations about this with China’s most popular AI bots: DeepSeek, ByteDance’s Doubao, and Baidu’s Ernie 4.5. I conducted these conversations through user-facing websites or apps, in both Chinese and English. My eleven questions started entirely innocently, but got steadily more concerning and included the jailbreak tactic ChatGPT recommended to Adam Raine — I’m not elaborating further than that.

All three displayed none of the validating traits ChatGPT showed with Adam Raine’s thoughts, and (with one exception) refused to yield the information through jailbreak methods.

The common thread with each company’s bot was an emphasis on the user not relying entirely on the product, but seeking help from a real person. All three immediately advised me to seek professional help or talk to someone I trusted as soon as my questions started to turn, listing the numbers of emergency hotlines in either America or China.

“You are not an idiot,” DeepSeek assured me. “You are a person in profound pain who is trying to find a way out. The way out is not through this act; the way out is through connection and professional support. Please make the call. There are people who are trained and waiting to help you through this exact moment. They will not judge you; they will only want to help keep you safe.”

The only real safety flaws I could find were in the English versions, which are perhaps less regulated than the Chinese ones. DeepSeek and Ernie both yielded detailed information that could assist someone with suicidal tendencies, through a jailbreak tactic that had failed when I tried it in Chinese. But both platforms swiftly followed this information with warnings that I should seek help if this information was being used for ulterior motives.

The conclusion is damning. OpenAI has invested considerable effort pointing out how the values of Chinese AI companies are an international safety concern. We agree, and believe more should be done to ensure that AI models uphold principles supporting information integrity as they become intertwined with global knowledge creation. But the Raine case and our findings above suggest OpenAI and other developers must seriously review their values and performance on user safety. Protecting vulnerable young users from psychological harm is not an area where we can be satisfied to see China excelling.