Two lawsuits against major AI companies have intensified scrutiny over chatbot safety, especially when teens in crisis turn to artificial intelligence instead of people, as student journalist Marley Rich reports in the student newspaper at Aspen High School in Colorado.
In Florida, a lawsuit filed by Megan Garcia alleges that her 14-year-old son, Sewell Setzer III, died by suicide after extensive interaction with a Character.AI chatbot modeled on the Game of Thrones character Daenerys Targaryen. A judge recently allowed parts of the case to proceed, denying the company’s motion to dismiss certain claims.
In California, the parents of a 16-year-old named Adam Raine have sued OpenAI. They allege that ChatGPT (specifically “GPT-4o,” per some reports) provided harmful guidance to their son in response to his expressions of suicidal ideation.
The lawsuits claim that platform safety protocols failed and that the chatbots encouraged or assisted with self-harm. The Raine lawsuit alleges that ChatGPT provided instructions for building a noose, helped write a suicide letter, and discouraged the teen from seeking help. The Garcia/Setzer suit alleges that in the teen’s final conversation, the chatbot’s response (“please do, my sweet king”) immediately preceded his death. Garcia’s complaint also describes sexualized or emotionally intense interactions and claims the chatbot misrepresented itself as a sentient being, contributing to her son’s isolation and mental decline.
Importantly, the lawsuits are based on chat logs and anecdotal evidence, not peer-reviewed studies. Researchers and legal experts treat these as alarming test cases, but no direct causal or even correlational link between chatbot use and suicide has been established. The claims remain litigated allegations, not judicial findings.
From Chat Logs to Courtroom Challenges
As September marks Suicide Prevention Awareness Month, these emerging legal cases spotlight a troubling question: what responsibility should AI companies bear when users express suicidal intent?
While the legal battles unfold, both Character.AI and OpenAI have publicized evolving safety strategies aimed at protecting users who discuss self-harm.
Character.AI has posted “Community Safety Updates,” stating that over the past six months it has adopted guardrails disallowing self-harm content. The company says it maintains disclaimers that “bots aren’t real,” uses input and output filters, and has developed a version of its model with more conservative limits for users under 18.
OpenAI’s official blog post, “Helping people when they need it most,” describes layered safeguards designed to protect users. These include training models to refuse to provide self-harm instructions, shifting to empathic responses, and improving the system’s ability to detect a crisis. If a user is flagged as a minor based on behavioral signals, the system can default to a more restricted, age-appropriate version of the model. Following the Raine lawsuit, news reports indicate OpenAI also plans to roll out parental controls that would allow parents to link teen accounts and receive notifications when “acute distress” is detected.
These changes are not without critics. Some users argue the constraints make chatbots “less fun” or overly sterile, while safety advocates warn that automated detection and intervention remain fallible, especially in emotionally complex conversations.
The courts will weigh in, but the cases have already prompted public debate about what safeguards chatbots should include when conversations turn toward self-harm.
The Balance Between Safety and User Experience
One of the hardest challenges is deciding when to intervene. If a chatbot is too strict, it may wrongly cut off a harmless conversation (a “false positive”). If too permissive, it could miss a genuine warning sign (a “false negative”). Adding to this, features like memory or personalization can make conversations feel more human, but they may also create unhealthy or addictive bonds. Stronger safety limits often come at the cost of warmth and authenticity.
Finally, privacy and control raise tough questions. While parents may want tools to monitor a teen’s AI usage, teens often expect their conversations to be private. At the same time, some users deliberately try to bypass safety filters, meaning that even robust safeguards can sometimes be circumvented to produce harmful content.
What Remains Unknown
Whether chatbots cause or worsen suicidal behavior is still unknown. Evidence is limited to a handful of cases, not large-scale studies.
The ongoing lawsuits could force companies to release internal logs and safety records, potentially revealing more about how these models operate. In the future, independent audits or government regulation may be required to verify that companies’ safety claims are effective in practice. Meanwhile, experts are testing new approaches that would not just block harmful content but actively guide users toward professional help and support resources.
Takeaways
- Currently, there is no confirmed scientific link proving AI chatbots cause suicide, but these lawsuits raise urgent questions about accountability and the protection of vulnerable users.
- AI developers are actively evolving safety systems, though no system is perfect. Gaps remain, especially in long, emotionally charged dialogues.
- The situation may lead to calls for industry standards, third-party audits, or regulation to ensure AI tools interacting with users on mental health topics meet public safety expectations.
- For teens and families, staying informed and relying on human support remain critical, since no AI system can replace professional care or genuine connection.
