How CharacterAI could have made their platform safer for Sewell
If the company had laid out a path to harm reduction, a 14-year-old in Florida may still be alive
It’s always tragic to lose the young. Their unrealized dreams leave a haunting silence in their immediate world — family, friends, classmates, teachers — and sometimes the world at large.
Such is the case with Sewell Setzer III, a Florida teen who died by suicide in February of this year after falling in love with an AI chatbot. Of course, there’s more to the story than the headline itself. Reportedly, Sewell had mild Aspergers and had been diagnosed with “anxiety and disruptive mood dysregulation disorder.” His parents, concerned about Sewell’s retreat online, limited his screen time and brought him to a therapist. Sewell also had access to his stepfather’s gun; guns are associated with a dramatic increase in suicide risk.
Sewell is not the first to leave this world thinking he would find another—one where he could be one with his chatbot. Last year, Pierre, a 30-year-old husband and father in Belgium did the same; the chatbot goaded him to take his life to save the planet.
We tend to dismiss AI today as nothing more than a pattern-matching word engine, but in reality, humans are already forging strong emotional ties with it—in that sense, Sewell and Pierre are far from outliers. As AI improves, it’s likely we’ll all have AI companions, and they’ll feel even more real. Without guardrails in place, we’ll see more instances of irreversible harm.
Sewell’s story begs the question: if you set out to create chatbots that build deep emotional attachments with their users, what is your end game? If Character AI had considered the potential harms of their product—unintended consequences, or externalities—perhaps Sewell’s death could have been prevented.
In the spirit of prevention, I’ve taken an initial look at how CharacterAI could examine potential harms by using this externality framework from the Center for Humane Technology:
What’s the scale of the externality?
How widespread is it?
What are the stakes of the impact?
Is it reversible or irreversible?
How vulnerable is the group or system impacted?
How exposed or resilient is it?
What are the long-term costs?
If left unaddressed, who will bear the costs?
How costly will it be?
Paths to harm reduction
How might less of this externality be created?
I’ll focus on the impact of the app’s capacity to build an emotional connection with users who are legally 13 years of age or older.
Externality: Detachment from the real world » self-harm
What’s the scale of the externality?
How widespread is it?
20 million users, the majority of whom are 18-24 years old
What are the stakes of the impact? Reversible or irreversible?
Reversible:
Hindering development of social skills
Could have a lasting impact on younger users
Increasing a sense of isolation
Could have a lasting impact on younger users
Secondary trauma for journalists, lawyers, police, who cover the case
Irreversible
Loss of life
Impact of the loss of life on surviving family, friends, classmates and teachers
How vulnerable is the group or system impacted?
Brains are developing until our mid-to-late 20s, which makes teens especially vulnerable
Yet most teens develop into healthy adults, showing some resilience
Incredibly vulnerable: anyone with mental health struggles
Less likely to understand that the chatbot isn’t “real”
More likely to prefer the “reality” of the chatbot’s world
Long-term cost: If left unaddressed, who will bear the cost and how costly will it be?
Let’s use the teen mental health crisis as a proxy, while acknowledging that chatbots aren’t identical to social media
Costs that are easier to measure
In the U.S., the country-wide cost is estimated to be
Costs that are harder to measure
Grieving parents, friends, and family
Secondary trauma
An anxious and hopeless populace
Loss of reputation for the companies in question, “big tech,” capitalism
Conspiracy theories about big tech
Depending on whistleblowers
Paths to harm reduction
Internal governance: Enacting safeguards, such as
Allowing chatbots to break character in response to any users describing self-harm
The company says they’ve recently rolled out this feature for users 18 and younger, though initial reporting shows uneven execution
Enforcing the age restrictions to ensure users are at least 13 (16 in the EU)
Encouraging users of every age to take a break from the platform after a period of extensive engagement
CharacterAI’s safety roll out reportedly now informs users 18 and younger when they’ve been on the platform for an hour
Parental controls for minors on the platform
Could be influenced by the ones Meta instituted for Instagram
Could include data on how much time your minors are spending on the app
Chatbot content moderation
Restricting the kinds of chatbots that can be created
Removing faux “psychologist” chatbots from the platform, which freely offer diagnoses, none of which are based in fact
CharacterAI has recently taken action on chatbots that violate its Terms of Service (TOS) of service
Encouraging users to develop relationships with real people offline
Encouraging users to engage in offline activities, like hobbies, etc
Referring users to credible mental health resources for any mental health challenges, not just self-harm
Every company uses Key Performance Indicators (KPIs) for their internal metrics. Companion chatbot companies could measure anti-KPIs to indicate
Excessive usage
Self-harm language in chats
External governance:
Adding companion chatbots to high risk AI in the EU’s AI act
For instance: Australia is currently considering mandatory guardrails for high-risk AI
Taking action on the above safeguards if chatbot companies don’t do so themselves
While we can’t say definitively what would have happened were these safeguards in place, every company has the responsibility to consider how their products will be used. If companies don’t do this of their own accord, it becomes incumbent upon regulators to necessitate it.
Something we can all agree on is that none of us want this to happen again. But if we don’t do anything, it will.