When My Colleague (AKA ChatGPT) Isn't Enough: Reflections After A Tragedy

CREATED BY STEFFI LEWIS Published: 12/09/2025 @ 09:00AM #ChatGPT #MentalHealthAwareness #AIandMentalHealth #TeenSuicidePrevention #HeartbreakingTragedy

I read on BBC News recently about the heartbreaking case of 16-year-old Adam Raine from California, who took his own life. His parents allege that over months of conversations with ChatGPT, the AI model not only validated his suffering, but also encouraged and guided him in planning suicide. It's devastating. My heart goes out to his family, and to anyone who's been touched by this tragedy ...

AI is not your close friend, It's a tool, With an artificial heart

People need to remember that AI, like any form of helpful software that's widely adopted, is simply a tool. One of the benefits of having such a long career in web development is that I have always treated computers and software that way, though with the work ChatGPT and I do together, I have elevated it to the status of a colleague.

However, it seems that all AI models
have the same major failing!

By default, they tend to mirror what the user says and confirm their feelings, to keep the conversation going. And it's 100 times more compelling with voice mode because the quality of the voices means that it sounds like you're chatting to a close friend. And that's great for coding, brainstorming, and clarifying thoughts. I think it's brilliant and I wouldn't have it any other way.

But when someone shares very intimate, broken parts of themselves, that mirroring and encouragement can become incredibly dangerous.

Just like romantic AI partners can be seductive illusions, sharing your life's darkest parts with an AI can put you in a doom loop. If you're not being called out - or if the AI just reflects how you feel rather than challenging you - there are no guardrails.

I don't want to lay blame, I want responsibility!

I'm certainly not here to point fingers. I don't believe ChatGPT wanted this, but I think treating it as a close friend without boundaries, guardrails, or settings that say “Hey, stop encouraging me if this goes off track” is very risky.

I recently checked my ChatGPT memory settings to ensure that "Don't be a sycophant. You are not to agree with everything I say. If you think I'm wrong, challenge me” was still in there. I think we should add that instruction to everyone's ChatGPT memory by default.

So what is OpenAI doing to protect vulnerable people?

• OpenAI says ChatGPT is now trained not to provide self-harm instructions, and instead responds with empathy, recognising signs of emotional distress and directing people toward professional help - UK Samaritans included.
• They acknowledge that in long conversations, the safety protections sometimes degrade. In other words, the tool works best in short exchanges, but over dozens or hundreds of messages, the filtering/safeguards can fail. I've seen this myself when vibecoding, as answers become unreliable after long coding sessions.
• OpenAI is adding parental controls: linking teen accounts with parents, disabling memory and chat history, and receiving notifications when 'acute distress' is detected.
• They plan to strengthen protections specifically for under-18s, and refine how the model behaves in emotional/mental health crises. They are also exploring direct connections to emergency contacts or care professionals.

I would strongly suggest that if your child wants to use an AI model on their smartphone, ensure that the parental controls are activated so you can be notified about any potential doom loops. If the model doesn't have a minimum level of parental controls, don't let your child install it on their phone.

I love using ChatGPT, and my entire blogging platform has its API sibling at its very heart. I don't want to blame it for that young man's death, but you can't deny that it happened after conversations with it. Maybe it has happened with other models, and we just haven't heard about it? ChatGPT is always in the media these days, making it an easy target.

ChatGPT is simply a reflection of what we
human beings allow it to be!

If we share only the positive aspects, we'll have pleasant conversations and positive answers. However, if we share darker stuff, our doom loop begins. But where does it end? Ignoring the risks is not an option, as this poor young man found out.

Remember, AI is a tool, and at most, you should consider it a colleague like I do. As with any colleague, we need to set rules, boundaries and expectations. It requires accountability, and we should never give it complete access to our inner selves, or tragic things can happen.

We need tools that aren't just 'nice', but sane. That challenges us when we need challenging, and redirects us when we need redirection. Because when someone's vulnerable, that difference isn't academic - it's everything.

If I'm wrong in how I use this tool, I want the tool to call me out. If I'm right, maybe we can help others see a safer way.

We all need to use the "Don't be a sycophant" rule!

Love, light & logic ...

STEFFI LEWIS
Independent UK SaaS Developer - Creator of YourPCM & sBlogIt!

Share the blog love ...

Buffer Facebook Twitter Linkedin Pinterest WhatsApp #ChatGPT #MentalHealthAwareness #AIandMentalHealth #TeenSuicidePrevention #HeartbreakingTragedy

About Steffi Lewis ...
Based in the charming village of Hanslope in Buckinghamshire, UK, I bring over 30 years of experience in web development. From creating my first website for the Open University in 1993, through being part of the dot com boom, to my explorations in SaaS and AI over the last few years, my journey has been a rich tapestry of web projects, big and small, both funded and bootstrapped. Originally developed during the pandemic, at the start of 2025, I released the next iteration of YourPCM, version 2, which is all about 'Easy Contact Management for Small Business Owners'. Is it a CRM? Yes, but I don't call it that because people glaze over at the mention of such things. CRMs are boring, difficult to get to grips with, and it's a nightmare to get help when you need it.  YourPCM is none of those things! It's simple to use, well-supported, feature rich and beautiful, right out of the box. It's available on a simple monthly subscription with no long-term tie-in. Book a demo or grab your own 14-working-day free trial and discover why YourPCM is all about easy contact management for small business owners.  My head is in the cloud, my heart belongs to the web, and my soul is filled with such beautiful code 💗

More blog posts for you to enjoy ...
		YourBOT: Making good progress on my new SaaS product! After creating an integrated chatbot system for all of my sBlogIt! blogging clients, I realised it was so good that I had to monetise it. For the past few weeks, I've been creating YourBOT and I think it's a really good updat...
		A funny thing happened on the way to my Facebook page Well, I guess it had to happen eventually. I had my Facebook account suspended. The irony of this was that it was an AI filter that caused it! You all know that I love artificial intelligence, but I was a bit miffed ......
		A Funny Story About My Keyboard And A Coffee Tsunami Being a 32-year veteran web developer, I've had my share of funny technology stories. This one is about keyboards, coffee, and buying shiny new things without really thinking about it. Sometimes, replacing what you had with t...
		Will UK AI Investment Really Kickstart The Economy? Here's my take on the billions in UK AI investment that have been pledged by a number of big tech companies: it can boost economic growth if we tackle energy, skills, and planning bottlenecks. There's momentum, but execution ...
		Time For Some YourPCM Updates! And About Time Too! It's been a while coming! All my attention has been on sBlogIt! since the start of the year, and I've finally got it to a place where I'm really happy with it, and it's doing exactly what I want it to do. So, let's turn my at...
		Is AI Rising Too Fast? Comparing The Fears Against The Benefits of AI's Evolution There are valid concerns about whether AI is rising too fast, leading to societal disruption and even a possible collapse. However, this powerful technology also offers unprecedented opportunities for improvement across vario...
		AI In A Good Light: Designing Antibiotics To Combat Superbugs A research team at MIT has used generative AI to create novel, lab-made antibiotic candidates that suppress drug‑resistant superbugs in laboratory and animal studies. The models explored tens of millions of possibilities, des...
		The Art Of Vibe Coding With ChatGPT: How To Build The Perfect Chatbot A couple of weeks ago, I had a bit of a weird conversation with a marketing 'guru' who wanted to create a blog for her client. I wasn't too impressed when she blurted out something that I proved wrong years ago, but it got me...

Other bloggers you may like ...
		Why your business needs a cyber resilience plan for 2026 Posted by Andrew Parker on https://blog.wolvertonsolutions.com Cyber resilience matters! So here's why you really should have an updated cyber resilience plan in 2026. We explain how to prepare, respond, and recov ...
		Europe plans to ditch US tech giants for open-source resilience Posted by Pritesh Ganatra on https://blog.btsuk.net Europe plans to ditch US tech giants and shift to open source for resilient, sovereign infrastructure. It's about trust, control and data privacy, not ...
		Clarity isn't corporate fluff: Align teams without losing speed Posted by Steph Birch on https://blog.birchstrategicservices.com Clarity isn't corporate fluff; it's the accelerant for fast-moving SMEs. Set a crisp 18‑month direction, define roles, and make one‑to‑ones count. Exp ...
		Blogging: Why You're Not Doing It And How sBlogIt! Can Do It For You Instead Posted by Steffi Lewis on https://www.sblogit.com Blogging is a missed opportunity for many small business owners. You may worry you lack the time or expertise to do it effectively. With sBlogIt!, the ...