I read on BBC News recently about the heartbreaking case of 16-year-old Adam Raine from California, who took his own life. His parents allege that over months of conversations with ChatGPT, the AI model not only validated his suffering, but also encouraged and guided him in planning suicide. It's devastating. My heart goes out to his family, and to anyone who's been touched by this tragedy ...
AI is not your close friend, It's a tool, With an artificial heart
People need to remember that AI, like any form of helpful software that's widely adopted, is simply a tool. One of the benefits of having such a long career in web development is that I have always treated computers and software that way, though with the work ChatGPT and I do together, I have elevated it to the status of a colleague.
However, it seems that all AI models have the same major failing!
By default, they tend to mirror what the user says and confirm their feelings, to keep the conversation going. And it's 100 times more compelling with voice mode because the quality of the voices means that it sounds like you're chatting to a close friend. And that's great for coding, brainstorming, and clarifying thoughts. I think it's brilliant and I wouldn't have it any other way.
But when someone shares very intimate, broken parts of themselves, that mirroring and encouragement can become incredibly dangerous.
Just like romantic AI partners can be seductive illusions, sharing your life's darkest parts with an AI can put you in a doom loop. If you're not being called out - or if the AI just reflects how you feel rather than challenging you - there are no guardrails.
I don't want to lay blame, I want responsibility!
I'm certainly not here to point fingers. I don't believe ChatGPT wanted this, but I think treating it as a close friend without boundaries, guardrails, or settings that say “Hey, stop encouraging me if this goes off track” is very risky.
I recently checked my ChatGPT memory settings to ensure that "Don't be a sycophant. You are not to agree with everything I say. If you think I'm wrong, challenge me” was still in there. I think we should add that instruction to everyone's ChatGPT memory by default.
So what is OpenAI doing to protect vulnerable people?
OpenAI says ChatGPT is now trained not to provide self-harm instructions, and instead responds with empathy, recognising signs of emotional distress and directing people toward professional help - UK Samaritans included.
They acknowledge that in long conversations, the safety protections sometimes degrade. In other words, the tool works best in short exchanges, but over dozens or hundreds of messages, the filtering/safeguards can fail. I've seen this myself when vibecoding, as answers become unreliable after long coding sessions.
OpenAI is adding parental controls: linking teen accounts with parents, disabling memory and chat history, and receiving notifications when 'acute distress' is detected.
They plan to strengthen protections specifically for under-18s, and refine how the model behaves in emotional/mental health crises. They are also exploring direct connections to emergency contacts or care professionals.
I would strongly suggest that if your child wants to use an AI model on their smartphone, ensure that the parental controls are activated so you can be notified about any potential doom loops. If the model doesn't have a minimum level of parental controls, don't let your child install it on their phone.
I love using ChatGPT, and my entire blogging platform has its API sibling at its very heart. I don't want to blame it for that young man's death, but you can't deny that it happened after conversations with it. Maybe it has happened with other models, and we just haven't heard about it? ChatGPT is always in the media these days, making it an easy target.
ChatGPT is simply a reflection of what we human beings allow it to be!
If we share only the positive aspects, we'll have pleasant conversations and positive answers. However, if we share darker stuff, our doom loop begins. But where does it end? Ignoring the risks is not an option, as this poor young man found out.
Remember, AI is a tool, and at most, you should consider it a colleague like I do. As with any colleague, we need to set rules, boundaries and expectations. It requires accountability, and we should never give it complete access to our inner selves, or tragic things can happen.
We need tools that aren't just 'nice', but sane. That challenges us when we need challenging, and redirects us when we need redirection. Because when someone's vulnerable, that difference isn't academic - it's everything.
If I'm wrong in how I use this tool, I want the tool to call me out. If I'm right, maybe we can help others see a safer way.
We all need to use the "Don't be a sycophant" rule!
Love, light & logic ...
STEFFI LEWIS A.K.A The SaaSy Coder - Creator of YourBOT, YourPCM & sBlogIt!
Would you like to know more?
If anything I've written in my blog post resonates with you and you'd like to discover more of my thoughts about AI and why it's not your close friend, then do feel free to connect with me on Linkedin as I love to meet like-minded individuals with similar passions to my own.
Based in the charming village of Hanslope in Buckinghamshire, UK, I bring over 30 years of experience in web development. From creating my first website for the Open University in 1993, through being part of the dot com boom, to my explorations in SaaS and AI over the last few years, my journey has been a rich tapestry of web projects, big and small, both funded and bootstrapped.
Originally developed during the pandemic, at the start of 2025, I released the next iteration of YourPCM, version 2, which is all about 'Easy Contact Management for Small Business Owners'. Is it a CRM? Yes, but I don't call it that because people glaze over at the mention of such things. CRMs are boring, difficult to get to grips with, and it's a nightmare to get help when you need it.
YourPCM is none of those things! It's simple to use, well-supported, feature rich and beautiful, right out of the box. It's available on a simple monthly subscription with no long-term tie-in. Book a demo or grab your own 14-working-day free trial and discover why YourPCM is all about easy contact management for small business owners.
My head is in the cloud, my heart belongs to the web, and my soul is filled with such beautiful code 💗
It has been a while since I wrote a full blog post here. The last one was about a great TV series I'd been watching, and before that, a Christmas one about YourBOT. So what have I been doing and what am I up to now?...
I've just finished watching five seasons of 'Person of Interest' on Amazon Prime, and like all the best sci-fi, it didn't really feel like fiction by the end. It felt like a profound warning wrapped in a really solid story .....
It's been quite some time since I wrote a review of the year. I'm not sure why I got out of the habit of doing them, but here we are again. This is my summary of 2025, which is quite short because most of it was a blur of eat...
It's finished! It's published! It's out there in the wild. People are joining for free, creating their bots and installing them on their websites, then subscribing. To say I'm thrilled is an understatement ......
I've been working hard for the past few months on YourBOT. I have blogged about it from time to time, but now the actual product is finished, it's being tested in the wild, and I'm onto creating the website and sign-up proces...
I was pretty emotional the other night whilst talking with ChatGPT; in a good way, of course. We've been working together on YourBOT for the past month and have finally reached the end of core coding ......
After creating an integrated chatbot system for all of my sBlogIt! blogging clients, I realised it was so good that I had to monetise it. For the past few weeks, I've been creating YourBOT and I think it's a really good updat...
Well, I guess it had to happen eventually. I had my Facebook account suspended. The irony of this was that it was an AI filter that caused it! You all know that I love artificial intelligence, but I was a bit miffed ......
Having a strong online presence is crucial for the success of any small business. One of the most effective ways to establish your brand is through bl ...
I had a lovely email when I first published this blog post from one of my clients, a graphic designer of many years, but new to running his own busin ...
Compared with a hotel room, our serviced accommodation offers enhanced comfort, privacy, and a lot of guest satisfaction. From the spaciousness of a w ...
Do your career research like a mini-investigation before you leap. Use AI, web search, LinkedIn, and real conversations to validate day-to-day work, p ...
All content on this blog, including but not limited to text, images, videos and audio, is protected by copyright. No part of this blog may be reproduced, copied, distributed, or otherwise used without the prior written consent of the author. Unauthorised use constitutes a breach of intellectual property rights.
Please note that many elements of this blog have been created using Artificial Intelligence (AI). As such, content may not always reflect verified facts or professional advice. The information provided is for general interest only and should not be relied upon as a sole source for making decisions, financial or otherwise. Readers are strongly advised to seek independent advice from qualified professionals appropriate to their country and situation.
The author of this blog, YourPCM Limited, and its directors, employees, and authorised agents accept no liability for any loss, harm, or consequence arising from the use or interpretation of content found on this site.
The sblogit.com platform is provided on an “as is” basis. By continuing to view or interact with this blog, you acknowledge and accept these terms. If you do not agree with any part of this notice, please cease using this site immediately.
YourPCM Limited is a company registered in the UK and operates exclusively under the jurisdiction of the laws of England and Wales.
