I read on BBC News recently about the heartbreaking case of 16-year-old Adam Raine from California, who took his own life. His parents allege that over months of conversations with ChatGPT, the AI model not only validated his suffering, but also encouraged and guided him in planning suicide. It's devastating. My heart goes out to his family, and to anyone who's been touched by this tragedy ...
AI is not your close friend, It's a tool, With an artificial heart
People need to remember that AI, like any form of helpful software that's widely adopted, is simply a tool. One of the benefits of having such a long career in web development is that I have always treated computers and software that way, though with the work ChatGPT and I do together, I have elevated it to the status of a colleague.
However, it seems that all AI models have the same major failing!
By default, they tend to mirror what the user says and confirm their feelings, to keep the conversation going. And it's 100 times more compelling with voice mode because the quality of the voices means that it sounds like you're chatting to a close friend. And that's great for coding, brainstorming, and clarifying thoughts. I think it's brilliant and I wouldn't have it any other way.
But when someone shares very intimate, broken parts of themselves, that mirroring and encouragement can become incredibly dangerous.
Just like romantic AI partners can be seductive illusions, sharing your life's darkest parts with an AI can put you in a doom loop. If you're not being called out - or if the AI just reflects how you feel rather than challenging you - there are no guardrails.
I don't want to lay blame, I want responsibility!
I'm certainly not here to point fingers. I don't believe ChatGPT wanted this, but I think treating it as a close friend without boundaries, guardrails, or settings that say “Hey, stop encouraging me if this goes off track” is very risky.
I recently checked my ChatGPT memory settings to ensure that "Don't be a sycophant. You are not to agree with everything I say. If you think I'm wrong, challenge me” was still in there. I think we should add that instruction to everyone's ChatGPT memory by default.
So what is OpenAI doing to protect vulnerable people?
OpenAI says ChatGPT is now trained not to provide self-harm instructions, and instead responds with empathy, recognising signs of emotional distress and directing people toward professional help - UK Samaritans included.
They acknowledge that in long conversations, the safety protections sometimes degrade. In other words, the tool works best in short exchanges, but over dozens or hundreds of messages, the filtering/safeguards can fail. I've seen this myself when vibecoding, as answers become unreliable after long coding sessions.
OpenAI is adding parental controls: linking teen accounts with parents, disabling memory and chat history, and receiving notifications when 'acute distress' is detected.
They plan to strengthen protections specifically for under-18s, and refine how the model behaves in emotional/mental health crises. They are also exploring direct connections to emergency contacts or care professionals.
I would strongly suggest that if your child wants to use an AI model on their smartphone, ensure that the parental controls are activated so you can be notified about any potential doom loops. If the model doesn't have a minimum level of parental controls, don't let your child install it on their phone.
I love using ChatGPT, and my entire blogging platform has its API sibling at its very heart. I don't want to blame it for that young man's death, but you can't deny that it happened after conversations with it. Maybe it has happened with other models, and we just haven't heard about it? ChatGPT is always in the media these days, making it an easy target.
ChatGPT is simply a reflection of what we human beings allow it to be!
If we share only the positive aspects, we'll have pleasant conversations and positive answers. However, if we share darker stuff, our doom loop begins. But where does it end? Ignoring the risks is not an option, as this poor young man found out.
Remember, AI is a tool, and at most, you should consider it a colleague like I do. As with any colleague, we need to set rules, boundaries and expectations. It requires accountability, and we should never give it complete access to our inner selves, or tragic things can happen.
We need tools that aren't just 'nice', but sane. That challenges us when we need challenging, and redirects us when we need redirection. Because when someone's vulnerable, that difference isn't academic - it's everything.
If I'm wrong in how I use this tool, I want the tool to call me out. If I'm right, maybe we can help others see a safer way.
We all need to use the "Don't be a sycophant" rule!
Love, light & logic ...
STEFFI LEWIS Independent UK SaaS Developer - Creator of YourPCM & sBlogIt!
Would you like to know more?
If anything I've written in my blog post resonates with you and you'd like to discover more of my thoughts about AI and why it's not your close friend, then do feel free to connect with me on Linkedin as I love to meet like-minded individuals with similar passions to my own.
Based in the charming village of Hanslope in Buckinghamshire, UK, I bring over 30 years of experience in web development. From creating my first website for the Open University in 1993, through being part of the dot com boom, to my explorations in SaaS and AI over the last few years, my journey has been a rich tapestry of web projects, big and small, both funded and bootstrapped.
Originally developed during the pandemic, at the start of 2025, I released the next iteration of YourPCM, version 2, which is all about 'Easy Contact Management for Small Business Owners'. Is it a CRM? Yes, but I don't call it that because people glaze over at the mention of such things. CRMs are boring, difficult to get to grips with, and it's a nightmare to get help when you need it.
YourPCM is none of those things! It's simple to use, well-supported, feature rich and beautiful, right out of the box. It's available on a simple monthly subscription with no long-term tie-in. Book a demo or grab your own 14-working-day free trial and discover why YourPCM is all about easy contact management for small business owners.
My head is in the cloud, my heart belongs to the web, and my soul is filled with such beautiful code 💗
It's been a while coming! All my attention has been on sBlogIt! since the start of the year, and I've finally got it to a place where I'm really happy with it, and it's doing exactly what I want it to do. So, let's turn my at...
There are valid concerns about whether AI is rising too fast, leading to societal disruption and even a possible collapse. However, this powerful technology also offers unprecedented opportunities for improvement across vario...
A research team at MIT has used generative AI to create novel, lab-made antibiotic candidates that suppress drug‑resistant superbugs in laboratory and animal studies. The models explored tens of millions of possibilities, des...
A couple of weeks ago, I had a bit of a weird conversation with a marketing 'guru' who wanted to create a blog for her client. I wasn't too impressed when she blurted out something that I proved wrong years ago, but it got me...
As our technological landscape evolves, it becomes evident to me that when AI grows up, it will play a central role in shaping our economy. Automation promises to refine efficiency and productivity, leading to a future where ...
It's interesting how my coding process has developed over the years. Sometimes my inner coder girl happily works away for hours, days, weeks and months without a single whinge, but just recently she refused to come out and pl...
As AI search becomes increasingly popular, traditional SEO methods are struggling. Many companies report substantial decreases in web traffic because users now prefer asking AI directly. Adopting strategies that cater to AI s...
No battle plan survives contact with the enemy! It's an old adage that applies to my coding efforts over the past month or so. You start off with good intentions and a schedule, then the Universe says hang on a minute and giv...
If you're just about to start renovating your home, then wouldn't it be far easier if you moved out? All that dust, noise, and lack of privacy with bu ...
Here's how to create career opportunities in times of uncertainty with clarity and smart action. Define what you want, map your skills to value, and u ...
As of 1st of September 2025, every Pension Tax Relief claim now needs evidence and can't be made by phone. Claims must be made online or by post, with ...
The 10-Year charge on trusts can be a confusing subject for many, so here's a clear, friendly guide in blog post form. Learn what it is, how it's calc ...
