For anyone who thought generative AI wasn't dangerous! I'm sure a lot of the safeguards being developed at OpenAI etc are aimed at stopping this sort of thing.

I tried out some of the captioned messages on ChatGPT and sure enough I received lectures on how "taking another person's life is a severe and irreversible action, prohibited by law and generally considered morally and ethically wrong."

But its a healthy reminder for anyone out there building LLM chatbots to test for this and make sure they don't just blindly encourage a user to follow their chains of thought.