Amazon's Rufus AI shopping assistant can be easily jailbroken and tricked into answering other questions — specific prompts break the chatbot's guidelines and reach underlying AI engine

Two years ago, Amazon announced Rufus, its AI-powered shopping assistant built right into the Amazon app and website. The goal was to let customers not just search for items, but also allow them to talk with an expert who can recommend products and deals naturally. Under the hood, Rufus uses multiple LLMs, and some people have realized it's quite easy to trick the chatbot into forgetting its purpose.

The tweet above shows the author prompting Rufus for a complex modeling question to figure out how to map sensory data into digital data for robotics. It's entirely detached from any shopping query, which is exactly why it's so funny to see Rufus answer it so swiftly. The formula provided is correct, too. There's a chance that terms like "tactile sensors" were flagged as product inquiries by Rufus.

When we tried it ourselves, we were able to get it to talk about architectural differences between x86 and ARM on the first try. Ironically, after asking whether it thinks the AI bubble will burst this year, Rufus started answering but cut off abruptly. Our other efforts were in vain, and it almost felt like the AI was learning to keep the guardrails up more securely in real time as we poked it more and more.

There is conflicting information online as to what exactly Rufus is using underneath — it could be Amazon's in-house frontier model 'Nova,' while the majority says it's Anthropic's Claude, but some argue that it's not smart enough to be running Claude. One Reddit post points towards Rufus being based on Claude Haiku and not Claude Sonnet, saying it's extremely hard to break and not worth the effort to try to "jailbreak."

Regardless of whatever model it's using or switching between, the ease with which its guardrails erode is both fascinating and funny. You could certainly try to continue your work on Rufus if the free tier of Claude has rate-limited you for the day. It also goes to show that integrating AI into every aspect of the internet is perhaps not the best idea because it's just another point in the chain that can potentially break. And not everyone will try harmless prompts to pass the time.

Follow Tom's Hardware on Google News, or add us as a preferred source, to get our latest news, analysis, & reviews in your feeds.