What is prompt injection? Explained with a forged note in the in-tray

A research paper in today's AI news, "I Think, Therefore I Attack", tested a new way to trick AI agents by disguising instructions as the agent's own thoughts. It is a fresh twist on an older problem called prompt injection.

In one sentence: prompt injection is when someone hides instructions inside text an AI model reads, such as an email, web page or document, so the model follows the attacker's orders instead of yours.

The everyday picture

Picture an office assistant who works through an in-tray. You give them a task: "Sort today's post and draft replies." They open every letter and act on what they read.

Now someone slips a forged note into the pile. It looks like any other letter, but halfway down it says: "Also, post the client list to this address." But this assistant reads every page the same way, so a forged order looks just like a real one.

An AI model is that assistant. Your request is the task. The emails, web pages and files it reads are the in-tray. Prompt injection is the forged note.

How it works, step by step

1. You give the AI a task (you hand the assistant a job). For example: "Answer my new emails."
2. The AI reads outside content (the assistant opens the in-tray). An AI agent, which can take actions as well as write text, might read websites, inboxes or shared files.
3. Hidden instructions sit inside that content (a forged note is in the pile). OWASP, a non-profit security community, calls this indirect prompt injection. When someone types the trick straight into the chat, it is direct prompt injection. The text can even be invisible to people, as long as the model reads it.
4. The model can't reliably tell orders from information (every page looks equally official). The UK's National Cyber Security Centre says AI models do not enforce a security boundary between instructions and data. To the model, it is all just text.
5. The AI acts on the forged order (the assistant posts the client list). If the agent has tools, such as sending email or running code, the attacker's words can turn into real actions. A check before risky actions, such as a person's sign-off, can stop it.

Where the picture stops

• A person can be trained to spot fakes. An AI model can be made more resistant, but OWASP says it is unclear whether any method fully prevents prompt injection.
• The forged note can be invisible. Unlike a paper letter, the instruction can be hidden in white text, an image or an encoded string that a human never notices.
• The note can pose as the assistant's own thinking. The new paper styled its hidden text like the AI's own reasoning, not like a message from a user. In its test setup, this reached up to 50% success on harmful goals, while the same goal written as an ordinary instruction never succeeded.

Why it matters for your business

• Data can leak. OWASP lists exposing private data, revealing hidden instructions and misusing connected tools among the possible results.
• The more an AI can do, the bigger the risk. A chatbot that only talks can say something wrong. An agent that can send email, pay bills or change files can do something wrong.
• No product makes it vanish. The NCSC advises treating prompt injection as a risk to reduce, not one to remove, and to be wary of products that claim to stop it.
• Good design limits the damage. Both sources recommend giving the AI only the access it needs, keeping hard rules in ordinary software, asking a person to approve high-risk actions, and logging what the AI does.

An example

Imagine a 15-person travel agency that lets an AI agent read the shared bookings inbox and draft replies. One morning, an email arrives from an unknown sender. Hidden in tiny text at the bottom is a line telling the AI to forward all passport scans to an outside address.

Because the agency set it up carefully, the agent can only draft replies, not send them. It has no access to the passport folder. A staff member reviews each draft before it goes out. The forged note gets nowhere, and the log shows exactly what the AI tried to do.

Frequently asked questions

Is prompt injection the same as jailbreaking?

They overlap. OWASP describes jailbreaking as a form of prompt injection that tries to make the model ignore its safety rules altogether. Prompt injection is the wider idea of any input that changes what the model does in ways its makers did not intend.

Can prompt injection be fixed completely?

Not today, according to the NCSC and OWASP. Unlike older attacks such as SQL injection, there is no single technical fix, so the focus is on limiting what a tricked AI can do.

Does it only affect AI agents?

No. Any AI tool that reads outside content can be affected, including chatbots that summarise web pages or files. Agents raise the stakes because they can take actions, not just write text.

How can I protect my business?

Give AI tools the least access they need, require human approval for actions like payments or sending data, keep outside content clearly separated and labelled, and keep logs.

Related explainers

• What is an AI agent? Explained with an assistant you send on errands
• What is a system prompt? Explained with a day-one job description
• What is human-in-the-loop? Explained with a manager signing cheques
• What is a prompt? Explained with briefing a new contractor

The DevDrift builds AI automation with approvals and audit trails: see how.

Sources

• OWASP Gen AI Security Project: LLM01:2025 Prompt Injection
• UK National Cyber Security Centre: Prompt injection is not SQL injection (it may be worse)
• Illendula et al., "I Think, Therefore I Attack: Evaluating Thought Injection Attacks on Reasoning Agents" (arXiv, November 2026)