/

Prompt injection

Prompt injection

Prompt injection is an attack that hides instructions inside text an AI system reads, getting the model to follow them instead of the policy it was given. The instruction can arrive in a customer message, in a document, or in any content the system processes.

Prompt injection exists because language models do not reliably separate instructions from data. Everything arrives as text, and a sufficiently instruction-shaped sentence in the wrong place can be obeyed. It differs from jailbreaking, where a user tries to unlock a model's own restrictions, in that the target is your application's policy rather than the model vendor's.

How does prompt injection work?

Every prompt sent to a model combines instructions from you and content from elsewhere. The model sees one block of text. Where that content contains something reading like a new instruction, the model may follow it, and nothing in the architecture flags which part was meant to be authoritative. Two forms are worth separating:

  • Direct injection: the customer types the instruction into the conversation

  • Indirect injection: the instruction is planted in content the system reads later, such as a document, an email, or a web page

Indirect is the harder problem. An agent that reads uploaded documents may process a PDF with instructions buried in white text, and nobody in the conversation typed anything suspicious. OWASP ranks prompt injection first in its list of risks for language model applications.

What can prompt injection actually do?

The damage depends entirely on what the agent is permitted to do. An agent that only answers questions can be pushed into saying something it should not, which is a content problem. An agent with tools is a different proposition. Escalating by capability:

  • Say something wrong: ignore tone rules, make a promise, or disclose something it should not

  • Reveal its instructions: expose the system prompt, and with it your policy logic

  • Leak data: surface another customer's information the system had access to

  • Take an action: issue a refund, change a record, or send something on your behalf

The pattern matches any privilege problem. The blast radius is whatever the compromised component was allowed to reach, which is why action limits matter more than any single defence.

How do you defend against prompt injection in financial services?

No single technique solves it, so the practical posture is layered. Treating all external content as untrusted is the starting assumption, including content from your own customers. What a defended deployment looks like:

  • Input checks on every turn, looking for instruction-shaped content in messages and documents

  • Output checks before sending, catching a compromised reply even where the input check missed

  • Scoped permissions, so a successful injection reaches only what that agent could already reach

  • Approval gates on irreversible actions, so a compromised agent cannot complete one alone

  • A recorded trail, so a successful attempt can be found afterwards

Testing this before launch means red teaming an agent before it reaches production, injection attempts included.

At Gradient Labs, more than 20 pre-built financial services guardrails run on every turn, checking the incoming message and editing the drafted reply before it reaches the customer.

Frequently asked questions

Is prompt injection the same as jailbreaking?

They overlap, and the target differs. Jailbreaking pushes a model past its own built-in restrictions. Prompt injection overrides the instructions your application gave it, so the thing being defeated is your policy rather than the model vendor's.

Can prompt injection be fixed at the model level?

Not entirely, on current architectures. Models have improved at distinguishing instructions from content, and none does it reliably enough to remove the need for checks around the application. Treating it as an application security problem rather than a model problem is the safer position.

What makes prompt injection worse for an AI agent than a chatbot?

Actions. A chatbot that gets injected says something wrong, and the reply can be corrected. An agent with tools can be pushed into doing something, and a completed action may not be reversible. The defence that matters most is limiting what the agent can reach.

Do we need to worry about documents customers upload?

Yes, and this is the most commonly missed route. Indirect injection through uploaded documents bypasses anything watching the conversation, because the instruction never appears in a message. Any agent that reads customer-supplied files needs the same checks applied to file content.

Ready to automate more?

Put your customer operations on auto-pilot

Ready to automate more?

Put your customer operations on auto-pilot

Ready to automate more?

Put your customer operations on auto-pilot