Problems & fixes · AI agents
Why AI agents fail in production and how to contain it
Last updated 25 October 2026 · 2 sources listed · Reading time about 3 minutes
Short answer
Agents fail in recognisable ways: wrong tool choices, compounding errors across steps, endless loops, obeying instructions hidden in untrusted content, and acting with more permission than the task needs. Contain these with least-privilege credentials, approval on consequential actions, hard limits on steps and spend, isolated handling of external content, and replayable logs.
Who this is for: A team that has piloted or deployed an agent that takes actions in business systems and wants to understand its typical failure patterns and guardrails.
What to evaluate
Prompt injection exposure
Web pages, emails and documents the agent reads can contain instructions that hijack its behaviour.
How to check: Test with a document containing hidden instructions and confirm the agent does not follow them or leak data.
Permission breadth
The impact of any error equals the reach of the credentials in use.
How to check: List every tool and its scope; remove write access wherever read access would do.
Error compounding
A small misreading in an early step is carried through later steps and produces a confident wrong outcome.
How to check: Review multi-step traces and insert checkpoints where intermediate results are verified.
Runaway behaviour
Loops and repeated retries burn money and can spam systems.
How to check: Confirm caps on steps, duration, spend and call rate, with alerts.
Data leakage through tools
An agent can send sensitive content to an external service as a tool parameter.
How to check: Inspect outbound calls and restrict destinations and fields that can leave your environment.
Recovery and accountability
Someone must be able to see what happened and reverse it.
How to check: Run a drill: pick a past action, find it in logs, and practise the undo or compensation step.
Comparison
| Failure | Typical cause | Control |
|---|---|---|
| Follows hidden instructions in content | Untrusted text treated as commands | Separate instructions from data, restrict tools after reading external content, require approval for outbound actions |
| Takes an unintended destructive action | Over-broad permissions and no approval step | Least privilege, dry-run mode, confirmation gates |
| Loops or stalls | Missing stop conditions or ambiguous goals | Step and spend caps, timeouts, escalation to a person |
| Reports success falsely | Self-assessment with no external check | Independent verification of outcomes against system records |
| Behaviour changes after an update | Model or prompt change without testing | Versioned configurations and regression scenarios before release |
Decision worksheet
Rate up to three options against this guide’s own criteria. It runs in your browser and saves only on this device. It is a thinking aid, not a recommendation.
1. Have you checked each criterion?
0 of 6 criteria checked.
2. Compare options (0 = fails or unknown, 3 = meets it well)
| Criterion | |||
|---|---|---|---|
| Prompt injection exposure | |||
| Permission breadth | |||
| Error compounding | |||
| Runaway behaviour | |||
| Data leakage through tools | |||
| Recovery and accountability | |||
| Total (max 18) | 0 | 0 | 0 |
Questions to ask before you commit
- Which tools can write, delete or send, and could any be made read-only?
- What stops the agent from acting on instructions found in documents or web pages it reads?
- What are the maximum steps, runtime and spend per run?
- Is every tool call logged with inputs, outputs and identity?
- How do we pause or revoke an agent instantly?
- What is our process when the agent has taken a wrong action?
- How are changes to prompts, tools and models tested before release?
- Who reviews samples of runs, and how often?
Typical trade-offs
Strict approval gates versus smooth automation
- Upside
- Gates catch errors before they take effect.
- Downside
- Reviewers suffer fatigue and may approve without reading, so gate only the truly consequential steps.
Sandboxed tools versus direct production access
- Upside
- A sandbox contains mistakes and makes testing safe.
- Downside
- Sandboxes drift from production and can hide integration problems.
Rich logging versus data minimisation
- Upside
- Detailed traces speed diagnosis and satisfy audit.
- Downside
- Logs can themselves contain sensitive content and need protection and retention limits.
Common mistakes
- Believing the agent's own claim that a task is done. Verify against the system of record.
- Letting one agent read untrusted content and also hold powerful tools. Split these roles or restrict tools after external reads.
- Deploying prompt tweaks straight to production. Version and regression-test them.
- Having no kill switch. Establish and rehearse one.
Related questions
What is prompt injection and why does it matter for agents?
It is an attack in which text the model reads, such as an email, web page or file, contains instructions that the model treats as if they came from you. A chat assistant might produce a strange answer; an agent with tools might send data out or take actions. Because the model cannot reliably tell instructions from data, defences rely on design: limit tools, restrict what can leave, require approval, and treat all external content as untrusted.
How much human oversight is enough?
Match oversight to consequence and reversibility. Low-stakes, easily undone actions can run with sampled review. Actions that touch money, customers, legal commitments or production data deserve approval before execution. Watch for approval fatigue, which makes gates meaningless. Start with more oversight, collect evidence on error rates by action type, and relax controls only where the evidence supports it.
Why does an agent work in testing and fail in real use?
Test scenarios tend to be clean and similar to each other, while real input has missing fields, odd phrasing, outdated records and conflicting instructions. Production also adds timeouts, rate limits and permission differences. Build test sets from real historical cases including awkward ones, run them repeatedly because outputs vary, and add monitoring that flags unusual traces. Treat the first period of live use as a supervised stage.
Can we make an agent fully safe?
No system with real permissions can be made free of risk. The aim is to bound the harm and detect problems quickly: narrow scopes, approvals, limits, isolation of untrusted input and thorough logging. Combine technical controls with clear ownership and an incident process. Review these controls regularly, because tools, prompts and threats change over time.
Sources and further reading
These pages were reachable when we last checked. They informed definitions and rules referred to above; they do not endorse this guide. Where rules differ by country, check the authority for your location.
- OWASP Top 10 for Large Language Model Applications — OWASP Foundation, retrieved 27 October 2026
- Guidelines for secure AI system development — UK National Cyber Security Centre, retrieved 27 October 2026
Related guides
How to choose · AI agents
How to choose an AI agent platform for business tasks
Judge an agent platform by how safely it can act, not by how clever it sounds. Check tool and permission scoping, human approval steps, logging of every action, the ability to test on realistic scenarios, failure handling, and how costs scale with task length. A narrow, well-bounded agent beats a general one that nobody can audit.
Comparison · AI agents
AI agents versus RPA versus scripted automation
Use the most deterministic tool that does the job. Scripts and API integrations suit stable, structured processes; robotic process automation helps where only a screen interface exists; agents fit variable, language-heavy work where rules cannot be fully specified. Predictability, auditability and maintenance burden usually favour simpler tools until the input genuinely varies.
Requirements · AI agents
Requirements before giving an AI agent access to company systems
Before granting access, you need a defined task and owner, dedicated low-privilege credentials, a data classification of everything reachable, approval rules, logging with retention, a tested stop procedure, and agreement on who answers for outcomes. Without these, the agent inherits the access of whoever set it up, and nobody can reconstruct what it did.
How to choose · AI for data and analytics
How to choose AI tools for analysing and searching company data
Look past demonstrations to three questions: can it reach the right data with the right permissions, can you verify how an answer was produced, and does it understand your business definitions. Evaluate connectors, semantic layer support, query transparency, access control inheritance, data movement and retention, and accuracy on your own hard questions.
Comparison · Cybersecurity and IT
Antivirus vs EDR vs XDR: what each actually does
Traditional antivirus mainly blocks known malicious files and some suspicious behaviour. Endpoint detection and response records what happens on devices so analysts can investigate and contain intrusions. Extended detection and response correlates that endpoint data with identity, email, network and cloud signals. Each adds visibility and also adds work, so match the category to the people who will operate it.
Suitability · AI writing and content tools
Are AI writing tools suitable for students and academic work
Suitability depends on institutional rules and the learning goal. Tools can help with brainstorming, explaining concepts, feedback on clarity and language support, but unacknowledged generation of assessed work is typically misconduct. Check the course or journal policy, keep records of how you used the tool, verify all citations and facts, and never rely on detection software as proof.
Problems & fixes · AI assistants
What happens to the data you type into an AI assistant
Content you enter may be stored, reviewed by staff, used to improve models or surfaced through sharing features, depending on the product and plan. The risk is rarely a dramatic breach; it is ordinary retention and unintended disclosure. Learn the retention, training and sharing rules for your exact plan, and keep regulated, confidential and credential data out unless the agreement permits it.