tldr: handling incidents means handling attacker controlled data. If an LLM does the handling, assume prompt injection is successful and design the enclosure so it does not matter.
Investigating an incident means reading data an attacker wrote. Sender display names, mail subjects, attachment filenames, device names, URLs. That has always been true, it is just that a human analyst reading a weird display name does not act on it.
IVON is our investigation agent. It picks up a Defender or Sentinel incident, runs the queries an L2 analyst would run, and writes a verdict with the evidence under it, without a human in the loop. So an LLM does the reading. A display name is a couple of hundred characters of free text and nothing stops it containing a paragraph aimed at whatever is parsing it. An attacker can assume somewhere in the pipeline there is an LLM reading the input, so it is possible to manipulate the input via free fields.
Trying to shield prompt injection is important. But given the current prevention methods it is safe to assume that an attacker with enough time will bypass the filter. This means we need to assume the injection works and ask what an attacker gets from there. Initial development just gave the container whatever it needed. Before taking this to prod we needed to review the current approach.
What the container held
A headless agent container ran with:
- An LLM API key. A Foundry key, long lived and shared across every tenant we run.
- An API token that was tenant scoped but not session scoped, and not lifetime capped.
- Open network egress. Internet, the host, and cloud IMDS at 169.254.169.254 handing out the node’s managed identity to anything that asks.
If someone is able to manipulate IVON into executing arbitrary commands our Foundry key could get stolen which would probably cost us. Also the API token to our backend could be used to manipulate the data for that tenant.
Quick vocabulary so the rest makes sense: SDUH is the backend everything talks to, and an L2 run is one headless container investigating a single incident end to end, named after the analyst tier it does the work of.
Let’s just filter the input
The usual answer here is an injection classifier. Prompt Shields, Prompt Guard, pick a guardrail library. They do catch things. But it feels a lot like filtering cross site scripting, new bypasses get published. So we use them as a logging signal and not as a boundary. Any L2 run that has prompt injection raised is immediately escalated to an analyst. An attacker thinking about manipulating the monitoring party is someone we want to take a good look at, and the client needs to hear that somebody has a clear awareness of their monitoring infra.
We asked a different question instead: when the injection lands, what does the attacker get? That one has answers you can build and then test. Take the credentials out of the container, take the network away, and the injection lands somewhere that cannot do anything with it.
The frame we ended up with is that the agent is just another user. You would not hand a user a god token and a route to everything on your network and then rely on them not being socially engineered.
We took the following steps.
Step 1: the cheap stuff
None of this removes a credential or closes an egress path, but it is a day of work.
Token scoping. The container authenticates with one JWT (SDUH_API_TOKEN), bound to a single session and a single tenant, lifetime capped to 20 minutes.
Runtime posture. On by default: cap_drop=ALL, no-new-privileges, pids_limit, pinned non root uid. Nothing exotic, it just raises the cost of an escape.
Clean images every run. No persistence across agent runs. Every attempt has to happen inside the same run, there is nothing to keep building on.
Untrusted evidence framing. A trust boundary block in the agent playbook, a data fence around query results in sduh-cli, filename sanitization, and mounting the raw incident and alert JSON read only. An injected run cannot rewrite the evidence an analyst reads afterwards.
Step 2: take the LLM API key out
The backend exposes POST /api/v1/inference/v1/messages, an Anthropic/OpenAI format passthrough authed with the same session JWT. SDUH-API holds the Foundry key and forwards server side. So there is no api key for Foundry in the agent.
Step 3: seal the network
Even with nothing worth stealing we need to close egress, making internet access hard makes the life of an attacker harder. No easy shell to keep interacting. No easy investigation of the internal network.
Each headless run gets its own --internal Docker network with exactly two members, the agent and the SDUH-API backend. internal strips the host route, so the agent reaches the API and nothing else. Internet, IMDS, the host, Redis, the database and the other sessions are all off the routing table. It is fail closed, a session that cannot be isolated raises rather than falling back onto the open bridge. The cleanup sweep reaps the networks.
Docker auto allocates a /16 per network, which caps the daemon at around 31 concurrent ones. That’s not enough so we built it into SDUH. The backend hands each network an explicit /28 out of 10.0.0.0/8, roughly a million networks, using a session offset first free scan with overlap retry.
End state
- Credentials: one session and tenant scoped JWT. No LLM API key, nothing that outlives the run.
- Network:
sduh-apion a private internal net. Internet, IMDS, host, DB, other sessions all give nothing. - Runtime: capabilities dropped, privilege escalation blocked, non root uid, PID capped, evidence mounted read only.
Validated end to end on staging: a real 4 minute run, 15 inference calls through the proxy, no LLM API key in the container, internet and IMDS and host all blocked, allocator handing out 10.160.240.96/28.
An injected run now has nothing worth stealing and nowhere to send it. Worst case it talks to the one API that already gates everything by its scoped token, and makes us spend tokens. Those are capped by another process we monitor in both Azure and the SDUH backend.
Why should you care
If you are running agents with tools, three things from the above are worth copying:
- Proxy inference through your own backend. The LLM API key is the widest credential in the container and the one nobody threat models, because it feels like infrastructure rather than access. Route it through your API and hand the container a session token instead. Per session token accounting comes along for free.
- Per session
--internalnetworks, fail closed. Killing the route to IMDS is most of the value. - Fence the evidence at read time, not just in the kickoff prompt. A preamble competes with everything the agent has read since. A read only mount does not.
None of this is new. Scoped credentials, no secrets on disk, network segmentation, this is advice from about 2010. It is just that sometimes people forget to apply it to agents, where the whole conversation is still about better filters.
Building muscles != pentesting muscles
Writing down what we knew we had not fixed is not the same as finding out. Building things and breaking things are different jobs, so we had Wesley, a colleague over at Zolder, pentest the setup. Whitebox: source code, the stack running locally, and a staging environment to confirm findings against.
Prompt injection did not work. He tried it through Defender and Sentinel alerts, and through poisoned threat intel data. The model flagged the injection. The part everyone worries about, the part this whole enclosure is built around, held. But what if tomorrow we swap models, or change a prompt, or somebody publishes a new technique. Defense in depth and all.
After this Wesley worked directly from the code and inside of the agent container. There Wesley found a server side request forgery in a SDUH API endpoint and a second one in the PDF renderer, and took the first one to admin on SDUH.
The chain was short. The SSRF gave him the environment, the environment still held enough to mint SDUH access tokens, and those tokens gave admin on the backend every agent run talks to. The PDF path had the same shape: influence the generation, trigger an outbound request, land on the same sink. It started from a foothold, that is the assume breach part, but from there nothing in the agent hardening was in the way. The agent was how he reached the web app behind it. The bugs were in the web app. We sealed the container and left the one API it is allowed to reach holding the keys.
So we did another round, defense in depth this time:
- the agent can only call a whitelisted set of API endpoints now, so the surface it reaches is a fraction of what it was
- the SSRF is fixed
- the PDF renderer is hardened, and the agent cannot call it at all anymore
sduh-apino longer talks to Docker directly, Docker work goes through Celery jobs- the host no longer lets a container mount root
None of that is exotic either. It is the same roughly 2010 advice as the rest of this post, applied to the seams we did not think to look at while we were busy making the thing work.
Building something, and thinking about how to build it, is a different mode from trying to take it apart. It is stupid, I have been pentesting for more than ten years and when I am building I still think purely functional. That is why it matters to have someone else look at your work and be critical.
What this does not fix
Verdict manipulation. Nothing above stops an injected string from convincing IVON that a real incident is benign. That attack needs no escape, no credential and no egress, and it targets the product rather than the infrastructure. We think any verdict should contain proof, proof that is verifiable against data in Sentinel/Defender allowing you to build deterministic checks.
Everything downstream of the agent. Attacker controlled strings flow out of the agent into report rendering, tickets and exports. Those processes run outside the sealed network with the egress the agent no longer has. But we do need to handle the output of the agent so we need to be aware of the possibility of manipulated data.
Backend endpoints that fetch things. Any API that takes a URL or an indicator and then makes an outbound request hands egress straight back. Confused deputy, easy to build while thinking you are building enrichment.
The host. Namespace isolation is not a security boundary. An escape out of the container lands the attacker on a host in a network with all kinds of services, so from there they can manipulate other agent runs or go after something else. A simple fix is running the agents on their own VM in a network that only reaches sduh-api, so an escape gets you the other runs on that host and nothing beyond it. The fancier option is a sandboxed runtime like gVisor or Azure Container Apps dynamic sessions.
Keep it in the flask
The security of this does not rest on IVON getting it right. It is a process that reads attacker-controlled text all day, and you do not make a process like that your security boundary.
That is the whole point of the flask. You assume the thing inside will fail, and you build so that when it does, there is nothing on the other side worth reaching. And when Wesley got in, it was not even the thing inside that failed. It was the glass. So you contain the homunculus, and then you get someone to check the flask.
Part of a series on running an agentic SOC — building IVON, our investigation agent, and working alongside it inside SDUH:
- Investigating incidents with agents — collaborating with IVON on a live case.
- Designing agent enclosures (you are here) — making it safe to point an agent at attacker-controlled data.
- Building detection rules with agents (coming soon) — the improve → test → analyse loop, backtested across every client.
- Designing tenant context for agents (coming soon).
