AI security engineer: untrusted input, one channel
Conventional software keeps instructions and data apart by construction. Code is code, input is input, and the boundary between them is where most of application security lives. A system built on a language model does not have that boundary: instructions and content arrive through the same channel, expressed in the same medium, and the model has no dependable way to decide which is which. A document it reads, an email it processes, a record a customer filled in, can all carry text written to redirect it, and no amount of input validation separates the two reliably. That is the property this role exists for, and it is why the useful controls are not about what the system is told. They are about what the system is permitted to do, which holds regardless of what it is persuaded to conclude.
AI security engineer, in short
Emerging role| In one sentence | Secures systems whose inputs are untrusted text and whose behaviour cannot be fully enumerated. |
|---|---|
| Judged on | Whether an attacker can make the system act outside its intended permissions. |
| Fails when | The threat model is written for software and the system is an agent with a write credential. |
| Most confused with | Application security, which it extends rather than replaces. |
Real work, but the scope differs enough between employers that the title alone tells you little. No compensation figures: see methodology.
What the role actually reviews
Identities and scope. What credential does this system run under, and what can that credential do. An agent with read-only access cannot delete a record whatever text it encounters. This is the foundation and it is frequently the first thing missing, because a shared service account with broad access was convenient during development.
The trust boundary on content. Which sources are trusted and which are not, written down. Most systems ship without anyone drawing that line, which means everything is trusted by default, including whatever a customer typed into a form.
The irreversible set. Which actions cannot be undone, and what stands between the system and those actions. This is the question that matters most and is usually answered implicitly by whoever added the tool.
What is logged. Not just actions: the reasoning that produced them. Otherwise an incident review is guesswork, and the same incident recurs.
Why instructional controls are theatre
Telling a system not to do something depends on it interpreting that instruction correctly in a situation nobody anticipated, which is precisely where its behaviour is least predictable. As a control it works until it is tested, which is the definition of one that does not work.
This matters for the role rather than just for the design, because a security function that approves instructional controls loses credibility with engineers quickly. They can see which constraints are real, and a reviewer who cannot is treated as a formality to be satisfied rather than a colleague to consult.
The corollary is that this job needs enough technical depth to distinguish the two. It is not a policy role with a technical veneer, and organisations that staff it as one get documents rather than security.
The failure that produces most incidents
Almost none of these begin as a security decision. They begin as a feature: a system that read and summarised acquires a tool that files a ticket, then one that updates a record, then one that sends a message. Each addition was requested, small, and reviewed by nobody.
By the time anyone looks, the permissions are wide, the approval flow was designed for a system whose worst outcome was a bad sentence, and the logging retains actions without context. Nothing was done carelessly.
The intervention that works is a rule rather than a process: adding any tool that writes requires the permission record to be updated first. It takes ten minutes and it is the only mechanism that reliably catches the transition from a system that suggests to one that acts.
Testing a system whose behaviour is not enumerable
Conventional security testing works from a list: these are the inputs, these are the paths, these are the boundaries. An agent has no complete list of behaviours, so a test suite cannot be exhaustive in the usual sense and pretending otherwise gives false assurance.
What works instead is testing the constraints rather than the behaviours. Can this credential do the thing it should not be able to do, regardless of how it is asked. Does the step cap hold. Does the approval gate actually block. Those are enumerable, they are what bounds the damage, and they can be tested the way anything else is.
The second half is adversarial work on the content path: feeding documents that carry instructions and observing whether the system acts on them. It will sometimes, and the finding is not a bug to fix but a measurement of how much the structural controls are carrying.
Where the role sits
At most organisations, inside application security rather than beside it, and it separates out where agents are numerous enough that the review load justifies a specialist.
The work connects directly to governance, which decides what may be built, and to AgentOps, which observes what is actually happening. A security function without either of those is reviewing designs with no record of the estate and no visibility into production, which is most of what makes the job hard.
