AgentOps engineer: watching something that chooses its path
Everything in conventional observability assumes the system does the same thing every time. A request arrives, a known sequence runs, and when something breaks the trace shows where. An agent decides its own sequence at runtime, which means two identical requests can take different paths, cost different amounts and take different lengths of time, all while succeeding. That breaks the instrument rather than the system. Latency becomes a distribution with a long tail rather than a number. Cost per request becomes a function of how many steps the thing decided to take. And the failure that matters most does not produce an error at all: the agent still answers, still returns a success, still responds in a reasonable time, and the answers have quietly got worse. Catching that requires running a fixed set of cases on a schedule, because no passive metric will show it.
AgentOps engineer, in short
Emerging role| In one sentence | Keeps agents running in production: tracing, cost, failure modes, and knowing when one has quietly got worse. |
|---|---|
| Judged on | Whether a degradation is caught before a user reports it. |
| Fails when | Observability is built for services and the thing being observed makes non-deterministic decisions. |
| Most confused with | MLOps, which watches models rather than the loops built around them. |
Real work, but the scope differs enough between employers that the title alone tells you little. No compensation figures: see methodology.
What a useful trace contains
Not just the steps. A trace that lists the calls made tells you what happened and not why, which is the question that actually gets asked when something goes wrong.
The trace needs, per step, what the agent was given, what it decided to do, why, what came back, and what that changed about the next decision. That last element is the one usually omitted and the one that makes a trace diagnosable rather than merely complete.
There is a retention question attached. These traces are large, they contain whatever data the agent handled, and keeping everything forever is neither affordable nor lawful in many contexts. The workable arrangement is full traces for a short window, sampled traces for longer, and permanent retention only of the decisions that had consequences, which is a policy decision somebody has to take rather than a default.
The three alerts that earn their place
Cost per request above a threshold. The cheapest early warning available. An agent that starts taking eleven steps where it used to take four is telling you something changed, and it shows up on the bill before it shows up anywhere else.
Step count above a threshold. The same signal without waiting for billing, and it also catches the loop that is about to run away. A hard cap should exist anyway; the alert tells you how often it is being hit, which is the interesting number.
Scheduled evaluation score dropping. The only one that catches silent quality decay. A fixed set of cases, run nightly, scored the same way each time. Everything else in this list is a proxy; this is the measurement.
Why quality decays without anything breaking
Several causes, none of which produce an error. The model behind the system is updated by its provider and behaves slightly differently. An upstream data source changes format and retrieval starts returning less relevant context. The distribution of what users send drifts away from what the system was tuned on.
In every case the agent keeps working. It retrieves something, reasons about it, and answers. The answers are worse and nothing in the infrastructure has an opinion about that.
This is why the scheduled evaluation run is not a nice-to-have in this role, it is the role. An organisation running agents without one is not monitoring quality; it is waiting for a user to complain, which is monitoring with a much longer feedback loop and a worse relationship attached.
Incident response when the cause is a decision
A conventional incident has a cause you can point at: a bad deploy, a failing dependency, a resource limit. An agent incident frequently has a cause that is a judgement the system made, which was reasonable given what it saw and wrong given what was true.
That changes the review. The question is not what broke but what the agent was given that led it there, and the answer lives in the trace rather than in the logs. Teams whose traces do not record the reasoning end these reviews with a shrug and a prompt change, which is how the same incident happens twice.
Where the job sits and where it is going
At most organisations this is part of platform or AI engineering rather than a separate headcount, and it separates out once there are enough agents in production that nobody can hold their behaviour in their head.
The skills are recognisable: observability, cost engineering, incident response, and the specific judgement of knowing which non-determinism is acceptable and which is a defect. That last one is not transferable from conventional operations, and it is what makes the role distinct rather than a relabelling.
The path out is toward platform engineering or toward AI engineering proper, and people arriving from this direction build noticeably more conservative agents, because they have spent a year cleaning up after ambitious ones.
