AI engineer: the job that stopped being about training
An AI engineer builds the system around a model rather than the model itself. Retrieval, tool use, agent loops, evaluation, guardrails, cost and latency: everything that turns a general capability into something that answers a specific class of question reliably enough to ship. The role separated from machine learning engineering for a straightforward reason. Once capable models could be called rather than trained, the hard part moved. It stopped being how to fit a model to data and became how to make a system built on one behave predictably when the inputs are arbitrary text from people who do not know what it can do. That is an engineering problem with an unusual property: the system fails fluently. It produces a confident, well-formed, wrong answer, and nothing crashes. Almost everything distinctive about this job follows from that one property.
AI engineer, in short
Established role| In one sentence | Builds the system around the model: retrieval, agents, evaluation, guardrails, latency and cost. |
|---|---|
| Judged on | Whether the system is accurate, fast and cheap enough, measured against an evaluation set that can be run on demand. |
| Fails when | The evaluation set stops resembling what users actually send, and the numbers keep improving while the product gets worse. |
| Most confused with | Machine learning engineer, which trains models rather than building on them. |
Hired as a distinct position across many companies, with a recognisable scope. No compensation figures: see methodology.
Why the role split off
For most of the last decade, applying machine learning to a problem meant assembling a dataset, training something on it, and serving the result. The scarce skill was the middle step, and the job title reflected that.
General-purpose models moved the scarcity. When a capable model can be called over an API, the question is no longer whether you can build one. It is whether you can build something around it that behaves: that retrieves the right context, that fails safely, that costs something defensible per request, and that you can tell has got worse.
That is a systems problem, and it draws on backend engineering far more than on statistics. Which is why a substantial share of people doing this job came from ordinary software engineering rather than from research, and why a strong backend engineer who has learned to diagnose model behaviour is frequently more effective in the role than a researcher who has not built production systems.
The evaluation problem, which is the whole job
Everything else in this role rests on one thing: knowing whether the system is good. That sounds trivial and is the part that goes wrong most often.
An evaluation set is a collection of inputs with known-good outputs, run on demand. It is the only instrument that tells you whether a change helped. Without one, every improvement is an opinion and every regression is invisible until a user finds it.
The characteristic failure is subtle. The evaluation set is built early, from what the team imagined users would send. Real usage drifts away from it, nobody updates it, and the numbers keep improving while the product gets worse. Teams in this state are working hard on a benchmark that has stopped describing anything.
The second failure is averaging. A change that raises the overall score by two points while destroying performance on one category is a bad change if that category is the one the business cares about. Aggregate numbers hide this by construction, which is why evaluation has to be sliced by the categories that matter to the people using the thing.
What the week contains
More evaluation work than anyone expects, and it is not clerical. Deciding what a correct answer is for an ambiguous input is a genuine judgement, and on anything domain-specific it requires arguing with someone who knows the domain.
Retrieval, which is where a large share of quality problems actually live. Most systems that give wrong answers are not reasoning badly; they were handed the wrong context. Chunking, indexing, ranking and knowing when to retrieve nothing at all are the unglamorous levers that move quality most.
Cost and latency, which decide whether a technically good system survives its first invoice. A design that makes four model calls where one would do is not a refinement, it is a quadrupling of the bill at volume, and this constraint tightens rather than loosens as usage grows.
And guardrails, meaning the design of what happens on the inputs where the system is least reliable. A system that declines to answer is often better than one that guesses, and deciding where that line sits is a product judgement the engineer usually ends up making.
How the role fails
Three ways, in roughly this order of frequency.
The evaluation set stops resembling reality. Described above, and the most common. The fix is procedural rather than technical: real production inputs, sampled and labelled on a schedule, feeding back into the set.
Chasing model capability instead of system reliability. A newer model arrives, the team upgrades, some things improve and some quietly break. Without a stable evaluation set this is invisible, and teams in this pattern spend a year moving sideways.
Building for a demo. Systems that work on curated inputs and fall apart on the real distribution. This is less an engineering failure than an incentive one, and it usually means nobody on the team has watched an actual user use the thing.
Where the role leads
Staff and principal engineering on the model platform is the standard path, and it is a deep one: the systems side of this field is not close to settled.
Founding an applied product is common, because the role gives an unusually clear view of what is now cheap to build and what is still hard. That is the most useful information a founder in this space can have.
And deployment work, which is the sideways move. An AI engineer who moves forward deployed keeps the technical foundation entirely and adds the half of the problem that evaluation sets cannot see: whether an organisation will actually use what was built.
