AI risk management: deciding which failures you accept
Every system of this kind will be wrong sometimes, and not in the way software is usually wrong. There is no defect to fix: producing a plausible, well-formed, incorrect answer is part of how these systems work, and it remains true after every improvement anyone makes. That turns a technical question into a decision somebody has to take, which is which errors the organisation is prepared to accept, on which inputs, affecting whom, and what the system does when it is unsure. Almost nobody writes this down. The result is that the position gets set by default, usually by whoever last adjusted a prompt, and the organisation discovers what its tolerance was from a complaint rather than from a decision. Half a page per system, written before launch, prevents that entirely.
The three questions that make up the position
Is being quietly wrong worse than declining to answer? For most business uses it is, by a wide margin, and stating it changes the design: a system that says it is unsure is more useful than one that guesses, even though it looks less capable in a demo.
Are all users alike? They are not. An expert catches an error a newcomer acts on, so the same output carries different risk depending on who reads it. Systems serving both populations either need different behaviour or need to be set for the less experienced one.
Should the system ever act rather than suggest? This is the largest single change in risk profile available, and it is usually reached by drift rather than by decision. A wrong suggestion is disregarded; a wrong action has happened.
Sort failures by reversibility, not by likelihood
Conventional risk work ranks by probability multiplied by impact. That framing is less useful here, because the probability of an individual error is both unknowable in advance and roughly constant across a category.
Reversibility sorts more usefully. An error that produces a wrong draft somebody edits costs a few minutes. An error that sends a message to a customer, deletes a record or transfers money cannot be recalled, and the response to it is entirely different.
This is why the practical control is a boundary rather than a threshold: automatic where the outcome can be undone, human approval where it cannot. That line is easier to agree, easier to implement, and easier to audit than any accuracy target.
Measure your tolerance, do not assert it
A position on acceptable error is worth nothing without a way to know whether you are inside it. That is what the evaluation set is for, and it has to be sliced by the same categories the risk position uses.
An aggregate accuracy figure cannot tell you whether you are within tolerance, because the tolerance differs by category. A system at ninety-four per cent overall might be at ninety-nine on the routine majority and at seventy on the unusual cases where the money is, and only a sliced measurement shows that.
The related discipline is running it on a schedule. A tolerance checked once at launch tells you about a system that no longer exists, because the model behind it has been updated and the inputs have drifted.
The risk nobody writes down: the system that is right and unused
Risk registers for these projects list wrong answers, data exposure and regulatory exposure. They almost never list the outcome that actually happens most often, which is that the system works correctly and nobody uses it.
That failure costs the whole investment plus the running fee, and it is more probable than any of the others. It belongs in the register with an owner against it, because naming it changes behaviour: a project with adoption listed as its largest risk plans differently from one that treats adoption as a rollout step at the end. It is also the one risk a vendor has an incentive not to raise, which is a reason to insist on it being raised.
What regulators are converging on
The risk-based regimes now in force in the European Union and in South Korea, and the detailed voluntary framework published in Singapore, all ask variations of the same questions: how consequential is this use, what human involvement exists, and what record shows it.
An organisation that has written its own position has most of the answer already, in its own vocabulary, which is a considerably better starting point than assembling one against a classification scheme after the fact. Our market pages set out which regime binds where, checked and dated.
The point worth holding on to is that this work is not compliance-driven. An organisation subject to no AI law at all still benefits from knowing which errors it accepts, because that is the difference between a considered position and one discovered from a customer.
Half a page, before the build
If you are about to deploy something and nobody has written down which errors are tolerable, that is worth thirty minutes with us before it ships rather than after.
A first conversation is thirty minutes and is not a sales call. If the answer is that you do not need us, that is a useful outcome and we will say so.
