When the Model Is Wrong, You're Still Accountable

“The AI got it wrong” is not a defence anyone will accept.

Not the regulator, not the auditor, not the member of the public who was harmed, not the board asking how this happened. They will all ask the same thing: who decided to rely on it, who checked it, and who was answerable for the outcome. And the answer to all three is going to be a person, or a group of people, on your side of the table. It was never going to be the model.

This is the quiet shift that the excitement skips over. You can delegate the work to a system. You cannot delegate the accountability for the work. Those two things feel like they move together, and they do not.

Why responsibility feels like it transferred

When a person does a task, responsibility sits with them in a way you can feel. When you hand the same task to software, something about the handoff makes the responsibility seem to go with it, as though the system now owns the outcome and you’ve stepped back to merely operating it.

Part of this is old and reasonable. We trust deterministic software precisely because it is accountable in its own way: it does the same thing every time, it was tested, and when it fails it fails in ways you can trace and reproduce. Decades of that have trained us to treat “the system handles it” as a real transfer of responsibility, because with that kind of system it largely was.

AI breaks the thing that made the trust safe, while keeping the feeling that earned it. It produces output fluent enough that “the system handles it” feels just as true, but it is not deterministic, not reliably testable in the same way, and not traceable to a rule you can point at. The sense that responsibility has moved is still there. The conditions that justified that sense are gone. So teams relax their supervision at exactly the moment the work needs more of it, not less.

The principle: accountability is a person, not a place in the pipeline

Here is the lens. For any outcome your organisation is answerable for, there must be a person who is answerable for it. Inserting a model into the middle of the process does not create a new place for that accountability to live. It just changes what the accountable person is now responsible for supervising.

Before, they were responsible for the work. Now they are responsible for the work and for the judgement that the model could be trusted to do it, for the checks that catch it when it can’t, and for the outcome when those checks miss. That is more responsibility, held by the same person, not less responsibility because a machine took some on. The machine takes on none. It can’t be sanctioned, can’t be struck off, can’t be called before an inquiry. Everything that flows from a wrong answer flows to the humans around it.

Design from that fact rather than against it. For every place AI touches a consequential decision, you should be able to name the person accountable, state what they are relying on the model to do, and describe how they would know if it failed. If you can’t name the person, you haven’t removed the accountability; you’ve just lost track of where it sits, which is a worse position, not a freer one.

What this looks like

Consider a financial-services team that puts a model between client data and the advice tooling, to draft suitability assessments for an adviser to review. Reasonable design: there’s a human in the loop, the adviser signs off, the model just drafts.

The failure isn’t dramatic. It’s that, over months, the drafts are good enough that the review quietly becomes a formality. The adviser is busy, the model is usually right, and reading every draft against the underlying data as carefully as if they’d written it themselves stops happening, not through negligence but through the ordinary erosion that “usually right” produces in any human checker. Then comes the assessment that’s wrong in a way that matters, signed off, acted on. When the regulator asks how a client received unsuitable advice, “the model drafted it and the adviser approved it” describes the mechanism precisely and excuses no one. The accountability was the adviser’s the whole time; the model’s fluency just made it easy to stop exercising it.

The lesson isn’t “don’t put a human in the loop.” It’s that a human in the loop is only accountability if the loop is real: if the checker has the time, the information, and the genuine ability to disagree with the model and be heard. A rubber stamp is not supervision, however senior the hand holding it.

What it means in practice

When you introduce AI into anything consequential, spend as much design effort on the accountability as on the capability. Name the answerable person. Make their check real: give them the source material the model worked from, the time to use it, and a path to override that doesn’t cost them anything to take. Watch for the slow decay of a checkpoint into a formality, because that decay is the default, not the exception, and it happens to good people under normal pressure.

In a regulated Australian context this isn’t optional diligence; it’s frequently the thing an obligation already requires. Accountability for advice, for care, for a decision affecting someone’s rights doesn’t have a clause that suspends it when a model is involved. The duty stayed exactly where it was. The model just joined the list of things the accountable person is now responsible for getting right.

Next: the constraint that doesn’t bend to any of this: the data you simply can’t put into the model in the first place.