The Knowledge That Doesn't Transfer

Ask who actually understands the AI system, and the honest answer is often one person.

That’s a familiar risk with any software, and normally a recoverable one. If the engineer who built a service leaves, the service is still readable. You can open the code, trace the path, reproduce the behaviour, and reconstruct the reasoning from what’s written down. Ordinary software carries its own explanation, and a new person can learn it by reading it. The knowledge was never really trapped in one head. It was in the system, and the person was just the fastest way to reach it.

AI quietly removes that safety. Much of an AI system’s behaviour lives in the model, the prompts, and the data it learned from, and none of that reads back like source code. The same input can produce different outputs depending on wording, context, or the model version underneath. What you most want to recover later, why it behaves the way it does and where it can be trusted, is not written anywhere you can open. It sits in the assumptions the builder made, the evaluations they ran, and the judgement calls they made while tuning it. When they go, that goes with them, and the working system stays behind looking as confident as ever.

Why this is easy to miss

For most of building software you never treat explainability as a separate thing to protect, because you can’t build a normal system without it. The traceability comes free. A team can spend years assuming “we can explain and hand over anything we built” is simply how systems are, rather than a property they were getting for nothing.

AI withdraws the free version while keeping the appearance of it. The system still runs, still produces fluent output, still looks like something you could document in an afternoon. So the knowledge quietly concentrates in whoever built it, and nobody notices, because the cost of that concentration only arrives when you need to hand the system over, change it safely, or explain it under pressure. By then the understanding you needed has already walked out the door, or was never written down in the first place.

The principle: you can’t be accountable for a system no one understands

Here is the lens. For anything your organisation answers for, enough people have to understand how it works to challenge it, correct it, and account for it. A system that only one person understands, or that no one fully understands, has not removed your accountability. It has detached the accountability from the understanding that would let you meet it.

This reframes AI knowledge as operational, not technical. The implementation is the smallest part of it. What has to be captured and shared is the reasoning around it: what the system is for, the data it leans on and where that data is weak, and who checks the output before it counts. That is the knowledge that lets someone question an output instead of trusting it because it reads well.

It also sets a floor under how AI capability should sit in a team. When understanding lives in one or two heads, everyone else is reduced to trusting the outputs, which is exactly the posture a regulated team cannot afford. Product, operations, and compliance don’t need to build the model. They need to understand it well enough to know when to doubt it.

What this looks like

Consider a health-tech team that has built a model into its triage workflow, drafting summaries that clinicians review. It works, it’s been running for months, and one engineer designed the prompts, chose the model, and holds in their head every reason it behaves the way it does and every case where it doesn’t.

Nothing goes wrong while that engineer is there. The failure surfaces when they leave, or when a regulator asks how the system reaches its summaries and whether it treats similar cases alike. The people left holding it can describe what it does. They cannot explain why, cannot safely change it, and cannot give the account the regulator is asking for, because the reasoning was never in the system to begin with. It was in a person, and the person is gone. The organisation is now accountable for a system it no longer understands, which is a worse position than having no system at all.

The fix isn’t heavier documentation for its own sake. It’s making the understanding survive the individual: the intent, the data limits, the failure modes, and the review points written down and genuinely shared, so more than one person can stand behind the system when asked to.

What it means in practice

Start with visibility, because most organisations can’t yet say where AI is being used or planned. Build the inventory. For each use, capture what it does, who owns it, what model sits behind it, what data it touches, what it decides, and who checks it before it counts. You can’t govern what you can’t see, and the riskiest AI in most organisations is the quiet experiment that became load-bearing while no one was watching.

Then spread the understanding deliberately, past the person who built each system, so more than one person understands it well enough to challenge an output and hold the accountability that never left them. In a regulated Australian context this maps straight onto duties that already exist: to give reasons, to treat like cases alike, to answer for decisions affecting people’s rights. None of those were softened by the arrival of a tool whose knowledge is harder to transfer than the systems that came before it. If anything they raise the bar, because the easiest way to fail them is to let a system nobody understands quietly make decisions you’re still answerable for.