Why AI cannot replace operational accountability

When money goes missing from a business, nobody asks the spreadsheet to explain itself. People ask who was responsible for the process, who signed off, and who is going to fix it. The tool may have made the error, or caught it, or hidden it. It is never the one in the room when the questions start.

That small observation matters more now than it did a few years ago, because our tools have started to look like colleagues. They draft, reconcile, flag, recommend and, increasingly, act. It is tempting to follow that thread to its end and conclude that the last human in the loop is a temporary arrangement, a cost we will engineer out once the models are good enough.

I don't think so, and not because I doubt how good the models will get. I run Innbase, which builds software for hotels, restaurants and bars, and before that I co-founded Middletrust, an escrow payment service in Nigeria. In both, the real product is that someone can be trusted with money and with consequences. That experience has left me with a firm view: AI can take over a growing share of the work in operations, but it cannot take over the accountability for it. The rest of this essay is about why, and about what to build as a result.

Work and accountability are different things

Work is the activity: matching a payment to an invoice, counting stock, deciding whether a discrepancy is a mistake or something worse. Accountability is a relationship that surrounds the activity. It means that when the result is wrong, there is a specific person who must answer for it.

That relationship has four parts. It is worth naming them, because each is something a system can or cannot supply.

• Authority. The power to make the decision, and to change it.
• Answerability. Being the one who can be questioned, and who owes an account to someone with standing to ask.
• Consequence. Bearing something real when the outcome is bad, whether that is reputation, pay, a job or a licence.
• Remedy. The ability and the duty to put things right, including making the affected person whole.

A model can be given authority, in the narrow sense that we let it act. It can produce something that resembles an account of what it did. But it carries no consequences and makes no amends. Without those two, answerability is only a description of events, and delegation has no delegator who can be held to it.

“But the model can explain itself”

The first objection is that modern systems can give reasons for what they do. Ask why it flagged a transaction and it will tell you, fluently.

Reasons are useful, and I want them in every system I build. But a reason is not an answer in the sense that matters. When a manager answers for a decision, the account comes with a stake. They stand behind it, they can be wrong in a way that costs them, and the person asking can judge their sincerity over years. A model's explanation is produced after the fact, can be fluent and wrong at the same time, and has nothing riding on it. It tells you how the decision could be justified. It cannot tell you who will stand behind it.

Accuracy is not accountability

The second objection is that people fail too, and a system that makes fewer mistakes deserves more trust. That is often true, and where it is true we should use the system.

But error rates and accountability answer different questions. The error rate tells you how often to expect a problem. Accountability tells you what happens on the day the problem arrives. Every system, human or machine, will eventually produce a result nobody wanted. On that day somebody has to freeze the account, call the customer, explain to the owner and change the process so it does not happen again. That is an act, performed by someone with standing, and it is also how organisations learn. Consequences are the feedback loop. Remove the person who feels them and you remove the pressure that makes things get better.

Accountability in name only

The more worrying failure is not that we drop the human. It is that we keep one in name. A person is placed over a system they cannot inspect, cannot realistically override and do not have time to check, and is made formally responsible for its output.

Researchers who study automation call this a moral crumple zone: blame lands on the human nearest the machine, even when that person had little real control. It is easy to fall into because it looks like oversight on an org chart.

Responsibility without control is not accountability. It is a scapegoat with a job title.

Real accountability needs authority that matches it. If someone is going to answer for what a system does, they need to see what it saw, understand roughly why it acted, and be able to stop it, reverse it or overrule it before it matters.

What escrow taught me

Middletrust was an escrow service: a way for two parties who do not know each other to do business, with a third party holding the money until both sides are satisfied. The software was the easy part to describe. Funds are locked, conditions are met, funds are released.

What I came to understand is that the trust was never really in the software. It was in the fact that an accountable party stood behind it, one who would hear a dispute, make a decision and carry the consequence if it was wrong. The code moved the money. The promise was institutional. I think that is true of nearly every operational system that matters, and it is the clearest way I know to explain why more intelligence in the tool does not remove the need for an owner.

Where this shows up in operations

In a hotel, restaurant or bar, operations is mostly small decisions about money and stock, repeated hundreds of times a day. A sale is recorded in one place and the cash arrives in another. A delivery is received at one quantity and invoiced at a different one. A discount is applied by someone with the authority to apply it, or by someone without.

Software is very good at the first step, which is finding where the records disagree. That is pattern work, and it is where AI helps most. The second step is deciding what the disagreement means: an honest error, a gap in the process, a supplier problem or something worse. The third step is acting on it, which can mean a difficult conversation with a colleague whose livelihood depends on the answer. The first step can be automated almost completely. The second and third involve judgement and consequence, and they need a name attached.

A blueprint for building with AI in operations

If accountability cannot be automated, the practical question is how to design systems that strengthen it. These are the principles I keep returning to.

1. Name an owner for every automated decision. If you cannot say who answers for a class of decisions, the system should not be making them yet.
2. Show the evidence, not just the conclusion. The owner should be able to see the records and the reasoning behind a flag or an action, so that agreeing with it is a real judgement and not a reflex.
3. Match autonomy to the cost of being wrong. Let the system act alone where a mistake is cheap and easy to reverse. Require a human decision where it is expensive, public or permanent.
4. Give the owner real power. They need the ability to pause, override and correct, and the time to do it. Responsibility without control is the crumple zone again.
5. Keep a record that outlives the model. Log what was seen, what was recommended, who decided and what happened next, so that an account can be given months later by someone who was not in the room.
6. Measure outcomes, not activity. Count the problems actually resolved and the losses actually prevented, not how many alerts were produced or tasks closed.

None of this slows AI down where it is strongest. It puts a shape around it, so that the speed lands somewhere safe.

The two a.m. test

I try to apply one test whenever a system takes on a new job. Imagine it is two in the morning, something has gone wrong, and the person it affects is on the phone. Who picks up? If the honest answer is that nobody does, the system has been given a responsibility that nobody holds.

AI will keep changing who does the work and how much of it gets done. That is a good thing, and it is what I am building for. It will not change who has to answer when the work goes wrong. The best operators I know understand this instinctively: they use every tool available, and they never confuse the tool with the owner. The companies that get this right will be the ones that make owners more capable, not the ones that try to make them unnecessary.