Consider a hypothetical rota agent that covers a shift by picking whoever's available and qualified. Over a few months, one nurse ends up covering more weekend shifts than anyone else on the team, not because a person chose to lean on her, but because she happened to satisfy the agent's rules most consistently. Nobody decided this was fair. Nobody decided it was unfair either. Nobody decided anything, and that's exactly the problem.
A review of individual shifts can miss the pattern across the rota. The design needs both views.
#What the ethical question actually is
Most agentic AI conversations are about the person running the system. Can she trust the output. Can she check the reasoning. That's real, and it matters.
It leaves out someone else entirely: the person the decision is actually about, who never opens a dashboard, never sees a citation, and often doesn't know an agent was involved at all.
An agent optimising for "available and qualified" isn't malicious. It's also not fair by default. Fairness isn't a byproduct of accuracy, it has to be designed in on purpose, the same way a hand-off rule or a citation requirement does.
#Fairness is the product, not a side effect
For agents that repeatedly affect the same people through staff cover, supplier approval or case flagging, build a review into the operating process. If an agent is deciding about the same people over and over, someone reviews the pattern, not just the individual calls, on a schedule, not only when a complaint arrives.
Design the system to surface patterns for review; a check on each individual decision cannot establish that the overall treatment is fair.
If a decision-maker wouldn't be comfortable explaining a pattern to the person it affects, that's the signal to fix the rule, not the output.
#Three questions before an agent decides about a person
We won't let an agent make repeated decisions about people, not just processes, until we can answer these three, in plain English, to the person the decisions land on:
- Who reviews the pattern, not just the individual decision? If nobody's looking at the shape of outcomes over weeks or months, a bias nobody intended can run for a long time before anyone notices.
- Does the person affected know an agent was involved? Finding out after the fact turns an accurate decision into a trust problem, even when nothing was actually wrong.
- What happens when the pattern looks off? Say so and adjust the rule, or wait for a complaint, that decision needs to be written down before the agent goes live, not discovered the first time it happens.
#Match the review to the consequences
Process decisions can still affect people. Repeatedly flagging one supplier’s invoices can delay payments; prioritising cases can change who receives attention first. Trace those consequences before deciding how much review is appropriate.
For low-impact internal formatting, a simple quality check may be enough. Where decisions affect access, workload or payment, name a reviewer, inspect outcomes over time and provide a route to challenge or correct them.
#Where to start
Pick one agent already making repeated decisions about the same group of people. Write down who, if anyone, currently reviews the pattern across those decisions, not the individual calls. That's usually where the ethical work needs to start, and that's usually the review that was never going to happen on its own.
If you can't name who reviews the pattern, the agent isn't the thing to fix first. The review is.
Recognise any of this in your own business? Tell us about it and we’ll say whether it’s worth automating.
Talk through your process
