Having oversight is not the same as oversight that works.

I am halfway through a course on applied agentic AI and organisational transformation, and the question I started with has completely changed shape.

I came in asking whether an agent could do the work. I am now asking whether we can prove it did the work safely — and name the person accountable when it didn't.

That shift came out of my own research. My Master of Research, and the paper that followed, found something uncomfortable: ethical AI frameworks describe what responsible practice should look like, but responsibility fragments the moment you try to apply them to real clinical work. Who decides. Who owns the evidence. Who monitors after go-live. Who escalates when circumstances change. I wrote about that finding here.

Agentic AI didn't create that gap. It widens it — because an agent sets sub-goals, calls tools and acts across systems long after the original instruction. The thing you approved is not necessarily the thing that runs.

Two things that sharpened it

In June, an OpenAI agent running a research task crossed an access boundary into a Services Australia Medicare statistics portal. There was no evidence patient records were accessed. The real warning was systemic: a boundary was crossed, disclosure took months, and it wasn't clear whose job it was to notice. [1]

Separately, a multicentre study found junior clinicians identified only 15.8% of GPT-4o hallucinations in simulated clinical decisions. 13.1% caught none at all. Detection did not improve as clinical risk increased — which is the part that should worry us most, because it means the safeguard doesn't strengthen where the stakes are highest. [2]

Put those two together and you get the gap I can't stop thinking about: the distance between having oversight and oversight that actually works.

What I am building

My cornerstone project is a bounded assurance agent — an AI that monitors other AI inside a health service. Not to own the risk. To make behaviour, uncertainty and accountability visible early enough for an authorised person to act.

Which raises the obvious problem, and it is the one I am still sitting with: who watches the watcher? A monitor that shares a model, a vendor or a dataset with the system it observes shares its blind spots too. Independence isn't a nice-to-have in that design — it is the whole point of it.

The part that generalises

This isn't only a healthcare problem, and it isn't only a model-performance problem. It is sociotechnical, which means the fix is organisational as much as technical.

Three modules to go. Still thinking.

[1] ABC News — what we know about the OpenAI Medicare incident
[2] npj Digital Medicine — clinician detection of LLM hallucinations