← Resources

AI assurance · Capstone

Who is accountable for the AI? Building an in-house assurance agent for health

For my capstone in MIT Professional Education's Applied Agentic AI for Organizational Transformation, I designed an in-house AI assurance agent for a health service. It starts from my research finding that accountability was the gap, and targets three gaps: who is accountable for each AI, what AI the service has, and who governs it across its whole life.

The agent prepares and people decide. The human in the loop is the foundation of safety for AI in a high risk sector like healthcare.

The gap my research found, and two more

Most health services now have an AI policy and an adopted framework. What they lack is a practical way to apply them. My research found the first gap below; my capstone work exposed the other two.

Accountability. When I studied how an Australian health service applied the NSW AI Assurance Framework to a wound care AI, its existing governance already covered community benefit, privacy and security, and transparency, and partly covered fairness. The gap was accountability. It was unclear who owned the AI, who accepted the risk and who watched it after go-live.

The register. A service cannot govern AI it does not know about. A register records the tools people submit, so AI that arrives through a vendor update or an in-house build can go unregistered.

The life cycle. Approval is often treated as the finish line. The risk runs from first idea to retirement, and a vendor can change the model long after the committee said yes.

In those gaps, the clinician becomes the backstop. Clinicians stay accountable for the AI outputs that inform their decisions, and they should. But they cannot be the only safeguard. The organisation has to assess the AI, govern it and keep monitoring it after go-live, when it can fail without an obvious sign and a vendor can change the model without anyone reopening the case.

Two barriers sit behind these gaps: time and expertise. Most services, especially smaller and regional ones, do not have the staff time for an AI governance committee's day-to-day work. There are few AI assurance specialists to call on, in a setting of high risk and regulation. The framework's own authors name AI-supported governance as a question for future research. That is where my capstone starts.

I also started from a strength. Health services are already very good at governance: clinical governance, risk, privacy, cyber security and procurement are well established. AI governance should not be another thing to run alongside them. It belongs within the governance suite a service already has.

There is also a deadline. As Naomi Alefelder of HG Intel sets out, from 10 December 2026 private and not-for-profit health providers must disclose in their privacy policy the kinds of significant decisions their software makes or substantially shapes. The trigger is any computer program, including rule-based tools nobody calls AI, and software bought years ago is not exempt. Public hospitals sit under state privacy law, but every board faces the same question she poses: “what did you do to know?”

Today: who carries the risk
Policy and frameworkon the shelf, with few specialists to apply itevidence in emails,contracts and minutesno namedownerno completeregisterapproved once,then forgottenThe clinician at the bedsidethe backstop
With the agent: accountability shared
Board and executiveaccept residual riskAI governance committeedecides at each gateNamed ownerbackup · sponsor · legalAI use registerevery AI, owner, next reviewOperations leadsupervises the agentThe agentprepares · monitors · reopensThe clinicianuses the output with judgementnot the only safeguardfirst idearetirementseven gates

What I designed

The AI Governance and Assurance Agent is one AI agent that does the coordinating work of an AI operations group, the team that supports an AI governance committee in the framework by Freeman and colleagues (2026). A named operations lead, a person, supervises it and is accountable for its work.

Ask it to “assess this AI” and it talks with the applicant in plain language, gathers the evidence itself from approved sources, assesses it in proportion to the risk, routes the specialist reviews, drafts the pack for the committee and keeps the registers current. When a vendor releases a new model, it reopens the case.

What it cannot do matters more. On its own it may only flag a gap, hold a case, or tell the right person. It never approves an AI, accepts a risk or releases a report. Those limits sit in code around the model, not in a prompt, so a weaker model produces more flagged gaps, never a wrong approval.

Seven gates, one rule

Every AI passes the same seven gates, from first idea to retirement. The depth of the review follows the risk tier. At every gate, the agent prepares and people decide.

  1. 1

    Screening and go-ahead

    Is it AI? Route and risk tier, regulatory flag, business case, named owner and sponsor

    go, pause or stop

  2. 2

    Design sign-off

    Data flows and permissions, bias plan, privacy impact, consumer and First Nations input

    approve the design or ask for revisions

  3. 3

    Approval to deploy

    Validation, independent evidence, security and vendor assurance, contract and exit terms

    decide; the executive accepts residual risk

  4. 4

    Go-live confirmed

    Go-live checks, downtime procedure, early monitoring against baseline

    confirm, extend the pilot or roll back

  5. 5

    Periodic or triggered review

    Monitoring, override rates, incidents, vendor change notices, evidence expiry

    continue, change conditions or review

  6. 6

    Re-approval or withdrawal

    What changed since approval, drift, benefits against baseline

    re-approve, restrict or withdraw

  7. 7

    Decommission confirmed

    Data returned or deleted, records kept, register updated, lessons recorded

    confirm withdrawal

The agent finds out and drafts the packPeople decide at every gate

So what does it cost?

At about 45 assessments and reviews a year, roughly 900 model calls, a premium hosted model would cost about US$122 a year. A mid-tier model, about US$27. In my plan the models run locally inside the service's own environment, so there are no per-token fees at all. I chose local models for data control, so that case content, vendor documents and policies stay in an Australian region, not to save money.

The running cost is about $130,000 a year. Almost two-thirds of it is the human in the loop: an operations lead who supervises the agent and an independent reviewer who checks its judgement against their own.

A year of running costs: about $130,000

the model itself

Each square is about $1,300.

  • People in the loop$80,000 Operations lead who supervises the agent (0.5 FTE, $65,000) and an independent reviewer who checks its judgement (0.1 FTE, $15,000)
  • Keeping it safe and running$50,000 Security patching of the model supply chain ($25,000), evaluation and regression testing ($13,000), hosting and storage ($12,000)
  • The model itselfUS$27–122 At about 45 assessments and reviews a year (about 900 model calls), priced on a mid-tier or premium hosted model

Year 1 is about $331,000, including hardware, build and integration, security and clinical safety assessments, training for about 60 people and 15% contingency. From year 2, about $130,000 a year. Planning estimates in Australian dollars.

The models run locally, so there are no per-token fees. The reason is data control, not saving money: case content, vendor documents and policies stay in an Australian region.

Human oversight only works if it is resourced. An approver with no time, no training and no authority to say no is a rubber stamp. So the plan measures oversight itself: how often reviewers override the agent, how long they spend on a review, and whether the agent ever rated a risk lower than the people did.

The privacy regulator now looks at the same thing. Alefelder notes that its guidelines count the ability and likelihood of human override as a factor, and that a human reviewer does not by itself take a tool out of scope. Under real workload, with alert fatigue, review can become a reflex click that parks accountability at the bedside.

What it looks like in practice

In the worked example, an electronic health record update switches on a note-taking feature that listens to consultations. Two wards are already using it. A nurse unit manager raises it with the agent in a chat, and the case moves through the gates.

Worked example: an ambient scribe

An electronic health record update switches on a note-taking feature that listens to consultations. Two wards are already using it. A nurse unit manager tells the agent in a chat. At approval to deploy, the agent checks ten evidence items:

None complete. Three partly found, seven not found: no local validation, unknown data residency, patients not told, no named owner. The agent recommends not approving it yet. People decide.

That is the agent doing its job: making a gap visible early, while it is still cheap to fix, and handing the decision to the people accountable for it.

What is new for agentic AI

The Freeman framework is practical and grounded, and its authors are clear that it was developed mainly on non-agentic AI. Agents act through tools, read documents that may contain hidden instructions, and change after approval. So I added eleven extensions: outside content is treated as data and never as instructions, limits hold even if the model misbehaves, autonomy is earned and reversible, evidence expires and cases reopen after a change, and the agent is assessed with its own checks. These additions are provisional and untested in a real service.

Three things for health leaders

Budget for people, not tokens. The model is the rounding error. The supervision is the safety.

Put the limits in code. A prompt that says “do not approve” is a request. A tool that cannot approve is a control.

Design for the change after approval. A one-off review goes out of date the day a vendor ships a new model.

Where it stands

This is a design for a hypothetical regional Victorian health service, with a prototype built on synthetic cases. It has not been deployed or validated. A first implementation would start in a sandbox, move to a 12-week pilot on a few low-risk cases with every output checked by a person, and scale no faster than the people reviewing it.

The whole design is on one page here: the AI Governance and Assurance Agent one-pager.

Thank you to the MIT Professional Education faculty and learning facilitators for a course that made me build, not only plan.

Freeman S, et al. From principles to practice: an actionable framework for AI governance in healthcare organisations. BMJ Digital Health & AI 2026;2:e000018.
Reddy S, et al. A governance model for the application of AI in health care. JAMIA 2020;27(3):491–497.
Department of Health (Victoria): Artificial intelligence in Victorian public health services.
Model cost: GPT for Work, LLM API pricing calculator, retrieved October 2026. Budget figures are planning estimates in Australian dollars.
Alefelder N. Software in your service is already making decisions about patients. In December, you’ll have to disclose which ones. HG Intel, 2026. Privacy and Other Legislation Amendment Act 2024 (Cth), APPs 1.7 to 1.9.

Start a conversation

A strategy that won't move, a programme that's stuck, an AI system you're not sure how to govern — I would like to hear about it.

A 30-minute conversation, no charge. Tell me what you are trying to move.

Start a conversation