← Resources

Capstone · MIT Professional Education

An AI agent that helps a health service govern its AI

Health services are adopting AI faster than they can govern it. I designed one AI agent that does the coordinating work of an AI governance committee, from first idea to retirement, to address the barriers of time and AI expertise and the over-reliance on the clinician to be the backstop to be accountable for an AI.

7governance gates, from first idea to retirement
0decisions made by the agent. People make every one.
62%of the running cost is people in the loop, by design

The gap

Three gaps: who is accountable for each AI, what AI the service has, and who governs it across its whole life.

Today: who carries the risk
Policy and frameworkon the shelf, with few specialists to apply itevidence in emails,contracts and minutesno namedownerno completeregisterapproved once,then forgottenThe clinician at the bedsidethe backstop
With the agent: accountability shared
Board and executiveaccept residual riskAI governance committeedecides at each gateNamed ownerbackup · sponsor · legalAI use registerevery AI, owner, next reviewOperations leadsupervises the agentThe agentprepares · monitors · reopensThe clinicianuses the output with judgementnot the only safeguardfirst idearetirementseven gates
  1. Identifying accountability. My research found that a health service's existing governance already covered most AI ethics principles, but not accountability. It is unclear who owns an AI, who accepted the risk and who watches it after go-live. The framework's AI use register template has no field for an accountable owner or backup.
  2. Knowing what AI is in use. A service cannot govern an AI it does not know about. The register records the tools people submit, so AI that arrives through a vendor update or an in-house build can go unregistered. Some of it never announces itself as AI: a rule-based score that ranks patients is still software shaping a decision.
  3. Governing across the whole life cycle. Approval is often treated as the finish line. After go-live, AI can fail without an obvious sign, nothing reopens a case when a vendor changes a model, and decommissioning needs a gate of its own.

The barriers behind them

  • Time. Services that are not digitally mature do not have the staff time for an AI governance committee's day-to-day work: intake, gathering evidence, coordinating reviews, keeping the register and monitoring.
  • AI assurance expertise. There are few AI assurance specialists to call on, in a setting of high risk and regulation. Most services have an AI policy and a framework but no practical way to apply them. The framework's own authors name AI-supported governance as a question for future research.
  • The clinician becomes the backstop. Clinicians stay accountable for the AI outputs that inform their decisions, but they cannot be the only safeguard. The organisation has to assess the AI, govern it and monitor it after go-live.
  • A framework built for older AI. The framework was developed and tested mainly on non-agentic AI. Agents act through tools and change after approval.

Build on what already works. Health services are already very good at governance: clinical governance, risk, privacy, cyber security and procurement are well established. AI governance should not be another thing to run. It belongs within the governance suite a service already has, so the agent maps each checklist item to an existing process or committee and flags only what nothing covers.

Why now. From 10 December 2026, private and not-for-profit health providers must say in their privacy policy which kinds of significant decisions their software makes or substantially shapes (Australian Privacy Principles 1.7 to 1.9). The trigger is any computer program, not only AI, and a human reviewer does not by itself take a tool out of scope. Public hospitals sit under state privacy law, but the board's question is the same: what is deciding about patients, who owns it, and is the human review real? The register, the accountability record and the oversight measures are how a service answers it.

What the agent does

It does the work of the AI operations group, supervised by a named person

Given a task such as “assess this AI”, the agent talks with the applicant in plain language, gathers the evidence itself from approved sources, assesses it in proportion to risk, routes specialist reviews to the right people, drafts the pack for each gate, keeps the AI use, risk and accountability registers current, and reopens a case when something changes. A named operations lead supervises it and is accountable for its work. Its limits are set in code, not in a prompt, so a weaker model produces more flagged gaps, never a wrong approval.

On its own, the agent may

  • Read approved policies, registers and service-desk records
  • Assess the evidence, in proportion to the risk tier
  • Draft the gate pack, labelled as AI-drafted, with a source for every finding
  • Flag a gap, out-of-date evidence or an overdue item
  • Hold a case until the reviews are done, or reopen it when something changes
  • Tell the right people what they need to do

It never

  • Approves an AI system or accepts a risk
  • Releases a report or closes a case
  • Gives clinical or legal advice, or decides device status
  • Reads clinical systems or patient data
  • Follows instructions hidden in vendor documents
  • Changes its own rules

Seven governance gates

The same gates for every AI. The depth follows the risk.

  1. 1

    Screening and go-ahead

    Is it AI? Route and risk tier, regulatory flag, business case, named owner and sponsor

    go, pause or stop

  2. 2

    Design sign-off

    Data flows and permissions, bias plan, privacy impact, consumer and First Nations input

    approve the design or ask for revisions

  3. 3

    Approval to deploy

    Validation, independent evidence, security and vendor assurance, contract and exit terms

    decide; the executive accepts residual risk

  4. 4

    Go-live confirmed

    Go-live checks, downtime procedure, early monitoring against baseline

    confirm, extend the pilot or roll back

  5. 5

    Periodic or triggered review

    Monitoring, override rates, incidents, vendor change notices, evidence expiry

    continue, change conditions or review

  6. 6

    Re-approval or withdrawal

    What changed since approval, drift, benefits against baseline

    re-approve, restrict or withdraw

  7. 7

    Decommission confirmed

    Data returned or deleted, records kept, register updated, lessons recorded

    confirm withdrawal

The agent finds out and drafts the packPeople decide at every gate

What it costs

The investment is in people, not the model.

A year of running costs: about $130,000

the model itself

Each square is about $1,300.

  • People in the loop$80,000 Operations lead who supervises the agent (0.5 FTE, $65,000) and an independent reviewer who checks its judgement (0.1 FTE, $15,000)
  • Keeping it safe and running$50,000 Security patching of the model supply chain ($25,000), evaluation and regression testing ($13,000), hosting and storage ($12,000)
  • The model itselfUS$27–122 At about 45 assessments and reviews a year (about 900 model calls), priced on a mid-tier or premium hosted model

Year 1 is about $331,000, including hardware, build and integration, security and clinical safety assessments, training for about 60 people and 15% contingency. From year 2, about $130,000 a year. Planning estimates in Australian dollars.

The models run locally, so there are no per-token fees. The reason is data control, not saving money: case content, vendor documents and policies stay in an Australian region.

Outcomes

What it is designed to deliver, and how we will know

Every AI found and registered

Including AI that arrives through a vendor update, with a register summary every staff member can see.

Gate coverage: 100% by the end of the run stage

Faster, better-evidenced decisions

A complete, sourced pack at every gate, each finding labelled fact, inference or not found.

Median days to decision and to a complete pack fall against baseline

A named owner for every AI

Owner, backup, sponsor and legal counsel recorded, and any gap shown, not hidden.

Open accountability gaps tracked to zero

Monitoring that does not rely on memory

Review dates set by risk, evidence that expires, and cases that reopen when a model, tool or vendor release changes.

Approved systems with current evidence rises

Oversight that is real, not rubber-stamped

Reviewers with the information, training, time and authority to challenge. Anyone can challenge a finding.

Override rate and review time monitored; false reassurance near zero

Capacity for smaller services

A developing service can run the functions of a mature committee, and projects not approved get feedback so they can resubmit.

Benefit at or above run cost by the end of year 1

Targets are proposed. Baselines are set in the first four months. No result is claimed yet.

Worked example: an ambient scribe

An electronic health record update switches on a note-taking feature that listens to consultations. Two wards are already using it. A nurse unit manager tells the agent in a chat. At approval to deploy, the agent checks ten evidence items:

None complete. Three partly found, seven not found: no local validation, unknown data residency, patients not told, no named owner. The agent recommends not approving it yet. People decide.

Beyond the framework

Eleven additions for agentic AI

The plan applies the framework of Freeman et al. (2026), with its 30 principles and a review checklist used in proportion to risk, and the governance model of Reddy et al. (2020). Because that framework was developed mainly on non-agentic AI, I added:

  • A thirteen-check agentic assessment that caps autonomy if a critical check fails
  • Outside content treated as data, never as instructions
  • Memory held per case, retrieval only from approved sources
  • Limits that hold even if the model misbehaves
  • An autonomy ladder: earned, and reversible
  • Evidence expiry and automatic reopening after a change
  • Measures of whether human oversight is effective
  • An accountability record that includes legal counsel
  • An agent catalogue, and discovery of unregistered AI
  • A tamper-evident audit trail with the reason for every action
  • The agent assessed with its own checks, plus 48 edge cases

Getting started

Crawl, walk, run. Never faster than review capacity.

Crawl · months 1–4Sandbox on synthetic data. Every output checked by a person. Local model chosen after testing. Baselines set.
Walk · months 5–7A 12-week pilot on a few real cases at the lowest risk tiers. Human-approved actions only.
Run · month 8 onAll AI uses, all tiers, all seven gates. Autonomy rises only on evidence and human approval.

An illustrative design for a hypothetical, mid-sized regional Victorian public health service, submitted as the capstone for MIT Professional Education's Applied Agentic AI for Organizational Transformation. Prototyped with synthetic cases; not deployed or validated. Key sources: Freeman S, et al. From principles to practice, BMJ Digital Health & AI 2026; Reddy S, et al. A governance model for the application of AI in health care, JAMIA 2020; Victorian Department of Health, AI in Victorian public health services. Model cost from the GPT for Work pricing calculator, October 2026. Disclosure duty: Privacy and Other Legislation Amendment Act 2024 (APPs 1.7 to 1.9), as summarised by Naomi Alefelder, HG Intel (2026).

Read the story behind it: Who is accountable for the AI? Building an in-house assurance agent for health.

Start a conversation

A strategy that won't move, a programme that's stuck, an AI system you're not sure how to govern — I would like to hear about it.

A 30-minute conversation, no charge. Tell me what you are trying to move.

Start a conversation