Governance and accountability

Governance settles who may change the tool, who watches its performance, who acts when it fails and who answers for it. Dutch hospitals already govern medical technology under the Convenant Veilige Toepassing van Medische Technologie in de medisch specialistische zorg (an agreement on the safe use of medical technology in hospital care), and an AI tool can fit into that structure.

Why governance matters¶

When an AI tool contributes to a clinical error, several questions arise at once. Was the output wrong? Did the clinician check it carefully enough? Was the tool used in a setting it was not validated for? Was the user trained? Was the model version up to date?

If nobody owns these questions, the answers default to the person closest to the patient. The nurse who acted on an alert ends up explaining the decision in a complaint procedure, while nobody is responsible for the alert threshold that produced it. The underlying fault is not reported or fixed, and the same failure can happen again.

Key governance roles¶

Name a person for each role before go-live:

Governance structures¶

Three structures carry most of the work. An AI governance committee, with clinical, technical, legal and patient members, meets at set intervals to review monitoring reports and decide on significant changes. An incident reporting route lets clinicians and patients report concerns about AI outputs and sends each report to both the quality and safety lead and the system owner. A change management process reviews every proposed model update, workflow change or extension to new wards or patient groups before it is put into practice.

Accountability under the EU AI Act¶

The EU AI Act European Parliament and Council (2024) splits responsibilities between providers, who develop a system and place it on the market or put it into service, and deployers, who use it in their professional work. For high-risk systems, providers answer for the design, the performance claims and the technical documentation. Deployers answer for using the tool within its intended purpose, assigning human oversight to competent staff and monitoring its operation (Art. 26). Both have incident reporting duties. A hospital that builds and uses its own tool is both provider and deployer, so your governance structures must assign both sets of tasks and record who holds them.

Patient rights and transparency¶

Under the WGBO, the Dutch law on medical treatment contracts, a care provider must inform the patient about proposed examinations and treatment (Civil Code Art. 7:448) and needs the patient’s consent (Art. 7:450). Decide as an institution how and when clinicians tell patients that an AI tool contributed to their care.

The GDPR European Parliament and Council (2016) gives patients the right not to be subject to a decision based solely on automated processing that has legal or similarly significant effects (Art. 22). Where such a decision is allowed, safeguards include the right to human intervention, to give one’s view and to contest the decision, and patients must receive meaningful information about the logic involved (Arts. 13(2)(f), 14(2)(g) and 15(1)(h)). If a clinician genuinely reviews the AI output and makes the decision, Art. 22 does not apply, but explaining the tool’s role to patients is still good practice.

The EU AI Act adds a right to an explanation of an individual decision (Art. 86), but only for people affected by a deployer’s decision based on an Annex III high-risk system with legal or similarly significant effects. A medical device that is high-risk only through Annex I, such as the running case’s sepsis tool once CE-marked, does not fall under it.

Set up procedures for patients to use these rights, and make sure clinical staff know how to respond.