Artificial intelligence (AI) is moving quickly from experimentation to operational reality across healthcare. AI governance in healthcare is now a central concern as organizations deploy AI to support administrative efficiency, clinician workflows, patient engagement, documentation, coding, revenue cycle activity, and decision support. The opportunity is real. So is the risk.
For C‑suite leaders and audit committees, the key question is whether AI adoption aligns with the organization’s mission, is governed with discipline, and can withstand a highly regulated, high‑consequence environment. In healthcare, poor AI decisions do not just create technology debt. They can create compliance problems, operational disruption, patient safety issues, and reputational damage.
Start With Strategy, Not Technology
A strong healthcare AI program starts with strategy, not tooling. Before discussing models, platforms, or vendors, leadership should be clear on five foundational elements: mission, vision, goals, guiding principles, and desired outcomes. Mission defines why the organization is using AI at all. Vision clarifies what the future state should look like. Goals translate that vision into business priorities. Guiding principles establish the organization’s risk posture and ethical boundaries. Desired outcomes define how success will be measured.
That may sound basic, but this strategy is where many AI programs fail. Organizations often move into pilots before deciding what problems AI should solve, what risks are acceptable, and which decisions should remain human-led. Gathering use-case information and taking the time to align the use cases with a long-term strategy can improve your ability to succeed. In healthcare, that lack of clarity is especially dangerous. AI cannot be treated as a disconnected innovation track. It must be tied to enterprise priorities such as quality of care, operational efficiency, workforce support, patient experience, compliance, and trust.
Build the Right AI Program Architecture
Once strategy is clear, leadership should focus on the architecture of the AI program itself. That architecture is both organizational and technical. On the organizational side, healthcare providers and related entities need clear decision rights, accountability, and escalation paths. Someone must own AI strategy. Someone must own risk. Someone must own implementation standards. And the boundaries between IT, security, compliance, legal, clinical leadership, privacy, and internal audit must be explicit rather than assumed.
Ground Governance in a Recognized Framework
This is where governance frameworks matter. The NIST AI Risk Management Framework is one of the most practical starting points because it is flexible, voluntary, and designed to help organizations incorporate trustworthiness into the design, development, deployment, and use of AI systems. It is structured around four core functions: govern, map, measure, and manage. In plain terms, that means establishing oversight, understanding the use case and context, evaluating risks and performance, and then treating and monitoring those risks over time. NIST also emphasizes issues that are highly relevant to healthcare, including accountability, documentation, human oversight, third-party dependencies, privacy risk, bias evaluation, and ongoing monitoring.
For healthcare organizations, governance cannot stop at a general AI framework. It has to connect directly to existing regulatory and control environments. HIPAA’s Security Rule already requires covered entities and business associates to implement administrative, physical, and technical safeguards for electronic protected health information (ePHI). The U.S. Department of Health & Human Services (HHS) also states that risk analysis is foundational and that organizations must assess risks and vulnerabilities affecting the confidentiality, integrity, and availability of ePHI. In other words, AI governance in healthcare should not be treated as separate from existing compliance obligations. It should be integrated into them.
Evaluate the Core Technology Stack
That integration becomes especially important when leadership evaluates the core technology stack. Boards and audit committees do not need to approve every technical component, but they do need confidence that the stack is fit for its purpose. That includes the underlying models, cloud services, orchestration layers, identity and access management, data pipelines, monitoring tools, logging, and third-party integrations.
In practice, the security and reliability of an AI program often depend more on the surrounding stack than on the AI model itself. Questions to consider are:
- Can the organization segment sensitive environments?
- Can it control which users and applications can access prompts, outputs, and fine-tuned models?
- Can it log system activity and review it?
- Can it prove where data came from, how it was transformed, and where it flowed?
- Can it disable or roll back an AI-enabled process if outcomes drift or controls fail?
Those are architecture questions, but they are also governance questions.
Identify Healthcare-Specific Cybersecurity & Data Protection Needs
Cybersecurity and data protection requirements are particularly acute in healthcare because AI often operates close to regulated, sensitive, and operationally critical data. The HIPAA Security Rule is explicit that organizations must protect ePHI with reasonable and appropriate safeguards, and HHS guidance highlights the importance of audit controls and protecting data integrity against improper alteration or destruction. Compromised data can create clinical quality issues and patient safety concerns.
Make Data Transformation a Priority
There is another issue that deserves more executive attention than it usually receives: data transformation. Many AI programs are delayed or weakened not by model selection, but by poor data readiness. If the underlying data is fragmented, inconsistently labeled, poorly governed, or operationally misunderstood, AI will scale confusion faster than it scales value.
In healthcare, this problem is amplified because data lives across clinical, operational, financial, and third-party environments, often with inconsistent standards and ownership. The question is not simply whether the organization has enough data. The question is whether it has usable, trusted, well-understood data for the intended AI use case. For leadership, this means asking if
- data lineage is clear,
- data quality is measurable,
- unstructured content is being handled appropriately,
- retention and access policies are aligned to the use case, and
- the organization can distinguish authoritative data from convenience data.
Sustain Change Through Governance & Momentum
AI governance is not sustained by policy alone. It requires operating model change, leadership sponsorship, education, and disciplined prioritization. The organizations that succeed are not the ones with the most pilots. Creating a repeatable path from use-case discovery to risk evaluation, technical design, deployment controls, monitoring, and review can help with success.
For audit committees, that means looking beyond whether a policy exists and asking practical questions.
- Does management have an inventory of material AI use cases?
- Are use cases tiered by risk and business significance?
- Is there a defined approval process for new AI deployments?
- Are third-party AI tools subjected to due diligence?
- Is data use traceable?
- Are controls being tested?
- Is internal audit equipped to assess AI-enabled processes, not just traditional IT controls?
How Forvis Mazars Can Help
Forvis Mazars helps healthcare organizations take practical, structured first steps toward effective AI governance. We begin with focused discovery and use-case identification to clarify where AI can deliver meaningful value, which use cases introduce elevated risk, what data each use case depends on, and where governance capabilities may need to mature. From there, we help develop a strategy road map that sequences governance alongside AI adoption, not after it.
That road map can address priority use cases, risk tiers, policy considerations, architectural implications, control expectations, workforce enablement, and assurance activities. While it may seem like a lot to consider upfront, establishing this go‑forward strategy before implementation can help reduce friction, align stakeholders, and support more confident decision making as AI initiatives scale.
Healthcare organizations do not need to address AI governance all at once, but they can benefit from treating it as a core enterprise capability rather than a standalone initiative. Forvis Mazars brings a healthcare‑informed, risk‑aware perspective that helps organizations move deliberately, grounded in strategy, disciplined architecture, data readiness, and governance. As a result, we can provide a clearer path to capturing AI’s potential while managing complexity and risk.
To discuss how Forvis Mazars can support AI governance in healthcare at your organization, connect with our healthcare and technology risk professionals today.