AI in Healthcare: Compliance, Privacy, and the Right Model

There’s a particular kind of tension you feel inside any hospital technology committee meeting right now. On one side: genuine, documented pressure to adopt AI — for diagnostic speed, for documentation burden, for cost containment. On the other: a regulatory environment that is, frankly, changing faster than most organizations can track. Add to that the legitimate fear of a data breach that averages nearly $10 million per incident, and you have the conditions for either paralysis or recklessness. Neither is acceptable.

This piece is an attempt to cut through the noise. Not to oversell what AI can do in healthcare — there’s plenty of that already — but to honestly examine what compliance and privacy actually require right now, where the real risks live in model selection, and how forward-looking healthcare organizations are navigating all of it without losing either their patients’ trust or their operating budgets.

The Breach Reality Nobody Wants to Advertise

Let’s start with the uncomfortable numbers, because they contextualize everything that follows.

In 2024, the U.S. recorded 566 healthcare data breaches large enough to require HHS reporting — more than double the 216 incidents reported in 2010. But the volume isn’t actually the most alarming shift. The nature of the breaches has changed fundamentally. In 2010, hacking and IT incidents accounted for about 4% of cases. By 2024, that figure had climbed to 81%. Ransomware — essentially nonexistent in healthcare a decade ago — was involved in roughly 69% of all breached records last year.

The financial picture is equally stark. Healthcare breaches now cost an average of $7.42 million per incident according to HIPAA Journal data — though IBM’s 2024 Cost of a Data Breach Report put the sector average even higher at $9.77 million, marking the fourteenth consecutive year that healthcare topped all industries. The providers who experience these breaches don’t absorb costs quietly: nearly half raise prices to offset breach expenses, with about one-third increasing prices by 15% or more. Patients ultimately pay.

Why does this matter in an article about AI? Because AI deployments dramatically expand the attack surface and the data flows within healthcare organizations. Every new model integration, every API connection to an EHR, every ambient scribing tool creates a new potential vector. Understanding the breach landscape is prerequisite knowledge for responsible AI adoption — not a separate conversation.

HIPAA Has Evolved — And Most Organizations Haven’t Caught Up

HIPAA is often treated as a fixed, known quantity. Pass the training modules, sign the business associate agreements, encrypt the drives. But the regulatory environment around HIPAA and AI has shifted considerably since 2023, and several organizations are operating on an outdated mental model of what compliance actually requires.

The January 2025 Security Rule Overhaul

On January 6, 2025, the HHS Office for Civil Rights proposed the first major update to the HIPAA Security Rule in 20 years. This isn’t minor housekeeping. The proposed changes remove the distinction between “required” and “addressable” safeguards — a distinction that had allowed organizations to treat certain security controls as optional if they could document a reasonable alternative. Under the new framework, those safeguards are simply required.

For AI-deploying organizations specifically, the implications include:

One thing worth stating clearly: introducing AI into a healthcare organization does not alter the fundamental HIPAA rules governing PHI usage. AI tools can access, use, and disclose protected health information only for explicitly permitted purposes. A model analyzing patient records for treatment optimization falls under permitted treatment uses. Training a general model with PHI for commercial purposes requires explicit patient authorization — a line that several technology vendors have blurred in ways that create liability for their healthcare clients.

Beyond HIPAA: The State-Level Patchwork

Federal HIPAA is only part of the story now. Texas, California, Colorado, and several other states have layered additional requirements on top of HIPAA — covering algorithmic transparency, mandatory human oversight of AI-influenced clinical decisions, and stricter de-identification standards to prevent re-identification attacks.

And then there’s the federal wildcard: on December 11, 2025, President Trump signed an executive order aiming to preempt state AI laws and establish a “single national framework” for AI regulation. Whether this preemption holds up legally is genuinely uncertain — it’s being challenged — but it means the compliance landscape at the state level may shift again. Organizations shouldn’t bank on preemption as an excuse to stop state-level compliance work. It may never fully materialize, and the penalties in the interim are real.

The EU AI Act: A Framework Worth Understanding Even If You’re U.S.-Based

For organizations with any European operations, patients, or data flows, the EU AI Act is no longer a future concern — it’s a present one. But even for U.S.-only entities, the Act represents the most comprehensive AI governance framework in existence and is shaping how global vendors build their products.

The most significant practical finding: a peer-reviewed analysis in npj Digital Medicine found that approximately 75% of commercial AI-enabled medical devices are in radiology and classified as Class IIa or above under the Medical Device Regulation — meaning the vast majority of deployed clinical AI will be classified as high-risk under the Act and subject to its most demanding requirements.

High-risk status brings genuine obligations, not just paperwork. Conformity assessments, continuous post-market monitoring, mandatory human oversight by a qualified clinician, and registration in the EU AI database. A 2025 survey by Pharmaceutical Technology found that around 60% of EU-based pharma companies planned to implement AI risk management systems by 2027 — suggesting that roughly 40% are running behind schedule.

Choosing the Right AI Model: Where Most Organizations Get It Wrong

The question of which AI model to deploy in a healthcare setting isn’t purely a technical one. It’s a governance question, a liability question, and — if you’re being honest about it — a question about what kinds of failure modes you’re willing to accept. The answer depends heavily on the use case, and treating “AI” as a monolithic category is one of the most common mistakes organizations make.

The General-Purpose LLM Temptation

Large language models like GPT-4, Claude, and Gemini are genuinely impressive. They passed medical board exams years ago. They can draft clinical summaries, respond to patient queries, and assist with documentation in ways that measurably reduce physician burden. Advocate Health evaluated over 225 AI solutions before selecting 40 to deploy, projecting documentation time reductions of more than 50%. These are real gains.

But there’s a persistent gap between benchmark performance and real-world clinical reliability that doesn’t get discussed enough. Clinical diagnostic tools can disadvantage patient subgroups due to poor model generalizability — frequently caused by unrepresentative training data. This isn’t a theoretical risk. Published research in Nature Medicine found that LLMs may produce biased treatment recommendations based on patients’ sociodemographic characteristics — race, gender, insurance status, and other factors that shouldn’t influence clinical reasoning but demonstrably do when embedded in training data.

Specialized vs. General: A Decision Framework

The honest answer to “which model should we use?” is: it depends on what you’re trying to do, and you probably need more than one.

A critical point about radiology specifically: a 2025 study on autonomous AI auditing in clinical settings found that models achieving expert-level performance on benchmark datasets often fail when confronted with subtle variations inherent in real-world medical imaging — what researchers call “distribution shift.” The gap between laboratory accuracy and deployed accuracy isn’t a minor calibration issue. It’s the difference between a tool that helps and one that misleads at exactly the wrong moment. The same research showed that targeted interventions could recover 15–25% of performance lost under real-world distribution shift — but only when organizations were running active post-deployment monitoring, which most aren’t.

The Open-Source Question

Open-source models — LLaMA, Mistral, and their fine-tuned descendants — have become genuinely viable for healthcare use cases, particularly where data privacy requirements prohibit sending PHI to third-party APIs. Organizations using models for patient simulation or internal analytics often opt for smaller specialized models rather than large commercial ones — prioritizing efficiency, data control, and the ability to run inference entirely within their own infrastructure.

The tradeoff is real, though. Running models on-premise requires ML engineering capacity that most health systems don’t have in-house. A smaller, internally-hosted model with strong data governance controls may be preferable to a powerful cloud API that leaks PHI — but only if the internal team can actually maintain, monitor, and update it responsibly.

Privacy by Design: What It Actually Requires in Practice

Privacy by design is one of those phrases that gets used constantly and operationalized rarely. What does it actually mean when you’re deploying an AI system that processes patient data?

1. Data minimization at the architecture level

   The model should receive the minimum PHI necessary for the specific task. This sounds obvious, but in practice, many integrations pipe entire patient records to LLMs for documentation tasks that only require a fraction of that data. Audit what your APIs actually send.

2. Genuine de-identification — not just field removal

   Images, audio, and video can expose faces, voices, on-screen names, and date stamps that text-only filters miss. A de-identification pipeline that strips free-text fields but doesn’t process embedded audio or metadata in imaging files is incomplete. This is a known gap in many vendor solutions.

3. Business Associate Agreements that actually cover AI behavior

   Standard BAA templates haven’t kept pace with AI. Make sure your BAA explicitly addresses whether the vendor can use your PHI to train or fine-tune their models — because several default contracts have permitted this in ways that covered entities didn’t realize.

4. Post-deployment monitoring as a compliance function, not just a technical one

   Healthcare organizations that treat AI monitoring as purely an IT responsibility miss the clinical dimension. When a model’s recommendations start drifting from evidence-based guidelines due to distribution shift, that’s a patient safety issue, not just a performance metric. Compliance teams and clinical governance committees need visibility into model behavior over time.

5. Access controls that account for AI-mediated access, not just direct access

   Traditional HIPAA minimum necessary standards focus on human access to records. AI systems can aggregate and synthesize data in ways that far exceed what any individual user would see. Role-based controls on AI system access to PHI need to be designed with this aggregation potential in mind.

6. Patient consent and transparency — the gap most organizations underestimate

   States like Texas and California now require disclosure when AI influences care decisions. This isn’t just a legal checkbox — patients are increasingly asking about AI involvement in their care, and being caught unprepared is a trust issue that compounds over time. Build disclosure into care pathways proactively, not reactively.

The Tracking Pixel Problem Nobody Saw Coming

One underappreciated category of privacy risk: web tracking technologies. Kaiser Foundation Health Plan reported a 2024 breach affecting 13.4 million members caused by unauthorized disclosure of data via tracking technologies — including the Meta Pixel and Google Analytics — embedded in its patient portal and apps. These weren’t hacking incidents in the traditional sense. They were third-party tags passing health-related URL parameters (condition searches, appointment types, prescription queries) to advertising platforms.

The HHS guidance on tracking technologies has been updated, but enforcement is still ramping up. If your patient-facing digital properties use any third-party analytics or advertising tags, you need a dedicated audit of what those tags are sending — before OCR does it for you.

Building a Governance Framework That Actually Works

Governance frameworks for healthcare AI have a tendency to look impressive on paper and collapse under operational reality. A few principles that seem to separate the organizations that make this work from those that don’t:

What a Functional AI Governance Structure Looks Like

The Joint Commission, working with CHAI (a nonprofit coalition including Mayo Clinic, Duke University, and other academic health systems), released guidance in September 2025 on responsible AI use in healthcare settings. The framework emphasizes structured deployment, validation, testing, and safeguards for patient data — with a particular emphasis on pre-deployment evaluation rather than post-deployment remediation.

• Designated AI governance committee with clinical, legal, privacy, and IT representation — not just IT
• Pre-deployment risk stratification: administrative vs. clinical-facing, low-risk vs. high-risk clinical
• Bias evaluation across demographic subgroups before any clinical deployment
• Mandatory Business Associate Agreements reviewed specifically for AI data usage rights
• Defined human oversight protocols for all AI-influenced clinical decisions
• Post-deployment monitoring with defined triggers for model review or suspension
• Patient disclosure language integrated into care workflows (not buried in portal terms of service)
• Incident response plan updated to explicitly address AI system failures as a breach category
• Staff AI literacy training documented (required under EU AI Act from February 2025; increasingly expected under U.S. guidance as well)
• Vendor contract language reviewed against current HIPAA Security Rule proposed requirements

An Honest Assessment: Where We Are and Where This Goes

It would be dishonest to write about healthcare AI in 2026 without acknowledging both the genuine promise and the genuine failures that have accumulated over the past few years.

The promise: AI is delivering real, documented value in specific domains. Ambient documentation is reducing physician burnout in measurable ways. Radiology AI is genuinely catching findings that fatigued radiologists miss. Predictive sepsis models, when deployed with careful workflow integration and appropriate threshold-setting, have reduced mortality in several documented deployments. The evidence from systematic reviews covering sepsis detection, deterioration prediction, and emergency triage models shows that successful deployments invest heavily in stakeholder co-design, threshold selection, staff training, and continuous performance monitoring. That investment is not optional.

The failures: We’ve also seen AI systems deployed carelessly, with minimal validation, by organizations responding to competitive pressure rather than clinical need. We’ve seen vendors make accuracy claims that held up on benchmark datasets and fell apart in clinical practice. We’ve seen data sent to third-party APIs in ways that violated BAA terms. We’ve seen “de-identified” datasets re-identified using publicly available auxiliary data. These aren’t hypothetical risks.

The regulatory environment — HIPAA’s evolution, the EU AI Act’s staggered obligations, emerging state laws — is creating pressure toward better governance. That’s probably net positive, even if the compliance workload feels burdensome in the short term. Organizations that build genuinely robust AI governance frameworks now will have competitive advantages as the regulatory requirements tighten further. Those that treat compliance as a checkbox exercise will find themselves perpetually behind.

One final honest point: the organizations getting this right are not the ones with the most sophisticated AI strategies. They’re the ones that started with a clear inventory of what they were deploying, who had governance authority, how they’d know if something was going wrong, and what they’d do about it. That clarity — not the sophistication of the models — turns out to be the competitive differentiator.