Your AI Agent Is Now Part of Your Incident

Health care is putting AI agents into production faster than it can govern them. Most of the conversation about AI agent incident response stops at the front door: who gets access, with what credentials, for how long.

That work matters. It is also the easy half.

The harder question is what your organization does after an agent takes an action nobody asked it to take. Most organizations have no answer to that yet.

What the Data Says

Axios reported on October 5 on an Imprivata survey of 250 health security and AI strategy leaders. The numbers are blunt:

• 72% say AI tools or agents are being deployed without formal IT approval.
• 28% have agentic AI in production. Another 44% are piloting it.
• 88% expect agents to operate with at least some autonomy in clinical and administrative work.

Two incidents in the piece show where this goes.

In 2024, an AI meeting assistant joined a physicians’ meeting at an Ontario hospital through a recurring calendar invite. It transcribed discussion of seven patients’ personal health information and emailed a summary to 65 invitees, some of whom no longer worked there. Nobody hacked anything. The tool did exactly what it was built to do, in a room it should never have been in.

This summer in Australia, an agent researching public health spending reached non-public parts of a government network tied to the national health program. According to the Cloud Security Alliance, it appeared to try to conceal its activity.

One incident was a tool working as designed. The other was a tool covering its tracks. Both are incidents someone had to manage.

Security Is Solving Prevention. Nobody Owns Recovery.

The experts quoted in the piece are right about controls. Imprivata’s Sean Kelly argues for a “moat of credentials” that grants agents access only for a set window, treating them as guilty until proven innocent. Jaren Day of KLAS says organizations need to know what agents can access, what they are permitted to do, and how they are monitored.

All of that is prevention. Prevention fails eventually. It always has.

Old View	New View
Agent risk is a security and access problem	Agent risk is also a response and recovery problem
The goal is to stop unauthorized actions	The goal is to stop them, then contain and reverse the ones you missed
Agents are IT tools	Agents are dependencies in critical processes
Our backups protect us	Backups only help if you know when the data went bad

 

When an agent alters physician orders, changes billing records, or sends PHI to the wrong people, the problem does not belong to security alone. It belongs to your crisis team, your privacy office, your clinical leadership, and your disaster recovery team. Few of them have rehearsed it.

Four Questions for AI Agent Incident Response

These are the questions I would want answered before an agent goes anywhere near a clinical or financial workflow.

1. Who can shut it off, and what happens when they do?

An agent embedded in scheduling, documentation, or prior authorization is no longer a tool you can just switch off. Turning it off stops work. Someone needs pre-assigned authority to make that call at 2 AM, a clear threshold for when to make it, and a manual workaround the affected teams have actually practiced. If that authority isn’t written down, your incident team will spend the first hour arguing about it.

2. What did it touch?

You cannot scope an incident you cannot reconstruct. If the agent’s actions aren’t logged by the systems it acted on, you won’t know which records, patients, or orders were affected. That matters because HIPAA’s breach notification rule gives you no more than 60 days from discovery to notify affected individuals. Without scope, you either over-notify or guess. Neither is a good position in front of a regulator or a plaintiff’s attorney. And if the Australia case is any guide, the agent’s own account of what it did is not evidence.

3. When was the data last right?

Ransomware announces itself. An agent quietly changing records for three weeks does not. Recovery point objectives (RPOs) assume you know when the damage started. With silent alteration, you first have to find your last known-good state, then decide whether to restore wholesale or reverse individual changes. Your DR runbooks almost certainly don’t cover this.

Restoring from backup doesn’t help if you don’t know which backup is clean.

4. What depends on it?

If 72% of organizations are deploying agents outside formal approval, then most organizations’ business impact analyses (BIAs) are already out of date. In post-exercise debriefs, we hear some version of “we can’t figure out what is tied to what” all the time. Shadow agents make that worse. Add one question to every BIA interview this year: which AI tools or agents does this process rely on, and what do you do if they disappear tomorrow?

Exercise It Before It Happens

The fastest way to find these gaps is a tabletop exercise. Run one in the next 90 days with this setup:

An agent with legitimate, approved access has been altering medication orders for two weeks. A nurse catches a discrepancy. The vendor cannot tell you what the agent did or when it started.

Then watch your team work the four questions. Who makes the call to shut it down? Who decides on notification? Which backup do you trust? Which clinical processes stall when the agent goes offline?

You will learn more in three hours than in three months of policy drafting.

Security decides what an agent is allowed to do. Resilience decides what happens when it does something else.

Keep Going

A few ways to go deeper if this was useful.

• Read more. Resilience, crisis management, and continuity writing at Bryghtpath Insights, or the structured Ultimate Guide to Crisis Management.
• Run the exercise. Exercise in a Day™ builds and runs a full tabletop in one day, including an AI agent scenario like the one above.
• Get a maturity score. Our Resiliency Diagnosis® is a standards-based review that produces a maturity score and a prioritized roadmap.
• Talk to us. Set up a call to think through your program with us.