Most AI governance conversations in insurance end at deployment. The model passed testing, compliance signed off, it's live. But an AI system can start behaving differently in production even when nobody touched the model, because the data, prompts, workflows, and connected systems around it keep changing.
I asked Sean Kennedy, CISO at insurance operations and technology company Patra, what post-deployment oversight actually looks like inside a live insurance workflow: what they monitor, how they set human-review thresholds, who pulls the kill switch, and what agencies and MGAs should ask vendors before signing. His answers are below, lightly formatted.
Give me a concrete example of AI drift in policy servicing or claims where the model itself didn't change but the behavior did. What changed around it, and how was it caught?
AI drift doesn’t necessarily mean the underlying model changed. The behavior can shift because something around the model changed, whether that is the prompt, the data being fed into it, a connected system or the workflow itself. The model may still appear to be functioning normally while producing results that are different from what was originally tested and approved.
One failure mode we specifically design against is in page selection rather than the model itself. A trained classifier determines which pages of a policy reach the model, and it is scoring documents whose formats carriers continue to revise. If the classifier’s recall starts to degrade, nothing necessarily errors out. The model still runs, but it may be extracting information from the wrong pages.
We haven’t had a confirmed case of that happening in production, which is partly the point. This is the type of degradation that can happen silently over time, so our evaluation and retraining efforts are designed to catch it before it creates a larger issue.
That is why production monitoring has to focus on the behavior of the full system, not just whether the model itself has changed.
What are you actually monitoring in production to spot drift, and how often is someone looking at it?
In production, we continuously track throughput, error and retry rates, latency, timeout counts and per-model token usage. We also evaluate precision and recall against a labeled holdout whenever we retrain a classifier or swap models. Those metrics help us see whether the system is continuing to perform the way it was intended to once it is operating against real production data.
Our dashboards are reviewed weekly, but we do not wait for the next scheduled review if something starts moving in the wrong direction. An error or timeout spike, or a queue starting to back up, pulls someone in the same day. We’re also adding a vectorizer out-of-vocabulary rate to help flag degradation on unlabeled production documents between holdout refreshes.
More broadly, organizations need documented tolerance thresholds for accuracy and drift so there is a clear point at which a change in behavior requires further review. Every model in production should also have a named owner. Monitoring only works if someone is accountable for looking at the results and determining when intervention is necessary.
How do you set thresholds for human review without ending up reviewing everything by hand? What does a reasonable threshold look like in practice?
Requiring a person to manually review every action can eliminate much of the speed and scale that made automation useful in the first place. The goal should be to determine where human judgment adds value based on the risk and complexity of the task rather than putting an approval step in front of everything the AI does.
High-volume, repetitive work may be better handled primarily by AI, with people focused on exceptions, anomalies and higher-risk decisions. At the same time, the consequences of an error matter. Anything that touches a policy, payment or customer record warrants a more meaningful human checkpoint, even when model confidence is high.
There also isn’t going to be one universal threshold that works across every application. Organizations need to establish acceptable tolerances before launch, measure the system against real production volume and expand autonomy only where the results support it.
Where's the line between escalation, restricting the system, and a full rollback? Who gets to make that call?
Organizations should set the line between escalation, restriction and rollback before a system goes live, based on the operating envelope established during pre-release QA. That way, the response is tied to defined thresholds rather than being decided in the middle of an incident.
Escalation is appropriate for minor metric drift or an isolated output anomaly. The system can stay live while the affected cases go through expedited human review, with the AI/Product QA Lead owning the root-cause analysis. If the issue cannot be cleared within the agreed review window, it gets promoted.
Restriction makes sense when the problem can be contained to a specific tool, route or user segment. In that case, the AI Engineering Lead can isolate that path using feature flags or additional human-in-the-loop gates while the rest of the application continues running. The restriction should only be lifted once that path passes the same checks used before release.
A full rollback is for behavior that cannot be contained, cannot be traced to a known component or creates a broader safety risk across core workflows. That is when the Head of AI or VP of Engineering executes the kill switch and reverts to the last stable release or a manual fallback.
Those calls only work if the kill switch has already been tested, decision rights are clearly documented, and inputs, outputs and safety flags are logged. Otherwise, the organization ends up renegotiating the response during the incident instead of following a plan that was already in place.
Once an AI system is live, who is accountable for it: the vendor, IT and security, the business owner, or compliance?
The organization deploying the AI is ultimately accountable for the outcome. That responsibility does not transfer to a vendor or to the technology simply because part of the decision-making process has been automated.
There are obviously multiple groups involved. Security needs to understand the controls and risks around the system, compliance needs to evaluate regulatory requirements, the business owner needs to understand how the AI is affecting the underlying process, and the vendor may have responsibilities around the technology itself.
But there still needs to be a clearly accountable owner for every system in production. Without that, it becomes too easy for responsibility to get distributed across several teams without anyone actually owning the outcome.
Why is the CISO the one talking about this? Is post-deployment AI oversight a security problem or an operations problem?
It is both. The CISO is in the conversation because a live AI system can break the same properties security already owns: confidentiality, integrity and availability.
Confidentiality fails when prompts, retrieved documents or tool outputs leave their intended scope. Integrity fails when an output, a decision or the record of what the system did cannot be trusted. A wrong claim result or a silently changed tool action is an integrity failure, not just a bad answer. Availability fails when a bad model or a stuck workflow takes a business process down.
The controls for those failures are also familiar to security teams: access controls, hard scope boundaries, monitoring, logging, incident response and a tested way to stop and restore the system.
That does not make post-deployment oversight only a security problem. If AI is embedded in policy administration, claims handling or endorsements, a bad output can also become an operational and compliance failure. The people who own those processes need to understand what the system is allowed to do, recognize when the output is wrong and know when to intervene.
Security owns whether the system can be trusted, contained, reconstructed and stopped. Operations owns whether it is being used correctly inside the workflow. Neither can cover the other half after deployment.
Most agencies and MGAs buy AI from vendors rather than build it. What should they ask a vendor about monitoring and change control before signing?
Organizations need to understand the controls around the system just as well as they understand what the technology is capable of doing. In my experience, AI deployments are often held up by missing guardrails rather than the model itself.
I would want to know whether there is an end-to-end audit trail of the inputs, outputs and human review so a decision can be reconstructed later. I would also ask how the vendor monitors behavior in production, what visibility the customer has into that monitoring and what happens when the underlying model, prompts or connected systems change.
Change control is particularly important. Organizations should understand who tests those changes, how they are notified and whether a change can reach their workflow without additional review. They should also understand the system’s scope boundaries, whether there is a tested kill switch and rollback process, the liability language and how their data is retained and used.
Those are not secondary implementation questions. They are part of determining whether the system can be governed responsibly once it is in production.
With state regulators adopting the NAIC AI bulletin, how much production monitoring do you think they'll expect, and how ready are most insurance operations for that?
I would expect regulators to look for evidence that an organization’s AI governance program is actually operating in practice, not simply that a policy exists.
That means being able to show how systems are monitored once they are in production, what thresholds have been established, who owns the system and what happens when one of those thresholds is crossed. Organizations should also be able to produce an audit trail and explain how they investigate and correct problems when they occur.
The larger point is that governance cannot stop at deployment. Models need to be continuously evaluated against production data, with documented tolerance thresholds and a clearly accountable owner. Bias, hallucinations and drift are not issues that get addressed once during development and then disappear.
Sean Kennedy is CISO at Patra. Connect with him on LinkedIn.