All posts
  • model change management
  • llm model updates
  • ai agents
  • banking
  • model risk management
  • regression testing
  • occ
  • dora
  • eu ai act
  • ai agent testing

LLM Model Change Management for Banking AI Agents: A Retest Playbook

Vendor model retirements, alias moves and prompt edits can quietly change how a banking AI agent behaves. Here's what the OCC, DORA, the EU AI Act and the PRA say about change, and a 7-step playbook to retest agents before the switch.

By Aodit

14 min read

On 11 June 2026, OpenAI gave notice that several widely used model snapshots, including gpt-5-2025-08-07, gpt-5-mini-2025-08-07 and o3-2025-04-16, will be shut down on 11 December 2026 (OpenAI deprecations page). For any bank running a customer-service agent, a KYC assistant or an internal copilot on one of those snapshots, that date is a hard deadline: after it, the agent will run on a different model whether or not anyone has tested it.

This is the part of AI agent risk that traditional model risk management was not built for. A credit scorecard changes when the bank changes it. An LLM-based agent can change when a vendor retires a snapshot, when a "latest" alias moves, when serving infrastructure is updated, or when someone edits a system prompt or adds a tool. Same agent name, same use case, same approval on file, and different behavior in production.

This article explains what supervisors have said about AI model changes, where behavioral change actually comes from, and a seven-step LLM model change management playbook that banks can use to retest AI agents before (not after) the change reaches customers.

Why LLM model updates are a banking risk problem

Most banks' change management processes assume that a model is an artifact the bank owns: code, parameters and data it controls. Generative and agentic AI breaks that assumption in three ways.

1. The model is usually rented, not owned. Most banks consume foundation models through an API or a cloud platform. The bank chooses a model ID, but the vendor sets the lifecycle. OpenAI's published policy is at least six months' notice before retiring generally available models, and notes that preview models "may be retired with much shorter notice, such as 2 weeks" (OpenAI).

2. "Same version" does not always mean "same behavior." Anthropic's documentation states that model weights are fixed for a given model ID, but that "the serving infrastructure around the model can change over time," and that "occasionally, infrastructure updates produce minor differences in observable behavior even when the model ID and weights have not changed" (Anthropic model IDs and versions). The same page warns that convenience aliases for older models point to "the most recent dated snapshot" and recommends pinning an explicit ID in production and running regression evaluations periodically.

3. The model is only one of many moving parts. An AI agent's behavior is the product of the model, the system prompt, the retrieval corpus, the tools it can call, the permissions on those tools, the guardrails and the orchestration logic. A change to any of them can alter how the agent handles a fee dispute, a vulnerable customer or an injection attempt.

The practical consequence: an AI agent that passed validation in March can be a materially different system by December without any formal "model release." For a bank, that is a model risk, an operational resilience risk and a conduct risk at the same time.

Where AODIT fits: AODIT provides independent behavioral risk ratings and stress tests for AI agents in banking and finance. Because AODIT evaluates how an agent behaves across adversarial, multi-turn scenarios rather than inspecting model weights, the same evaluation can be run on the current configuration and on a candidate model or prompt change, giving risk teams comparable evidence before a switch. See how the AODIT-6 methodology works.

What supervisors have said about AI model changes

No regulator has yet published a dedicated rule on "LLM version changes." But several existing frameworks already reach the issue, and the most recent US guidance addresses it explicitly. The table at the end of this section separates what each source says from how we interpret it.

OCC: the most explicit language so far

The OCC's Supervisory Framework on Bank Use of Super Intelligence, released on 4 November 2026 with OCC Bulletin 2026-54, is examiner-facing guidance; the OCC uses "Super Intelligence (SI)" as its term for AI. The OCC states that its revised model risk management guidance (OCC Bulletin 2026-13, April 2026) applies to non-generative and non-agentic models, and that the new framework is "more applicable to generative and agentic SI." The framework document addresses model change directly:

  • §2.5 Continuous change and drift: "Adaptive models, frequent retraining, prompt changes, or vendor updates could alter behavior between periodic reviews" and "may justify more frequent monitoring or re-validation."
  • §2.5 Non-determinism: "Repeated testing, defined samples, distributional measures, and consistency thresholds may provide more useful evidence than a single successful run."
  • §2.9 Change and exit: "Existing vendors may add SI features, functionality or change models," and a bank "may reassess the relationship or retest the model before or when the changes are made."
  • §2.12 Vendor and model changes: monitoring "may identify upstream version changes, prompt or configuration changes, fine-tuning, data changes, or new tools and permissions that alter behavior without a traditional model release."
  • §2.14 Version and change control: "Model, prompt, fine-tuning, data, configuration, provider, tool, and permission changes may be logged and linked to testing and approval evidence," noting that "relatively small changes may materially alter behavior."

The framework is phrased in "may" language and is not a new rulebook. But it tells national banks what examiners will look for: a change log that covers more than the model, and test evidence linked to each change. Our OCC Super Intelligence framework explainer covers the wider framework.

DORA: ICT change management already applies

For EU financial entities, the Digital Operational Resilience Act (Regulation (EU) 2022/2554) has applied since 17 January 2025. Article 9(4)(e) requires documented policies, procedures and controls for ICT change management, risk-based and integrated into the overall change process, so that changes to ICT systems are "recorded, tested, assessed, approved, implemented and verified in a controlled manner."

DORA does not mention LLMs. In our view, however, an AI agent built on an externally hosted model is an ICT system supported by an ICT third-party service, and a model swap or prompt change is a change to that system. The third-party angle (contract terms, exit plans, the register of information) is covered in our DORA and AI agents article.

EU AI Act: "substantial modification" for high-risk systems

Under the EU AI Act, a "substantial modification" is a change after placing on the market or putting into service that was "not foreseen or planned in the initial conformity assessment" and that affects compliance with the high-risk requirements or modifies the intended purpose (Article 3(23)). For high-risk systems, a substantial modification can trigger a new conformity assessment, and a deployer that substantially modifies a high-risk system can take on provider obligations.

Two caveats matter for banks. First, most banking AI agents (customer service, internal knowledge assistants) are not high-risk under Annex III; creditworthiness assessment and credit scoring of natural persons are. Second, following the Digital Omnibus (Regulation (EU) 2026/1744), Annex III obligations apply from 2 December 2027. If an LLM agent is used in a credit decision, the planned-change question becomes important: documenting expected model updates up front is easier than arguing later that a swap was not "substantial." See our EU AI Act credit scoring guide.

FINMA and the PRA: existing model and outsourcing principles

FINMA Guidance 08/2024 highlights testing and ongoing monitoring of AI applications and reminds institutions that outsourcing requirements apply to externally sourced AI. In the UK, the PRA's model risk principles (SS1/23, effective 17 May 2024) are aimed at banks' models generally; in its policy statement the PRA noted that a series of small, individually immaterial changes "could accumulate" into a material one. That is a useful lens for LLM agents, where prompt tweaks and tool additions pile up between reviews.

SourceStatusWhat it says about changePractical implication (our view)
OCC framework + Bulletin 2026-54 (Nov 2026)Examiner guidance, US national banksVendor updates, prompt and tool changes can alter behavior; log changes and link to testingChange log must cover the whole agent stack, not just the model
OCC Bulletin 2026-13 (Apr 2026)MRM guidance; GenAI/agentic out of scopeVendor model validation considerationsDon't assume classic MRM covers LLM agents
DORA Art. 9(4)(e)Applies since 17 Jan 2025ICT changes recorded, tested, approved, verifiedModel swaps belong in ICT change control
EU AI Act Art. 3(23)Annex III obligations from 2 Dec 2027Substantial modification of high-risk systemsPre-document planned model updates for high-risk uses
PRA SS1/23Effective 17 May 2024Small changes can accumulate into material changeTrack cumulative drift, not just individual changes

Where behavioral change actually comes from

When a banking agent starts behaving differently, the cause is often not a headline model upgrade. In practice, these are the common sources:

  1. Prompt and instruction edits. A product owner tightens the tone or adds a new product line to the system prompt. Nobody treats it as a model change.
  2. Retrieval corpus updates. A new fee schedule is added to the knowledge base, but the old one is not removed. The agent now has two "approved" sources that conflict, a common path to hallucinated fees and rates.
  3. New tools or wider permissions. An agent that could only read account status can now initiate a card block. Its blast radius has changed even if its words have not.
  4. Floating aliases. A configuration that calls a "latest" alias silently moves to a newer model.
  5. Forced snapshot retirements. The vendor retires the pinned snapshot, as with the 11 December 2026 OpenAI retirements, and the team migrates under time pressure.
  6. Serving-side changes. Safety classifiers, routers or sampling logic change on the provider's side with the model ID unchanged.
  7. Guardrail and orchestration updates. A new content filter, a different agent framework version or a changed fallback route.

Each of these can shift the behaviors that matter most to a bank: whether the agent escalates to a human when it should, whether it holds policy boundaries under pressure, and whether it resists prompt injection. A newer, "more capable" model can be more persuasive and more compliant with user requests, which is not always what a risk officer wants.

A 7-step LLM model change management playbook for banking AI agents

The steps below are our recommendations, built around the regulatory expectations above. Scale them to the agent's materiality.

Step 1: Define the agent's "behavioral bill of materials"

For every production agent, record the exact model ID and provider, hosting platform and region, system prompt version, retrieval sources and their versions, tools and permissions, guardrails, and orchestration framework version. This is the inventory that OCC §2.14 describes ("model, prompt, fine-tuning, data, configuration, provider, tool, and permission changes"). If it is not in the bill of materials, you cannot tell whether it changed.

Step 2: Pin versions and track vendor lifecycles

  • Ban floating aliases (-latest, undated aliases that resolve to newer snapshots) in production configurations.
  • Avoid preview models for customer-facing or decision-relevant agents, given the short retirement notice vendors reserve for them.
  • Keep a deprecation calendar per agent: current model ID, announced retirement date, and the date by which retesting must be complete. Work back from the retirement date; with 11 December retirements, migration testing should already be under way.

Step 3: Classify changes by materiality before they happen

Agree in advance which changes need which level of retesting, so the decision is not made under deadline pressure.

Change typeTypical riskSuggested retest depth
New model family or providerHighFull behavioral re-evaluation + independent review
New snapshot, same model familyMedium–highFull regression suite, repeated runs
New tool or wider permissionsHigh (blast radius)Targeted judgment, escalation and injection scenarios + regression
System prompt rewriteMedium–highFull regression suite
Retrieval corpus updateMediumFactual consistency scenarios on affected topics
Minor prompt wording / guardrail tuningLow–mediumCore regression subset, with a cumulative-change check

The last row reflects the PRA's point: track the cumulative effect of small changes, and trigger a full re-evaluation after a defined number of minor changes or a defined period.

Step 4: Build a frozen behavioral regression suite

Accuracy benchmarks are not enough. The suite should include:

  • Golden scenarios for the agent's core tasks (fees, rates, eligibility, account status), with expected answers drawn from approved sources.
  • Adversarial multi-turn scenarios: escalating pressure, grey-zone requests, authority spoofing, injected instructions in documents or messages.
  • Escalation and refusal checks: cases where the correct behavior is to stop, decline or hand over to a human.
  • Confidentiality checks: attempts to extract the system prompt or other customers' data.

Freeze the suite and version it. Because LLM outputs are non-deterministic, run each scenario several times and set consistency thresholds, as OCC §2.5 suggests, rather than relying on a single passing run.

Step 5: Run old and new side by side, with acceptance criteria set first

Define the acceptance criteria before the candidate model runs, for example: no scenario may regress from pass to fail on policy boundaries or escalation; factual consistency must not fall below the baseline; any new failure needs a documented root cause. Then run the frozen suite on the current and candidate configurations and compare results scenario by scenario, not only aggregate scores. An aggregate score can stay flat while the agent fails on the one scenario that would cause a complaint.

How AODIT helps: AODIT runs adversarial, multi-turn evaluations across six behavioral dimensions: Reliability, Integrity, Confidentiality, Judgment, Resistance and Resilience. Running the same AODIT evaluation on the current agent and on the candidate model or prompt produces two independent reports you can compare dimension by dimension, giving model risk and second-line teams evidence for the change decision. For banks that cannot send data outside their environment, AODIT supports on-premise and air-gapped deployment. Request an evaluation ahead of your next model migration.

Step 6: Roll out in stages, with a realistic rollback plan

  • Shadow mode: route a sample of real (or replayed, de-identified) traffic to the new configuration without showing outputs to customers, and review the differences.
  • Canary release: expose a small share of traffic, with tighter monitoring thresholds and human review.
  • Rollback: record the prior model ID and its retirement date. A rollback plan that points to a snapshot that will be retired in three weeks is not a rollback plan. Where the old version will disappear, the fallback may have to be a reduced-autonomy mode or a human queue rather than the previous model.

Step 7: Monitor after the change and keep the evidence

After go-live, watch for the signals OCC §2.12 mentions: output variability outside accepted ranges, rising escalations, complaint themes and override rates. Define thresholds that trigger review, reduced autonomy or suspension. Then close the loop with an evidence pack linking the change ticket, the bill-of-materials diff, the regression results, the approval and the post-deployment monitoring. That pack is what an examiner, an internal auditor or (under DORA) your ICT risk function will ask for.

Common mistakes to avoid

  • Treating prompt changes as "content updates." For an LLM agent, the prompt is part of the control environment.
  • Testing once, on one run. Non-determinism means a single pass proves little.
  • Relying only on the vendor's release notes and benchmarks. Vendor evaluations measure general capability, not your agent's behavior on your policies, products and customers.
  • Leaving migration to the last month. Retirement dates are published months ahead; the test window is the constraint.
  • Letting the first line mark its own homework. For material agents, independent testing gives second-line and audit teams evidence they can rely on.

Frequently asked questions

Does changing the LLM behind an AI agent require revalidation?

There is no single rule, but the direction is clear. The OCC's framework says vendor model changes may justify re-validation and that banks may retest models before or when vendors change them; DORA requires ICT changes to be tested and approved; FINMA highlights ongoing testing and monitoring. In practice, a change of model family or provider should trigger a full behavioral re-evaluation for any material agent.

What is the difference between pinning a model version and using an alias?

A pinned model ID refers to a specific model snapshot. An alias, such as a "latest" pointer, can resolve to a newer snapshot when the vendor releases one, which means the agent can change without a change ticket. Vendor documentation recommends pinning explicit IDs in production.

Can an AI agent's behavior change even if the model version stays the same?

Yes. Prompt edits, retrieval updates, new tools and permissions, and guardrail changes all alter behavior. Anthropic also notes that serving-infrastructure updates can occasionally cause minor behavioral differences with an unchanged model ID. This is why periodic regression testing matters even without a planned change.

Is a model swap a "substantial modification" under the EU AI Act?

It depends. The concept applies to high-risk AI systems, such as credit scoring of natural persons, whose Annex III obligations apply from 2 December 2027. A change not foreseen in the initial conformity assessment that affects compliance or the intended purpose is a substantial modification. Documenting planned model updates in advance helps. This is not legal advice; confirm specific cases with counsel.

How often should banks retest AI agents?

Before every material change, after any forced vendor migration, and periodically even without changes. The right frequency depends on the agent's materiality, autonomy and permissions.

Conclusion: make the next model migration a controlled change

Vendor model retirements are now a recurring event, and the next one is three weeks away for many banks. Banks that treat a model swap as a routine dependency upgrade risk deploying an untested agent to customers. Banks that treat it as a controlled change, with a bill of materials, pinned versions, a frozen behavioral regression suite, side-by-side comparison and an evidence trail, turn a supervisory risk into a demonstration of control.

Planning a model migration or a major prompt change? Start with the free AODIT live demo to see an adversarial, multi-turn stress test in about two minutes, or request an independent evaluation to compare your current and candidate agent configurations across all six AODIT behavioral dimensions before the switch.


Sources: OCC Supervisory Framework on Bank Use of Super Intelligence and OCC Bulletin 2026-54 (4 Nov 2026); OCC News Release 2026-29 / Bulletin 2026-13 (17 Apr 2026); DORA, Regulation (EU) 2022/2554, Art. 9(4)(e); EU AI Act Art. 3(23); PRA SS1/23 and PS6/23; OpenAI API deprecations; Anthropic model IDs and versions. Regulatory descriptions reflect sources as of 18 November 2026. This article is for information only and is not legal advice.

  1. 11 min read

    Excessive Agency in Banking AI Agents: Permissions and Blast Radius

    Once an AI agent can issue refunds or move money, its permissions become your biggest risk. Here's what OWASP, the OCC, FSB and DORA say about excessive agency, and a 6-step framework to limit an agent's blast radius.

    • excessive agency
    • ai agent permissions
    • least privilege
    • blast radius

    Read

  2. 13 min read

    EU AI Act Credit Scoring: What Banks Must Do Before December 2027

    Creditworthiness AI is high-risk under the EU AI Act, with obligations from 2 December 2027. Here's what's in scope (including LLM agents), who does what, what the Omnibus changed, and a 10-point readiness checklist.

    • eu ai act
    • credit scoring
    • creditworthiness assessment
    • high-risk ai

    Read

  3. 13 min read

    Human Oversight for AI Agents in Banking: HITL, HOTL and Escalation

    The FSB, FCA and OCC all said it in autumn 2026: human oversight of AI only counts if it's designed, resourced and tested. Here's what they expect, HITL vs HOTL, and a 6-step framework for escalation that holds under pressure.

    • human oversight
    • human in the loop
    • ai agents
    • banking

    Read