Every Model Upgrade Is a Migration: Harness Checklist

Questions this guide answers

How should marketing teams handle AI model upgrades?
How do you test a new LLM before switching a marketing workflow to it?
What is harness engineering?
Should I pin an LLM model version or use the latest alias?
How do you build an eval set for a marketing AI workflow?
What should I ask an AEO vendor about model changes?

Direct answer

Treat every model change as a migration, even a new version of a model you already use: freeze real examples from your own work, decide in advance what the new model must pass, run it alongside the old one before it ships anything, and keep switching back to a one-line change. Keep what should not change—your approved facts, output formats, and eval set—outside the model, so the model becomes the replaceable part.

The cadence makes this routine: between July 9 and Sept. 22, 2026, Anthropic, Google, and OpenAI shipped more than 10 general-purpose models, 3 of them on Sept. 22 alone, and Anthropic says Claude Sonnet 5.5 and Haiku 5.5 "will follow in the coming weeks."

Why I'm writing this

Eason's year-one essay described what we built as the models kept changing under the product. This note is the method behind it: what we freeze, what we gate, and what we refuse to swap. It is written for anyone who now owns a harness, whether or not they call it that: marketing-ops and AI-ops leads, technical marketers who maintain Skills and MCP connections, and CTOs deciding whether a vendor's numbers will survive its next model change.

In Action Externalization I argued that when doing gets cheap, deciding gets expensive. A model upgrade is a small, recurring case of that, and my position is simple: the model is a replaceable part. Your context and your evals are not.

The release cadence, dated

These are the general-purpose model releases from Anthropic, Google, and OpenAI in the 11 weeks from July 9 to Sept. 22, 2026, leaving out restricted security variants, each checked against the vendor's announcement or dated coverage.

Date (2026)	Lab	Release
July 9	OpenAI	GPT-5.6, in 3 variants: Luna, Terra, and Sol
July 21	Google	Gemini 3.6 Flash and 3.5 Flash-Lite
July 24	Anthropic	Claude Opus 5
Aug. 13	Google	Gemini 3.7 Flash, "just three weeks after Gemini 3.6 Flash"
Sept. 1	Anthropic	Claude Fable 5.1
Sept. 2	Google	Gemini 3.8 Flash, "our third Flash release in only six weeks"
Sept. 3	OpenAI	GPT-6 Astra
Sept. 22	Anthropic and OpenAI	Claude Opus 5.5 (Sonnet 5.5 and Haiku 5.5 "will follow in the coming weeks"); OpenAI's GPT-6 Sol and GPT-6 Luna

What the dates mean

Each row is a different model, not a faster copy of the last one. Staying put is not a permanent option either. Anthropic's deprecation policy says requests to retired models "will fail," with at least 60 days' notice for publicly released models, and Claude Opus 4.1 was retired on Aug. 5, 2026.

Capability moves on a similar clock. METR's Time Horizon 1.1 update (Jan. 29, 2026) estimates that the length of task, in human-expert time, at which frontier models succeed half the time doubled about every 131 days for models since 2023, and about every 89 days for models since 2024. METR's tasks are mostly software, machine learning, and cybersecurity, and it calls the trend "somewhat sensitive to task composition," so read it as a direction, not a forecast for your copywriting. The direction is enough: whatever your workflow assumes a model cannot do has a short shelf life.

What a harness is, and why it goes stale

A harness is everything around the model that decides what it reads, what it may do, and how its work gets checked. OpenAI's Feb. 11, 2026, post on harness engineering put the division of labor as "Humans steer. Agents execute." and the team's primary job as "to design environments, specify intent, and build feedback loops that allow Codex agents to do reliable work." Anthropic's context engineering essay (Sept. 29, 2025) covered the reading half: "the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome."

On April 8, 2026, Anthropic named the maintenance problem: "Harnesses encode assumptions that go stale as models improve." Its example: Claude Sonnet 4.5 tended to wrap up tasks early as it sensed its context limit approaching, so Anthropic added context resets to the harness. On Claude Opus 4.5 the behavior was gone, and "The resets had become dead weight."

For a marketing team, the harness is concrete: the Skill folders and system prompts, the MCP connections an assistant can call, the brand facts and examples it reads, the templates its output must fit, and the person who approves what ships. Agent Skills became an open standard on Dec. 18, 2025, so one Skill folder can run in several clients and meet several models. If your team built Skills or agent automations this year, you built a harness. The question is whether anyone wrote down what it assumes about the model.

What breaks when the model changes

In my experience, a model change rarely announces itself with an error. It shows up as output that looks fine and is subtly different. I check these first:

Format drift. A preamble appears, fields get renamed, headings shift, answers get longer, and whatever parses or templates the output downstream breaks quietly.
Interface changes. Parameters get deprecated. Anthropic's documentation says that on Claude 4.7 and later models, setting temperature, top_p, or top_k to a non-default value returns a 400 error.
Stale workarounds. Instructions that compensated for an old model's weakness become dead weight, and they can over-steer a model that no longer has it.
Scores that change scale. Model-generated scores for lead fit, content quality, or relevance are often ordered but not calibrated. A new model's 70 is not the old model's 70, so every fixed threshold quietly changes meaning.
Silent swaps. OpenAI's chat-latest alias "points to the latest Instant model currently used in ChatGPT," and "The underlying model snapshot will be regularly updated." Consumer apps and agent workspaces change defaults on their own schedules.
Forced moves. A retired model stops answering, on the vendor's deadline, not yours.

What should change on an upgrade, and what shouldn't

The fastest way to make upgrades boring is to decide, in writing, which layer is allowed to move.

Layer	On a model upgrade	Owner
The model	Replaceable, behind a gate you control	Workflow owner
Prompts, Skills, and harness code	Revised on purpose and versioned with the model they were tested on	Workflow owner
Output contracts: format, required fields, banned claims	Stable; changed only deliberately, with downstream users told	Workflow owner and whoever consumes the output
Eval set and gold labels	Frozen; grows deliberately, never edited to fit a new model	Workflow owner, plus an independent second labeler
Approved facts and claims	Unchanged; they change when the business changes	Product marketing, with legal for restricted claims
A model that measures something over time	Not swapped mid-series	Measurement owner
Human approval before anything is published	Unchanged	A named reviewer

The model is the replaceable part

Everything below the second row is what a marketing team accumulates: the formats its systems depend on, the examples it trusts, the facts it stands behind, the instruments it reports from, and the people accountable for what ships. The top row is an input that keeps getting better and cheaper on the vendor's schedule. A team whose facts and evals live inside its prompts rebuilds both on every upgrade; a team that keeps them outside the model can take the upgrade on a Tuesday. Eason made the marketing version of this argument in Production is cheap, trust is scarce; this is the same idea with a test suite attached.

How we run a model change at SolCrys

This is what we do. Eason's essay gave the outline; this is the detail behind it. None of it is exotic; the discipline is not skipping steps when a new model looks obviously better.

Gold labels we can defend

We freeze real outputs into a fixed set, so every comparison runs on identical inputs and a result can be reproduced. The answer key comes from 2 independent labeling passes, run by strong models (on our newer sets, from 2 different model families), that follow a written guide and never see what our pipeline produced, so the gold is not anchored to the system being graded. We measure agreement between the passes with Krippendorff's alpha, and 0.67 is our release bar. When agreement falls short, the rubric is ambiguous, and we fix the guide rather than the data. Only material disagreements go to adjudication, a third model pass that sees both labels and must cite the rubric in its rationale.

We keep 2 kinds of eval apart. A regression lock tells you output changed, not whether it is right; an accuracy eval grades against the independent gold set. Gold is expensive to build, so we reserve it for numbers that drive decisions.

Gates that name the costly error

A new model has to pass gates before it takes over a job, and a gate is not an average: it names the error the product can't afford. The eval behind Answer Accuracy hard-fails when the judge misses a clear error in an AI answer and reports false alarms without blocking, because for that feature a miss is the expensive mistake.

Two more rules keep gates honest. We grade the real thing: the production prompt, model, and parser, not a copy rewritten for the test. And we re-baseline on purpose. Updating the expected results to match whatever a new model produced is how a regression gets laundered into the baseline.

Swappable models, fixed instruments

Model calls go through a provider-neutral layer, so moving a job to a different model or provider is a configuration change rather than a rewrite of every call site, and key structured outputs have schema contract tests, so a model that renames a field fails a test instead of breaking a report. One place is exempt from that flexibility: we never substitute the model that queries an answer engine, because that model is part of the measuring instrument, and changing it would change what is being measured.

Rank, don't threshold

When a model's scores are ordered but not calibrated, we rank instead of cutting at a fixed number, and a model's self-reported confidence is a label, not a gate. Eason's essay has the Signals example, where a fixed threshold threw away most of the best items. The general rule matters more than the example: a threshold is a calibration claim, and a new model voids it.

Facts that live outside the model

Answer Accuracy grades each AI answer against a company's entire Corporate Context and cannot mark a claim wrong without pointing to the evidence. That design is what lets it survive a model change: the judge can be upgraded, and the facts it grades against do not move with it.

The checklist for your own AI workflows

This is what I would hand a marketing team running its own agents, whether that means Skills in Claude, an agent working through your files in Claude Cowork, or an automation calling a model API. None of it requires SolCrys.

Step	What to do	What it catches
1. Inventory	List every place a model makes a decision: the Skill or tool, the model name, whether it's pinned, and the owner.	The workflow nobody knew depended on a model
2. Freeze an eval set	Save a few dozen real inputs per job, including ones that went wrong. Never edit them to suit a new model.	Comparisons run on flattering examples
3. Label blind, twice	Two people, or two strong models from different families, write expected outputs from a written guide without seeing any model's answer. Check agreement; adjudicate disagreements.	A gold set that quietly agrees with whatever you already ship
4. Write gates before testing	Make the costly error a hard fail, and set a floor per metric, not for the average.	An upgrade that is better on average and worse where it matters
5. Write output contracts	Define format, required fields, length, citation style, and banned claims; validate every output mechanically.	Format drift that breaks templates downstream
6. Pin the model	Call a fixed snapshot where the API allows it; otherwise record the model name and date on every output.	Silent swaps
7. Shadow run	Run the candidate beside the current model on live inputs for 1–2 weeks; ship only the current output and review the differences.	Failures your frozen set didn't contain
8. Keep rollback to one line	Version prompts and Skills with the model they were tested on, so reverting is a configuration change.	A bad upgrade that takes a week to undo
9. Remove dead weight	After a switch, delete workarounds the new model doesn't need, then re-run the gates.	Instructions that over-steer a better model
10. Don't swap the instrument	If a model measures something you report over time, don't change it mid-series; run both in parallel and re-baseline on purpose.	A trend line that moved because the ruler changed

Pin, or at least record

Pinning means calling a model name that won't change underneath you, and vendors handle it differently. Anthropic's models overview says every Claude model ID "is a pinned snapshot, including the dateless IDs used from the 4.6 generation on." OpenAI says "Snapshots let you lock in a specific version of the model so that performance and behavior remain consistent," and also offers aliases that move. Inside consumer apps and agent workspaces, the vendor decides which models are offered and when old ones leave the menu. That is the best argument for the frozen eval set: it works whether or not the vendor lets you hold still.

Shadow before you switch

A frozen set catches the failures you already know about. A shadow run catches the ones you don't: the candidate processes the same live inputs as the current model, its output goes to a review queue instead of production, and someone reads the differences. Treat the diff, not the average, as the result, and add every new failure case to the eval set.

Keep a person on the write path

Agents can now act, not just draft. Keep write permissions narrow and granted on purpose, keep a named person approving anything that ships under your brand, and expect that person to read most closely right after an upgrade.

An illustrative migration

Illustrative scenario only. A B2B observability company's content team runs a Skill that turns engineering release notes into a changelog entry and an FAQ update. A new model version ships, and a teammate says its drafts read better.

The frozen set holds 40 past release notes with gold entries written in 2 blind passes and adjudicated. The gates, written before anyone ran the new model: 0 dropped breaking-change notices (a hard fail), 100% valid output for the CMS template, and no more than a 5-point drop on a readability rubric.

The candidate wins on readability, drops the breaking-change notice on 3 of 40 notes, and emits a heading the CMS template rejects on 2. It fails. The team adds one line to the output contract and a schema check, and the re-run passes. A 2-week shadow run surfaces one new case, a release note with no user-facing change, which becomes item 41. The switch is one line, and so is the rollback.

The numbers are invented; the lesson isn't. "Reads better" was true, and it still wasn't a reason to switch.

What to ask an AEO vendor about model changes

If a vendor's product runs models on your behalf, ours included, these questions separate a governed system from a demo. Our methodology checklist covers what happens when an answer engine changes its model; these cover the models inside the product.

When you change the model behind a feature, what does it have to pass first, and can you show me the gate?
Is the model that queries the answer engines ever changed? If so, how do you keep my trend line comparable?
How were your gold labels made: how many independent passes, what agreement bar, and who adjudicated?
Are your scores calibrated, or only ordered? What happens to a threshold when the model changes?
Where do my approved facts live: in prompts, or in a store the model reads?
What can an agent write through your product, who grants that permission, and who approves what ships?

How SolCrys fits

SolCrys is an AEO platform built as a Measure → Diagnose → Execute → Verify loop, and the method above keeps that loop steady while the models underneath it change. Three parts of the product map to this post:

Corporate Context is the stable layer. Your approved facts, claims, and guardrails live at the organization level, not in prompts. Answer Accuracy grades AI answers against them, and recommendations are grounded in them rather than in a model's guesses.
Measurement holds its instrument fixed. The same frozen prompt set runs on the same engines (ChatGPT, Google AI Overviews, Gemini, Perplexity, and Claude, by plan), and the model that queries an engine is not substituted. The measurement methodology covers the rest.
Your agents get typed tools and open Skills. The SolCrys MCP server works with Claude, ChatGPT, and Cursor, and our Skills are MIT-licensed on GitHub, so you can fork them, version them with the model you tested, and run this checklist on them. Writes are narrow and opt-in: with a permission an admin grants, an assistant can mark a task published or, where enabled, propose a Corporate Context draft that a person must approve.

What SolCrys doesn't do

SolCrys doesn't run evals on your own agent workflows, host your eval sets, or choose which model your AI clients use. It never publishes to your site: approval in the Action Hub is a recorded step, and your team ships every change through its own process. And a before-and-after movement in AI answers is an observation, not proof that your change caused it.

To see the loop on your own brand, Start Free: a free workspace with 10 ChatGPT prompts and 1 content audit, email verification, and no credit card. To go deeper on how we gate model changes, talk to us.

About the author

Jia Chang is Co-Founder & CTO of SolCrys. AI architect with 15+ years building production AI systems, most recently as an engineering leader at Microsoft. Connect on LinkedIn.

Written September 2026 as the method companion to Eason's A Year in View. For the architecture underneath, see 6 architectural decisions; for why agents reach SolCrys through MCP, see why we bet on MCP early.

Sources

FAQ

How should marketing teams handle AI model upgrades?

As migrations, not settings changes. Freeze a set of real examples from your own work, write gates before you test (including a hard fail on the error you can't afford), shadow the new model on live inputs before it ships anything, and keep rollback to a one-line change. Keep approved facts, output formats, and the eval set outside the model so they survive the switch.

What is harness engineering?

Harness engineering is designing everything around a model that decides what it reads, what it may do, and how its work is checked: instructions, tools, context, tests, and review. OpenAI used the term in a Feb. 11, 2026, post about building software with agents, summed up as "Humans steer. Agents execute." Anthropic added the maintenance warning on April 8, 2026: "Harnesses encode assumptions that go stale as models improve."

How do you test a new LLM before switching a marketing workflow to it?

Run it on a frozen eval set of real inputs from that workflow, graded against expected outputs written blind by 2 independent labelers and adjudicated where they disagree. Apply gates you wrote before testing, validate every output against your format contract, then run the candidate in shadow on live inputs before it replaces the current model.

Should I pin a model version or use the latest alias?

Pin in production wherever the API allows it, and upgrade on purpose after the new model passes your gates. Anthropic says every Claude model ID is a pinned snapshot; OpenAI offers snapshots and also moving aliases such as chat-latest, whose underlying snapshot it says "will be regularly updated." Where you can't pin, record the model name and date on every output and lean harder on the frozen eval set.

What is a shadow run?

A period in which a candidate model processes the same live inputs as the current model, but only the current model's output ships. Someone reviews the differences. It catches failures your frozen eval set didn't contain, and each new case it finds should be added to the eval set.

Why shouldn't an AEO platform change the model that queries AI engines?

Because that model is part of the measuring instrument. If it changes, a shift in your visibility trend could come from the new model rather than from the engine or your content. SolCrys never substitutes the model that queries an engine for that reason.

What does "rank, don't threshold" mean?

When a model's scores put items in the right order but the numbers aren't calibrated, a fixed cutoff throws away good items and keeps weak ones, and it silently changes meaning when the model changes. Ranking the items and reviewing from the top keeps the useful part of the score, the order, without trusting its scale.

Does SolCrys test the models in my own AI workflows?

No. SolCrys gates the models inside its own product and keeps its measurement instrument fixed. Through the MCP server and open-source Skills, your assistants can read your SolCrys data and Corporate Context, but evaluating and pinning the models in your own workflows stays with your team. The checklist on this page is how we would do it.

Free ChatGPT visibility check

See where AI answers skip your brand — then fix it, free

Start a free workspace with your domain: 10 buyer-intent prompts through ChatGPT show where you are mentioned, cited, or skipped, and who gets recommended instead. A free content audit in the same workspace hands you the first fix to ship.

Start Free

Free · No credit card · About 5 minutes

Related guides

How SolCrys Works

Building an AEO Platform: 6 Architectural Decisions

An AEO platform's measurement, action, and verification surfaces all rest on a small number of architectural choices that aren't usually published. Jia Chang on six of ours — what each tradeoff looks like, what we chose, and what we'd revisit.

MCP & Skills

Why we bet on MCP early: agent-readiness as the new AEO surface

Jia Chang, SolCrys CTO, on why we built our MCP server before customers asked for it, why we open-sourced the Skills layer on top of it, and what it signals about the next surface marketing teams need to optimize for.