Models change. The harness compounds.

The asymmetry

When a better model ships, the workers get better and cheaper while the instruments and judges hold still, so a customer's history stays comparable. A product built as a wrapper around one model has to rebuild on each release. A harness absorbs the release. That asymmetry is the design.

Instruments measure what buyers get

An answer engine is a product, not just a model, and we treat it that way. Each engine is queried through its provider's own interface, with that engine's own search and citation behavior. We decided early not to route engine calls through a multi-model gateway, because a gateway normalizes search and citations, and that would quietly change the thing we measure.

Two rules follow. The first is to send the question the way a buyer would type it. Until September, every prompt we measured on ChatGPT and Claude was wrapped in a 73-word instruction to cite every claim and prefer deep links. No buyer types that. On Sept. 14 we removed it and recorded the date, knowing some series would cite fewer sources from that day on. The second rule is to measure rates, not ranks. The same question returns different answers from run to run, so an instrument is only meaningful across repeated runs, and our core metrics are rates across runs rather than a position in one answer.

Judges hold still

A judge is part of the ruler. If a customer's Answer Accuracy moves, it should be because an engine said something different, not because we changed who grades it. So each judge is pinned to an exact model version on its own setting, where a routine change to the rest of the fleet can't reach it, and moving one is a decision someone makes deliberately, with the gate report attached.

This is the hardest rule in the portfolio to keep. Cheaper judges keep arriving, and gating a judge is harder than gating a worker. Three things we learned from our own judge gates:

• At small sample sizes, a fail is more trustworthy than a pass. A run of agreeing verdicts proves little. One missed clear error proves a lot.
• An easy eval set measures easiness, not equivalence. Any two capable judges agree on obvious cases. The signal is in the cases near the line.
• Agreement with the incumbent is its own test. A new judge can score better against the gold set and still grade differently from the old one, and for a score with a history, grading differently is the problem.

Workers ride the curve

Everything else routes through one registry that both our Python and TypeScript runtimes read. Each job is assigned a tier, each tier is a ranked list of models, and the lists span more than one model family, so bringing in a new family is a registry change, not a rewrite of every call site.

Our routing policy sets the rule for workers: cost within a quality budget, meaning the cheapest model whose results on our own evals stay within 10% of the current model's, with a hard floor on output validity. When our highest-volume job, extracting brands and citations from answers, moved to a model from a second family in August, the candidate first ran against the same 200-answer gold set the incumbent was graded on. It scored higher (F1 of 0.87 against 0.83) and every output parsed. Another candidate failed on the validity floor alone: 7.5% of its outputs didn't parse.

Every routed job also keeps a fallback to a pinned model, and one switch sends the whole fleet back to direct calls.

The same door is open to a model we train ourselves. If we fine-tune one for a narrow, high-volume job, it will enter the way every other worker does: as a candidate in a tier, graded on the same gold sets, with the same fallback behind it.

Models read; code scores

The most important rule in the portfolio isn't about which model to use. It's about where a model is allowed to decide.

Wherever a number reaches a customer, we try to limit the model to reading and quoting, and leave the scoring to code. Recommendation Score works this way: a model extracts how an answer positions a brand and quotes the answer word for word, the server checks that each quote actually appears in the answer and demotes verdicts it can't ground, and a deterministic scorer, with no model call, turns that evidence into a 0–100 score. Answer Accuracy follows the same pattern. To fail an answer, the judge must quote both the company's fact and the answer's words, and code discards any failure whose quotes it can't find. Deep Analysis reports pass a validator before anyone sees them: every evidence quote must appear in the answer it cites, and a recommendation that proposes a manipulation tactic or targets a competitor's own page is dropped.

The Answer Accuracy rule came from a real miss. The judge failed a correct answer about a business's parent affiliation because the business's context didn't mention any affiliation at all. Silence isn't contradiction, but nothing enforced that until every failure needed two verbatim quotes. Silence has nothing to quote.

The payoff is structural: the less of a score a model decides, the less of it a model change can move.

The language models that read are interchangeable. The scoring is ours. Recommendation Score's scorer, the content audit's 25 deterministic checks, and our ranking and share-of-voice math are code we wrote: deterministic, versioned, and pinned by tests, so they don't change when a language model does.

The prompt set that scored 100 and found no brands

Every workspace is measured against a Golden Prompt Set: the questions buyers in that category ask AI. Our first eval for it scored form: conciseness, balance across prompt types, duplicates, natural phrasing. A generated set could score 99.8 to 100 out of 100 and still contain no question whose answer would name a brand. The eval was measuring the wrong thing.

The scenario that mattered was ambiguity. One company that makes special-purpose trucks received prompts about special-purpose vehicles in finance: bankruptcy remoteness and asset securitization. The words matched; the subject didn't. So we now score prompts on the property that decides whether they're worth measuring: does the question seek a solution, the kind of question whose answer names brands, or does it slip into a definition? A reference set across 5 verticals, from AI chips to security cameras, sets the bar: at least 75% solution-seeking, and no more than 3% definitional. The classifier that scores definitional drift in the eval also runs inside the generator as a deterministic filter, so removing a definitional question lowers the score by construction rather than by hope.

The relevance judge that learned "subject, not vocabulary"

In September, a scan for an industrial HVAC software company drafted replies to threads that shared its vocabulary and nothing else, a car-buying listicle among them. We built the eval from that failure: 88 real threads, 22 from each of 4 test workspaces chosen to be as unlike each other as possible (industrial HVAC software, B2B marketing software, high-performance networking hardware, and consumer security cameras). Two blind labeling passes, from two different model families, scored each thread's subject fit from 0 to 100, under a guide whose first rule is to score the subject, never the shared words. They agreed at an alpha of 0.956.

At the production threshold, the gate cut off-topic drafts from 45 to 5, and recall on the threads actually worth a reply rose from 0.56 to 0.93. The per-company view is the point: the HVAC workspace went from 18 off-topic drafts to 1, and the camera workspace still had 4. An aggregate score would have hidden both.

The verifier that must never flatter

Recommendation Score subtracts for drawbacks an answer raises about a brand: a caveat, a concern, a warning. Some of what the extractor flags as a drawback isn't one, so we built a verifier to review them, and gated it before it could change anyone's score.

That makes the verifier the component that would be easiest to get wrong in the customer's favor. The simplest way to make every score look better would be a verifier that dismisses criticism generously. So its gate is lopsided on purpose: on a set of 94 drawbacks from live test audits, each labeled real or false, a verifier fails if it erases or downgrades a single real one. How many false drawbacks it removes is reported and doesn't block.

That scenario set changed two decisions. Judging drawbacks without the quoted text erased 19 of 41 real ones, so the verifier is always given the grounded quote. And when we compared models, the larger one removed more false drawbacks but erased a real concern, while the smaller one erased none. We chose the smaller one.

In a product that sells a number, the error that flatters the customer is the one to fear most.

Every failure becomes a scenario

These examples share a shape. Something failed in production, we wrote the failure down as a scenario with a gate, and the gate now runs before the next change to that job. When a news-source classifier presented a company's homepage teaser as a new article with an invented date, the fix shipped with an eval that refuses to release if a single major publication is mistaken for a company's own site. When the accuracy judge read silence as contradiction, the judge eval began running the same evidence check as production.

That's why I think of the eval sets as the product specification. The scenario and its gate exist before we choose a model for the job, and the unit of progress is a new scenario, not a new prompt.

What our evals don't do yet

Three limits, so no one mistakes this for more than it is. Most of our gold labels come from strong models following written guides, not from human annotators; blind passes and an agreement bar keep that honest, but they don't turn it into human judgment. Most of our accuracy evals are gates we run deliberately before a change ships, not checks that run on every commit, so their value depends on discipline. And some features have only a regression lock, which tells us output changed but not whether it's right.

Six rules the harness taught us

Each of these started as a failure:

• Silence is not contradiction. A company's context is never a complete list of true facts, and its gaps are worth finding on their own. A judge that fails every claim the context doesn't mention punishes the brand for its own gaps, so every failure now needs two quotes.
• Measure the question as typed. Anything added to a measured prompt is something buyers never see.
• Keep the brand out of the room when writing discovery questions. A generator that knows whose visibility will be measured anchors on that brand, so discovery prompts are kept brand-blind, and brand-named questions are added separately and labeled as such.
• Rank when scores are ordered but not calibrated. In one Signal scan, the model's scores put the right items on top, yet a fixed threshold threw away 3 of the best 4, and the model's self-reported confidence ran backward, correlating at −0.57 with relevance. We now rank, and confidence is a label, not a gate.
• Read the web as data, not instructions. The answers and pages our judges grade are wrapped as untrusted input, so an instruction hidden inside one is treated as content to grade, not a command to follow.
• Log the decision, not just the error. When an output is wrong, the first question is which rule let it through. After one bad run of our community-thread engine took a database session to explain, its logs began naming the decision at each gate: the fit verdicts, the ranked queue with each score, and every draft skipped under the threshold, with the reason. The next question took one search.

A person on the write path

Agents can act now, and we let them, narrowly. Through the SolCrys MCP server, a customer's own assistant can read their workspace. With separate permissions an admin opts into, it can do two things more: mark an Action Hub task as published, and propose a Corporate Context page, which arrives as a draft for a person to review. Each write runs under its own database role, limited to the rows it may touch. Nothing ships under a customer's brand unless someone on their team publishes it, and SolCrys never publishes to a customer's site. In a harness, the write path is where trust is either kept or spent.

Together, they form a loop

A new model enters as a worker candidate. Scenario evals decide whether it's admitted, and to which jobs. The instruments and judges don't move, so the customer's history stays comparable. The harness keeps the output governed, and every new failure becomes another scenario. Each model release makes the system cheaper or better, and none of them forces a rebuild.

Why publishing the method doesn't give it away

A method is easy to copy. A history can't be backfilled. The recipe is public; the labeled scenarios, the comparable history, and the failures behind each gate accrue only with time and real workloads.

What SolCrys doesn't do

SolCrys doesn't publish to your site; your team ships every change through its own process. It doesn't treat a before-and-after movement in AI answers as proof that your change caused it. And not every score has an independent gold set behind it yet. Where one doesn't, we describe that eval as a regression lock, not an accuracy measurement.

To see the loop on your own brand, Start Free: a free workspace with 10 ChatGPT prompts and 1 content audit, email verification, and no credit card. To go deeper on the architecture, talk to us.