SolCrys Logo

Buyer Guides

Six questions to evaluate any AEO platform's data methodology - including ours

We wrote this checklist as the questions we'd want a buyer to send us in writing before signing - and we expect you to send the same questions to every other vendor you're evaluating. Where do prompts come from? Are responses captured from the consumer surface or the API? Which available model or surface signals are queried? How is engine non-determinism handled? Can a single data point be reproduced? What happens when an engine changes its default? We answer all six on this page and link to our methodology pages so you can audit our answers. We expect you to score us against the same standard as anyone else. A vendor (including us) that bristles at any of these is signaling something about how they handle scrutiny.

Published · Updated

Questions this guide answers

  • How do I evaluate an AEO platform's data?
  • What questions should I ask an AI visibility vendor about methodology?
  • How do I know if AEO data is accurate?
  • How does SolCrys answer the methodology questions buyers should ask?

Direct answer

We've watched buyers in this category get sold beautiful dashboards built on weak foundations. The six questions below are the ones we'd want you to ask us - and we'd want you to ask everyone else you're evaluating with exactly the same words: where do prompts come from, are responses captured from the consumer surface or the API, which available model or surface signals are queried, how is engine non-determinism handled, can a single data point be reproduced, and what happens when an engine changes its default?

This page is the data-trust companion to our broader AEO Platform Buyer's Guide. The buyer's guide covers full vendor selection - positioning fit, feature breadth, support, integrations, and agency scenarios. This page zooms into the data-and-methodology dimension specifically, because that's the part most buyers under-investigate before signing. We answer each of the six questions for SolCrys at the end of each section, and we link to the underlying methodology page so you can audit us.

Why we treat methodology as the under-investigated dimension

Buyers comparing AEO platforms typically focus on visible feature parity, engine coverage, and support. The harder question - whether the underlying data is reliable - is easier to wave away with a slide. So most buyers skip it. We think that's a mistake, because two specific failure modes recur in this category.

Synthetic prompts dressed as research: a vendor says 'we track 300 prompts in your category' but the prompts were generated by a language model at indexing time, not grounded in real buyer behavior. The questions real users ask AI assistants are typically much longer and more problem-stated than synthetic SEO-keyword-derived prompts. If the prompts don't reflect real questions, the visibility numbers don't reflect real exposure - so no number on the dashboard means what it claims to.

API-only tracking sold as consumer measurement: a vendor says 'we track ChatGPT' but in practice queries an API endpoint that uses different default models, system prompts, and web-search behavior than the consumer chat product. The two surfaces can diverge enough that fixing one does not move the other - so the buyer ships the wrong fixes for months.

The 6 methodology questions

Each question lists what you are listening for, common red flags, and why it matters.

Question 1: Where do your tracked prompts come from?

What you should listen for: specific data sources, not vague phrases like 'AI-generated based on your category.' Strong answers include some combination of intent volume data, trending community questions from public Q&A platforms, live follow-up questions captured from the engines themselves, and customer-supplied prompts at a transparent ratio.

Red flags: 'We use AI to generate prompts based on your category' (translation: synthetic), vague phrases like 'ensemble deep-learning models on 50+ sources' without naming them, refusal to disclose template-vs-customer ratio, and no mechanism to decline or replace the vendor's default prompts.

How we answer: every Golden Prompt Set is grounded on four real-world signals - intent volume, public community Q&A trending questions, AI query volume signals, and live follow-up questions from supported surfaces where reliable. We reserve a meaningful share of every workspace for prompts you supply yourself, and we'll show you the source provenance for any prompt in your set. Full methodology: see our Golden Prompt Set methodology page.

Question 2: Are responses captured from the consumer surface or the API?

What you should listen for: a clear distinction between API, SERP, and rendered-surface data. Strong answers explain why Google AI Overviews requires SERP capture, why API paths may differ from consumer surfaces, and how each data point is tagged so you can tell what was measured.

Red flags: 'We use the API because it's more reliable' (translation: the data does not match what consumer users see); inability to explain how Google AI Overviews is captured (it has no public API); implicit claim that API and consumer-surface data are interchangeable.

How we answer: we tag engine and capture method on every data point. Supported workspaces use configured engine adapters for ChatGPT, Gemini, Google AI Overviews / AI Mode, and Perplexity based on plan scope. Retail assistants such as Alexa for Shopping and Sparky are scoped as add-ons when access and reliability are validated. Full methodology: see our Visibility Measurement methodology page.

Question 3: Which specific available model or surface signals do you track?

What you should listen for: named models or configured adapters where available, plus a disclosure process when defaults change. Strong answers describe how model signals are stored, how tracking is updated when a provider changes behavior, and where customers can see material changes.

Red flags: 'We track ChatGPT' without specifying the variant or surface; 'the platform decides which model gets used' (data is non-reproducible); no disclosure process for material model or surface changes.

How we answer: we store engine, configured adapter, timestamp, and available model signal with tracked responses. When a provider changes a configured model or surface materially, we update tracking as soon as the change is verified and disclose the change so you can interpret any trend-line discontinuities.

Question 4: How do you handle the fact that AI engines give different answers each time?

What you should listen for: explicit acknowledgement of non-determinism, plus a methodology for it. Strong answers describe repeated capture, daily priority-prompt monitoring, rolling-window aggregates, and an explicit policy for how single snapshots are labeled versus trend movements. Vendors that claim formal confidence intervals should be able to describe the sample sizes that support them; if they cannot, treat that as a marketing label, not a method.

Red flags: 'Our data is highly accurate' without explaining how non-determinism is handled; one run per refresh cycle treated as one data point; no rolling windows or acknowledgement that engines are non-deterministic.

How we answer: active workspaces refresh priority prompts daily and aggregate into rolling 7-day and 30-day windows. Single snapshots are labeled as snapshots, not as trend changes. We do not currently report formal confidence intervals - sample sizes vary too much across prompt sets and engines to put an honest statistical label on the chart - and we'd rather show you the underlying responses than overstate the math. Free Audit reports include a single snapshot for a directional read and we say so.

Question 5: Can a specific data point be reproduced?

What you should listen for: yes, with specifics. The vendor should walk you from any chart to the underlying captured response, including prompt text, engine, available model or surface signal, capture method, timestamp, response, and citations. For teams that need raw data access, the vendor should explain current export options without hand-waving.

Red flags: 'Our methodology is proprietary' (translation: you can't audit it); no drill-down from charts to raw responses; case studies that show only AI response screenshots without capture metadata.

How we answer: every chart in your SolCrys dashboard drills back to the prompt text, engine, available model or surface signal, capture method, timestamp, captured response, and citations where available. If a result ever looks wrong, request the response evidence and we can show what was captured.

Question 6: What happens when an engine changes its default model?

What you should listen for: a documented update process plus disclosure to customers. Strong answers describe ongoing monitoring of provider announcements, an SLA for updating tracking, dashboard surfacing of model changes, and automated alerting so the team does not miss provider updates.

Red flags: 'We use the latest model' without specifying how 'latest' is defined or who watches for changes; no changelog visible to customers; inability to describe what happened the last time a major engine updated its default.

How we answer: we monitor provider announcements and model deprecations on an ongoing basis, with internal alerting so we don't miss provider updates. When a default changes, we update tracking as soon as the change is verified and post the change to our public changelog so you can interpret any discontinuities in your trend lines. The model signal is part of every per-data-point audit trail when the engine discloses it.

How to run the checklist in a vendor evaluation

We treat the six questions as documentation requests, not accusations - and we encourage you to send the same questions to us in writing too.

  • Send the six questions in writing before the call. This filters out anyone who can't answer in writing.
  • Score each answer 0/1/2 (no answer / vague / specific). A vendor below 8/12 isn't yet ready for production buyer work.
  • Ask for documentation on the strongest claims. If we say we disclose model or surface changes, we should be able to show you where that happens.
  • Run a side-by-side test. Pick five prompts in your category, ask each vendor to run them, then run them yourself manually. The vendor's reported responses should align in substance with your manual checks.
  • Read each vendor's methodology pages before the call. If a vendor doesn't have one, that's already an answer.

What we are explicit about not claiming - including for ourselves

For this checklist to be credible, we name what no AEO platform - including SolCrys - can honestly promise.

  • No platform can guarantee citation lift, including us. Engine behavior is influenced by hundreds of inputs no vendor can fully control. The right promise is measurement and recommended actions, not outcomes.
  • No platform fully covers every AI engine. Coverage trade-offs are real. We track the engines our customers' buyers actually use; we don't claim universal coverage and we don't think anyone should.
  • No platform is immune to engine changes. What matters is how quickly we respond and how transparently we disclose updates - and the same standard applies to anyone you're evaluating.

FAQ

If you had to pick one question, which matters most?

Question 5 (reproducibility). If we can't show you the raw captured response behind any chart, none of the other questions matter - every claim becomes unfalsifiable. Reproducibility is the foundation that makes the other methodology answers checkable. If you ask us only one question, ask that one.

How is this different from your AEO Platform Buyer's Guide?

Our AEO Platform Buyer's Guide covers the full vendor selection - positioning fit, feature breadth, support, integrations, and agency scenarios. This page zooms in on the data-and-methodology dimension specifically, because that's the part most buyers underweight in vendor selection but it determines whether the data they rely on is trustworthy.

Should I ask all six questions even if the vendor seems credible?

Yes. The questions are documentation requests, not accusations. A credible vendor (us included) will welcome them and have ready answers. A vendor that bristles at any of them is signaling something about how they handle scrutiny.

What about smaller or newer AEO platforms that haven't published methodology pages yet?

Some early-stage platforms have legitimate methodology that's not yet documented publicly. In that case, ask for a methodology document under NDA. If they can produce one in writing, they pass. If they can't, treat them like any other vendor that won't show their work.

Is there a single failing answer that should disqualify a vendor?

Two: 'The platform decides which model gets used' (data is non-reproducible) and 'Our methodology is proprietary' (methodology that can't be audited can't be trusted). Implementation details can be confidential; the methodology itself should be transparent.

How does SolCrys score on its own checklist?

We publish our prompt-selection methodology (Golden Prompt Set), our measurement methodology, our model/surface disclosure process, our handling of engine non-determinism, and our per-data-point drill-down approach. We've tried to model the standard we ask other vendors to meet. If you find a gap, tell us - we publish this checklist precisely so you can hold us to it.

Related guides

Buyer Guides

AI Visibility Dashboard vs AEO Execution Engine

AEO platforms split into two architectures: dashboards measure and report; execution engines diagnose, fix, and verify. This guide compares 6 use cases, walks through 5 real scenarios, and ends with a decision tree.

How SolCrys Works

Golden Prompt Set Methodology

We ground every AEO prompt set on real intent volume, public community questions, AI query signals, and live engine follow-ups - not synthetic keyword lists. Here's how we build it.

How SolCrys Works

AI Visibility Measurement Methodology

How we capture your AI visibility data across supported engines, with each response traceable to a prompt, engine, capture method, available model or surface signal, and timestamp. Consumer-surface and retail-assistant validation are scoped where technically reliable.

How SolCrys Works

SolCrys FAQ

How we build prompt sets, capture AI visibility data, handle engine non-determinism, run our Free Audit, and document methodology. The 19 questions buyers ask us most often during evaluation.

Free AI visibility audit

Find out where your brand is missing, miscited, or misrepresented.

SolCrys maps high-intent prompts to mentions, citations, answer accuracy, and content gaps so your team can prioritize the next pages to ship.

Get a free audit