Before You Trust an AI Visibility Score, Make the Tool Show Its Work

An AI visibility score is easy to produce. A decision you can defend is harder.

If a tool says your company has a score of 42, the number creates urgency—but it does not tell you what happened. Which buyer question was tested? Which AI experience answered? When did the check run? Was your company absent, cited as a source, or mentioned in the answer? Without that evidence, a score is presentation, not diagnosis.

That distinction matters for a marketing leader deciding whether to change content, fix site access, add structured data, or simply retest a volatile result. Before you trust an AI visibility tool, make it show its work.

The score should be the summary, not the evidence

Traditional rank reports usually preserve the keyword, search engine, location, date, position, and landing page. AI-search measurement needs an equally inspectable record, even though the result is different.

At minimum, a useful check should preserve:

• the exact buyer question;
• the AI system or search experience tested;
• the date and time of the check;
• the answer or enough of it to audit the finding;
• whether the business was named, linked, cited, or absent;
• which other businesses or sources appeared; and
• the limits of what one check can establish.

A composite score can help summarize many checks. It should never erase the query-level evidence underneath it.

Separate four questions that vendors often blur together

1. Can the system discover the page?

Discovery is a technical question. For ChatGPT search, OpenAI tells publishers that public pages can appear and that sites should not block OAI-SearchBot if they want content included in summaries and snippets. OpenAI also distinguishes OAI-SearchBot from GPTBot, which is used for training controls. Those are concrete checks a site owner can make; neither is a promise that a particular answer will mention the business.

2. Can a search engine understand the facts?

Structured data can provide explicit clues about what a page describes. Google recommends JSON-LD in many situations because it is generally easier to implement and maintain. But Google's documentation frames structured data as a way to help Search understand content and make pages eligible for supported appearances—not as a guaranteed placement mechanism.

This is where marketing claims often run ahead of the evidence. Valid schema can be useful while the business remains absent from a particular AI answer. Both things can be true.

3. Did the business appear for this question?

This is an observation, not a forecast. The tool should show the question and the result. If it names a competitor, that competitor should actually appear in the recorded output. If it reports an absence, the record should make clear which system and moment produced that absence.

4. Will a change cause future recommendations?

That is a causal claim, and it deserves a much higher evidence bar. A before-and-after change can be encouraging, but other inputs may also have changed. Google itself recommends before-and-after testing when measuring structured-data effects and notes that page traffic can vary for other reasons. A responsible tool should help you run and document tests without claiming control over an external AI platform.

Seven questions to ask before paying for an AI visibility tool

1. Can I see the exact query? If not, you cannot judge whether the test represents a real buying situation.
2. Can I see the underlying result? A score without evidence cannot be audited by your team or client.
3. Does the tool distinguish mentions, citations, links, and recommendations? Those are different outcomes with different commercial meaning.
4. Does every result include a system and timestamp? AI answers can change; the record should say what was observed and when.
5. Can I control the question set? Generic prompts may flatter a dashboard while missing the questions that influence your buyers.
6. Does the tool explain uncertainty? One query is a directional signal. Repeated checks across a defined set are more useful for trend analysis.
7. Does it connect findings to work you can actually perform? Crawl access, factual page content, structured data, internal linking, and buyer-question coverage are actionable. Guaranteed placement is not.

A practical first test

Start with one decision-stage question a real customer might ask without using your brand name. For an ecommerce company, that might be a product-comparison question constrained by use case. For an insurer, it might concern coverage for a specific type of business. For an agency, it might ask which provider fits a particular budget or technical need.

Record the answer, the companies and sources that appear, and the date. Then inspect the pages that plausibly support those answers. Look for concrete differences: clear product or service facts, accessible pages, credible third-party references, useful comparison content, and structured information that matches what a person can see.

Do not turn the first result into a sweeping conclusion. Use it to form a better question and a controlled next action.

Our position: evidence first, optimization second

AI-search optimization is young enough that confident dashboards can outrun what anyone can prove. The Federal Trade Commission's guidance is a useful standard for vendors and buyers alike: claims about what an AI product can do should be supported by evidence.

For an AI visibility tool, that means the product should preserve the evidence behind its output, describe the scope accurately, and avoid turning correlation into a guarantee. Buyers should demand the same discipline they would expect from analytics, SEO, or paid-media reporting.

A score may help you scan a dashboard. The underlying observation is what helps you decide.

Start with evidence: Run one real buyer question with AI SearchMagiq's free visibility checker, then use the result as a directional starting point—not a promise.