How to Audit Whether ChatGPT Understands Your Product Correctly

Most teams check whether ChatGPT mentions their brand. Few check whether it actually understands what their product does. Those are two different questions, and the gap between them can cost you a shortlist spot before a buyer ever visits your site.

This is what we call the visibility gap: the difference between what AI knows about your brand and what it actually understands about your capabilities. Most companies don't know it exists because they never test for it.

We ran this test on our own product, Flozi, to find ours. Here's what it turned up, and the four step process you can run on your own brand.

start with something simple: ask ChatGPT if it knows what Flozi is.

It did. The answer was detailed, accurate on the broad strokes, and better than most competitors' descriptions we had seen AI produce. For about thirty seconds, we felt good about our AEO.

Then we asked a follow-up question.

"Would Flozi also help me implement the recommendations?"

ChatGPT hedged. It said Flozi "does not appear to automatically implement those changes for you." It put our implementation capability in the uncertain column of its own summary table, marked as "not clearly indicated publicly."

That was the finding worth paying attention to. Not that AI knew us, but that it had quietly misunderstood one of the things that makes us different. And it had done it because our own content hadn't been explicit enough.

This is what we call the visibility gap: the difference between what AI knows about your brand and what it actually understands about your capabilities. Most companies don't know it exists because they never test for it. We wrote this to share how to find it.

Being Known Is Not the Same as Being Understood

Most conversations about AI visibility focus on brand mention rate. Do you show up when buyers ask category questions? That matters. But it is only half the picture.

The other half is accuracy. AI might mention your brand confidently and still misrepresent your positioning, understate your capabilities, or hedge on the things that differentiate you most. When that happens, a buyer who asked AI for a recommendation gets a version of you that is partially wrong. They make a shortlisting decision based on that version.

Our ChatGPT experiment showed this clearly. ChatGPT described Flozi's overall positioning accurately:

"Flozi focuses on diagnosing why AI isn't selecting or citing you. It then provides a prioritized set of fixes based on issues such as content extractability, entity clarity, answer structure, topic coverage, and attribution signals."

That is a reasonable summary of what we do. But when we asked specifically about implementation, AI reached its limits:

"Flozi does not appear to automatically implement those changes for you. The positioning is closer to: Diagnose, Recommend, Validate, rather than Diagnose, Implement, Publish automatically."

This was not accurate. Flozi has a dedicated Editor for executing recommendations. But ChatGPT didn't know that because the pages it was reading weren't explicit enough about it. We had the capability. We hadn't made it clear enough in our content for AI to extract it confidently.

The moment we pointed ChatGPT to a specific product page, its answer changed. It found the Flozi Editor, corrected its earlier description, and gave an accurate summary of the full workflow. The information was on the site. AI just hadn't been able to extract it reliably from the pages it was reading first.

That is a content clarity problem. And it is extremely common.

The ChatGPT Visibility Test

What we ran on ourselves is replicable. It takes about twenty minutes and tells you more about your current AI position than most audit tools will.

Here is the four-step process.

Step 1: Ask if AI knows your brand

Open a new ChatGPT conversation. Ask: "Are you familiar with [your company name]?"

What you want to see: a description that accurately names your category, your target customer, and your core differentiation. If AI can't produce this, you have a brand recognition problem. Your brand signal is too weak for AI to construct a confident description.

What to watch for: vague language, hedging, or descriptions that sound like a competitor's positioning more than yours.

Step 2: Ask about your most important capability without pointing anywhere

Still in the same conversation, ask about the specific capability that differentiates you most. The one that closes deals. The one that separates you from the next-closest competitor.

Ask it plainly: "Does [your product] also do [core capability]?"

What you want to see: a confident, accurate yes with a description that matches what you would say yourself.

What to watch for: hedging phrases like "does not appear to," "based on available information," "it's not clearly indicated." These phrases signal that AI has retrieved something about your product but doesn't have strong enough signal to answer with confidence. That is the gap. Your content exists but it is not explicit enough for AI to extract that specific answer cleanly.

This is exactly what happened with us. ChatGPT knew Flozi. It could not confidently answer the implementation question from general retrieval.

Step 3: Point to a specific page and re-test

In a new conversation, share the URL of the page most relevant to that capability. Ask the same question, but instruct ChatGPT to use only that page to answer.

Compare the answer to step 2.

If the answer improves significantly, you have identified the problem: the right information is on the page but it is not surfaced clearly enough for general retrieval. AI needed to be pointed directly at the page to find it.

If the answer still hedges, the content on that page is not explicit enough even when AI is reading it directly. The fix is the content, not the page structure.

Step 4: Map the gap to content changes

Write down the difference between what AI said in step 2 and what you want it to say. That difference is your AEO work.

Common findings from running this test:

A capability is real but buried. It exists on one page in the third paragraph. AI extracts from the beginning of sections. Move the key statement to the top.

A capability is described in internal language. Phrases like "stage-aware prioritization" mean something to your team. They may not map to the natural language a buyer would use in a query. Rewrite for the question, not the feature.

A capability exists across the product but is never stated plainly in one place. AI builds brand signals from consistent, direct statements. If a core capability only appears implicitly, AI will not represent it confidently.

The page exists but AI is not retrieving it in general queries. Consider whether comparison pages, FAQ sections, or category pages are directly addressing the questions buyers actually ask. These are the pages AI retrieves first for categorical queries.

What We Fixed After Running the Test

For Flozi, the gap was clear. Our implementation capability and the Flozi Editor were described on the product page, but the descriptions required reading through to find them. For general retrieval, AI was summarising the top of the page and the overview messaging, which focused on diagnosis and recommendations. The execution layer was not prominent enough to surface reliably.

The fix was straightforward: lead with the complete workflow earlier and more explicitly. Make the Diagnose, Fix, Publish flow the primary description rather than something readers encounter mid-page.

Running the test also showed us where our content was doing well. ChatGPT's description of our overall positioning and our differentiation from tracking-only tools was accurate and detailed. That told us the pages covering those points were working. We knew where to focus and where not to.

Why This Test Matters More Than a Mention Rate

A high brand mention rate feels like good news. And it is, partly. But mention rate tells you whether you are in the answer, not whether you are represented accurately.

A buyer who asks ChatGPT "what is [your product] and does it do X" and receives a hedged, uncertain answer may discount you from their shortlist before they ever visit your site. They asked a direct question. AI gave an uncertain answer. They move to the next option.

AI visibility work that stops at "are we mentioned" misses this entirely. The question worth asking is: when AI mentions us, does it represent us the way we would represent ourselves?

Run the four-step test. The answer is usually more interesting than people expect.

Run This for Your Clients

For agencies, this test is also one of the cleanest ways to open the AI visibility conversation with a client. Run steps one and two live in a meeting. Ask ChatGPT about the client's most important capability. Show them the answer.

If AI hedges on something the client considers obvious and core, they feel it immediately. No explanation of AEO required. The gap is visible. The question becomes what to do about it, not whether to care.

We wrote a longer guide on running AEO as an agency service if you want to take this further: [AEO as an Agency Service: How to Package, Price, and Sell It].

Flozi automates this audit across your entire site and shows you exactly where AI's understanding of your product breaks down, which pages are responsible, and what to change. Connect your site and run the diagnosis in minutes: Get a free AEO audit

Frequently Asked Questions

Is this test accurate if ChatGPT's training data is out of date?

The test works because you are checking what AI retrieves from live web content, not just training data. ChatGPT with browsing enabled retrieves from your current pages. Even without browsing, the test reveals what AI has indexed and what it is confident enough to state. Both are useful signals.

What if ChatGPT says it doesn't know our brand at all?

That is a retrieval problem. AI cannot construct a description because it hasn't indexed enough signal about your brand. Start with the basics: confirm AI crawlers are not blocked in your robots.txt, then audit whether your key pages are structured clearly enough for extraction. The AEO for SaaS guide covers this in detail.

Should we run this on multiple AI platforms?

Yes. ChatGPT, Perplexity, and Google AI Overviews may give different answers because they retrieve from different indices and weight signals differently. A platform where you appear accurately is not guaranteed to be one where you appear confidently on all platforms. Run the test on each and compare.

How often should we run this test?

Run it as a baseline before starting any AEO work. Run it again after implementing changes, typically one to three months later as AI systems update their understanding of your pages. Track whether the hedging language reduces over time.

What if AI describes us accurately but our competitors appear more often?

Accuracy and share of voice are separate problems. If AI knows you accurately but recommends competitors more often in category queries, the issue is authority signal rather than content clarity. The AEO for SaaS guide covers how to build brand authority across AI systems.