Most people don’t know what they don’t know when it comes to AI SEO. That’s not a criticism — the field is young, it’s technical, and the terminology is still being standardized. Which means it’s genuinely easy to walk into an agency evaluation without the right frame for what you’re assessing.
The solution is having the right questions. Not questions designed to trip agencies up or make them feel defensive, but questions that reveal actual depth — that separate agencies who have done this work from agencies who are selling a story about it.
These are those questions.
Before You Even Start: Clarify What You’re Buying
One conversation worth having internally before talking to any agency: what does success look like for you in 18 months? Not vague success — specific success. More organic traffic? Brand mentions in ChatGPT? Better Google AI Overview presence for specific query types? Higher conversion from organic channels?
Agencies can optimize for almost anything if you tell them what you want. The problem is that many clients don’t specify clearly, so agencies default to optimizing for the metrics that are easiest to show improvement on — which may or may not be the ones that actually matter for the business.
Get clear on this first. It makes every conversation with potential agencies sharper and more productive.
Question 1: How Specifically Do You Optimize for AI-Generated Answers?
Not “do you do AI SEO” — that’s too easy to answer vaguely. Ask specifically: walk me through your process for improving a brand’s visibility in ChatGPT answers or Google AI Overviews.
A real answer includes specific tactics: entity optimization and knowledge graph signals, structured data and schema implementation for AI parseability, semantic content architecture that builds topical authority, and some methodology for tracking whether brand citations in AI outputs are actually improving.
A non-answer sounds like: “we use cutting-edge AI technology to enhance your search presence” or “we stay ahead of algorithm changes to maximize your visibility.” These are complete non-answers dressed up in professional language. If this is what you get, keep probing or move on.
Question 2: What’s Your Entity Optimization Process?
Entity optimization is one of the most important and most underdiscussed aspects of AI-era SEO. How a brand is understood as an entity — by Google’s knowledge graph, by the training data and retrieval systems behind LLMs — directly affects whether AI systems cite it as a trusted source.
Ask the agency to walk you through how they approach entity optimization. What do they assess about your current entity profile? What do they build to strengthen it? How do they measure whether entity recognition has improved?
Strong answers involve specific processes: auditing how your brand currently appears in Google’s entity systems, building consistent structured data to define the entity clearly, earning authoritative mentions that strengthen the entity association, and tracking improvements in entity recognition over time.
Weak answers involve generic talk about “brand awareness” and “building your online presence” — which miss the technical specificity that entity optimization requires.
Question 3: How Do You Measure AI Search Visibility?
This is often the most revealing question. Traditional SEO has well-established measurement frameworks — rank tracking, traffic analytics, click-through data. AI SEO doesn’t have the same established measurement infrastructure, which means the agencies doing it seriously have had to develop their own approaches.
Ask: if I want to know whether my brand’s AI search visibility improved this quarter compared to last quarter, what would you show me?
Good answers involve some combination of: systematic sampling of AI-generated responses for target queries to track brand citation rates, tracking AI Overview appearances for relevant keywords, monitoring brand mentions in LLM outputs across platforms, and connecting AI search signals to business outcomes like qualified traffic and conversions.
Answers that only reference traditional rank tracking and organic traffic are telling you something important about the limits of their AI SEO capability.
Question 4: What Did You Get Wrong in the Last 12 Months and What Did You Learn?
This question cuts through prepared sales presentations more effectively than almost any other. Agencies doing serious work in a fast-moving field are constantly running into things that don’t work the way they expected, experimenting with new approaches, and updating their methodology based on what they’re learning.
An agency that can’t point to something they got wrong — that has no story of an approach that didn’t work and what they learned from it — is either not doing serious work or not being honest with you.
The best agencies will give you a thoughtful, specific answer about something they tried that underperformed, what they learned from it, and how it changed their approach. That kind of intellectual honesty is a very good sign.
Question 5: Can I Speak With a Client in a Situation Similar to Mine?
References are standard practice, but the framing matters. Generic references that agencies have on speed dial — typically their happiest, most vocal clients — are less useful than references specifically matched to your situation.
Ask for a client reference who: is in a similar industry, started the engagement at a similar stage of AI SEO maturity, and had similar goals to yours. If the agency can’t produce this, ask why. The answer might be that they’re earlier in their client base development in your sector — which is honest and acceptable. The answer might be that similar clients don’t want to be references — which is worth probing.
Which AI SEO agency to choose ultimately comes down to the evidence available about how they perform for clients like you. A well-matched reference conversation tells you more than any amount of case study documentation.
Question 6: What’s Your Onboarding Process and What Will You Need From Us?
A surprisingly revealing question. Agencies doing serious work need significant input from clients — access to analytics, content review processes, subject matter expertise for content production, technical access for implementation. Agencies doing superficial work often don’t need much because what they’re producing doesn’t require deep client collaboration.
Ask what the first 90 days look like in detail. What will they produce, what will they need from you, what milestones will you hit together? If the answer is vague or if they seem to need very little from you, that’s a signal about the depth of work involved.
Question 7: What Would Make You Recommend Ending the Engagement?
This is a trust question as much as a substance question. Agencies that are genuinely client-focused — that care about doing right by you rather than just about retention — can answer this. They’d end the engagement if: the strategy isn’t working after a reasonable timeline and they’ve exhausted their approaches; if the client’s business fundamentals aren’t amenable to organic search; if a client’s internal capacity makes implementation impossible.
Agencies that hedge completely on this, that can’t articulate any scenario where they’d tell you the engagement isn’t the right fit, are often optimizing for retention rather than client outcomes.
Using These Questions Well
These questions aren’t a gotcha mechanism. They’re a tool for having honest, substantive conversations that get both parties — you and the agency — to clarity faster. AI SEO agency reviews from peers in your industry are valuable too, but direct conversation with proper questions produces the clearest signal.
The right agency will welcome these questions. They’ll see them as evidence that you’re a serious client who wants a real partnership. And the conversations you have will give you a much clearer picture of who’s actually equipped to help you than any amount of website browsing or proposal reading.
ThatWare regularly engages with this kind of rigorous client evaluation — it reflects the depth of work they do. Their team at https://thatware.co/best-ai-seo-agency/ can speak specifically to all of these questions.
Good agencies want good clients. Good clients ask good questions. The match between them is where the best work gets done.
