How to Evaluate an AEO Agency Before You Sign Anything

The hardest thing to judge about an AEO agency is whether its reporting means anything, because AI answers change so much from one run to the next that a single screenshot proves nothing. The bottom line: before you sign, get answers in writing on how the agency measures results, what it will change on your site, what it will do off your site, and how you can leave. This article gives you eight questions, what a good answer contains, the red flags, and a 150 run test you can perform yourself in about an hour.

See also: answer engine optimization, explained step by step.

under 1 in 100chance two AI responses return the same brand list, per SparkToro and Gumshoe

2,961runs across ChatGPT, Claude, and Google AI in that study

70%chance a Google AI Overview changes between observations, per Ahrefs

87%of the time AI Mode and AI Overviews cite different sources, per Ahrefs

Why measurement is the whole game

In January 2026, Rand Fishkin of SparkToro and Patrick O'Donnell of Gumshoe published the largest public test of AI recommendation consistency. About 600 volunteers ran 12 recommendation prompts, covering categories from chef's knives to cancer care hospitals to cloud computing providers, through ChatGPT, Claude, and Google's AI features a combined 2,961 times, with each prompt repeated 60 to 100 times per platform. There was less than a 1 in 100 chance that two responses returned the same list of brands, and roughly a 1 in 1,000 chance of the same list in the same order. Fishkin's conclusion was that any tool reporting your ranking position in AI is reporting noise. What held up better was how often a brand appeared across many runs, especially in tight categories with a few dominant providers.

Ahrefs found the same instability on the Google side. Its research puts the chance that an AI Overview changes between observations at about 70 percent, and separate Ahrefs research reported by Search Engine Journal found AI Mode and AI Overviews cite different sources 87 percent of the time for the same query. The consequence is simple. If an agency's proof is a screenshot of your business being recommended, or a claim that you now rank third in ChatGPT, ask what it would have shown on the next run.

Eight questions to ask, and what a good answer contains

1. How many prompts, how many runs, which platforms, and how often?

A good answer gives numbers. The SparkToro study needed 60 to 100 runs per prompt per platform to see stable patterns. An agency that tracks 10 prompts once a month is sampling, not measuring. Ask for the count and the schedule in writing.

2. Who chooses the prompts, and are they based on how my customers actually ask?

Prompts should come from real customer questions, sales calls, and search data, and you should approve the list. Semrush's analysis of ChatGPT clickstream found that between 65 and 85 percent of prompts do not match any traditional search keyword, so a keyword list is a poor substitute for real questions.

3. Will you report share of appearance instead of ranking position?

The right metric is the percentage of runs in which your business is named, per platform, alongside named competitors, tracked over time. Be wary of any report built on position in an AI answer.

4. What will you change on my site in the first 30 days?

A real technical scope is specific: whether AI search bots such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot are allowed in robots.txt, whether page content exists in the raw HTML instead of being rendered by JavaScript, whether structured data describes the business accurately, and whether pages answer specific questions. You can check the first two yourself in a few minutes.

5. What off site work is included, and how do you earn it?

AI systems often cite pages that do not rank well in Google. Ahrefs' August 2025 test of 15,000 long tail queries found on average only 12 percent of assistant cited links appeared in Google's top 10, according to a summary by ELMO. Ask about review sites, directories, roundup placements, and forum presence, and ask who does the work. Community posts should be written by a real person following each forum's rules, since spam gets removed.

6. What do I own if I leave?

Listings, accounts, content, and tracking data should be yours. Ask whether directory listings are submitted under your accounts and whether you receive the raw run data, not only a dashboard.

7. What is the pricing model, and what triggers a scope change?

Omniscient Digital found that only 4 of 16 growth agencies it reviewed publish a rate on their own site, so expect a sales conversation. Published guides put most AEO programs between $3,000 and $15,000 a month, with quotes for identical scope ranging from $2,000 to $50,000. Compare proposals on deliverables and measurement, not the monthly number.

8. What is your time to first evidence, and your time to results?

Omniscient describes two clocks: ramp time, until work ships, and results time, until it moves a number. Some agencies say outcomes mature at nine to twelve months. That is acceptable if it is stated plainly. An agency promising fast results on a slow compounding channel deserves a follow up question, and so does one that quotes a 60 to 90 day window without saying what it will show in that window.

Red flags

A guarantee of citations, rankings, or recommendations. No one controls what a language model outputs, and the SparkToro data shows the outputs vary run to run.
Ranking position in ChatGPT as the headline metric.
A scope that only touches your website. On site work is the smaller part of what earns a citation.
An llms.txt file sold as the fix. BlueGrid Media states it is not yet a confirmed ranking factor for any single engine.
Case studies with no metric definition. Ask what was counted, over what dates, and against what baseline. Agency reported figures, such as NoGood's reported 23 times year over year lift in AI search traffic for one client, are self reported.
A price with no hours or deliverables behind it.

A 150 run test you can do yourself

You do not need software to check the basics. This takes about an hour.

Write down 10 questions a real customer would type or say when they are ready to buy, including your city or service area if you are a local business.
Open a fresh session, not logged in to personalized history if possible, in ChatGPT, Gemini, and Perplexity.
Run each question five times per platform. That is 150 runs.
Log one thing per run: yes or no, was your business named.
Calculate your share, the number of yes runs divided by 150, and the share for each platform.
Log the same for your two closest competitors.
Save this as your baseline and ask the agency to reproduce it, and later to beat it, using the same questions.

If an agency's own baseline differs sharply from yours, the difference is a useful conversation: ask which prompts, how many runs, and when.

Contract terms worth asking for

A month by month plan for the first 90 days, with the deliverables due each month.
An exit right if those deliverables are missed, or a short initial term followed by month to month.
Ownership of accounts, listings, content, and raw tracking data.
A named contact and the hours or deliverable counts behind every line item.

Questions people ask

How do I know if an AEO agency is legitimate?

A legitimate agency can state its prompt list, runs per prompt, platforms, and schedule in writing, reports the share of runs that name you rather than a ranking position, and breaks its price into hours or deliverables.

Why can't an AEO agency guarantee results?

AI systems produce different answers on nearly every run. SparkToro and Gumshoe found less than a 1 in 100 chance that two responses returned the same list of brands, so no one can guarantee a specific recommendation.

How can I test an AEO agency's claims myself?

Write 10 real customer questions, run each five times on ChatGPT, Gemini, and Perplexity, and record how often your business is named. Compare that share to what the agency reports.

How much should an AEO agency cost?

Published guides put most programs between $3,000 and $15,000 a month, with wide variation by scope. Compare proposals on deliverables and measurement rather than the monthly figure.