How to Evaluate an SEO Case Study Critically
How to evaluate an SEO case study critically: a 7-step checklist covering timeframes, algorithm updates, and the questions to ask before you trust it.
To evaluate an SEO case study critically, check five things before you believe a single number in it: the time period covered, whether a Google core update happened during that window, whether the results are shown in absolute numbers or vague percentages, whether other marketing channels changed at the same time, and whether the starting point (site age, existing backlinks, domain history) is comparable to your own business. Most case studies fail at least two of these checks. That doesn’t mean the agency is lying. It usually means the case study was written to sell, not to inform, and the two goals pull in different directions.
Why Most SEO Case Studies Don’t Prove What They Claim
A case study showing “340% traffic growth in 6 months” sounds like proof. It’s actually a correlation with the causation left for you to assume. Traffic can grow for reasons that have nothing to do with the SEO work being shown off: a product launch, a PR mention, a seasonal spike, a competitor going out of business, or Google’s algorithm shifting in the site’s favor independent of anything the agency did.
Search Engine Journal’s own guidance on this is blunt: rankings improving after a change proves nothing in isolation unless you know what else changed, whether there was any kind of control comparison, and how long the observation window was. Most published SEO case studies don’t answer any of those three questions, because answering them honestly would make the result look far less dramatic.
There’s a timing problem too. Google confirmed three core updates in 2025, landing in March, June, and December, on top of what Google’s own Danny Sullivan has described as roughly nine algorithm changes a day across the year. A six-month case study almost certainly overlaps with at least one confirmed core update. If the case study doesn’t mention that, either the agency didn’t check, or they checked and left it out because it complicates the story.
The Checklist: How to Evaluate Any SEO Case Study
Work through these in order. Each one takes under a minute once you know what to look for.
- Find the exact date range. “Over six months” is not a date range. “March 1 to September 1, 2025” is. If the agency won’t give you specific dates, that’s already informative.
- Cross-check against Google’s confirmed update calendar. If a core update landed inside the reporting window, ask how they isolated its effect from their own work. Most won’t have an answer.
- Look for absolute numbers, not just percentages. “200% increase” from 50 visits to 150 visits is technically true and practically meaningless. Ask for the actual traffic or ranking numbers, not just the multiplier.
- Ask what else changed in that period. New product line, paid ads launched, a press mention, a Reddit thread going viral, a competitor’s site going down. Any of these can move organic numbers without touching SEO at all.
- Check the starting point. A five-year-old domain with existing backlinks improves faster than a brand-new site. If the case study doesn’t mention domain age or starting authority, ask directly.
- Confirm the metric matches a business outcome. Traffic and rankings are inputs. Leads, calls, and revenue are outputs. A case study that only shows traffic charts is telling you half the story, possibly the less important half.
- Ask for a reference, not just a screenshot. A screenshot can be cropped to hide a plateau or a dip. A client willing to speak to you directly is much harder to stage.
- What was the exact date range, and did a core update land inside it?
- What were the actual numbers before the multiplier or percentage was applied?
- What else was this client doing in marketing during the same period?
- What was the site’s domain age and backlink profile at the start?
- Can I speak to this client directly, not just see a testimonial quote?
What a Credible Case Study Looks Like Versus a Weak One
| Element | Credible case study | Weak case study |
|---|---|---|
| Timeframe | Specific start and end dates | Vague (“over 6 months”) |
| Numbers | Absolute figures plus percentage | Percentage only |
| Business context | Mentions other marketing activity in the window | Implies SEO was the only variable |
| Starting point | States domain age and prior SEO history | No mention of where the site started |
| Outcome metric | Leads, calls, or revenue tied to the traffic gain | Traffic or rankings only |
| Verification | Client available for a reference call | Anonymized or unverifiable client name |
None of this means every case study missing a detail is fabricated. Plenty of genuinely good agencies write thin case studies because marketing copy tends toward brevity, not because the underlying work was weak. The point of the checklist isn’t to catch liars. It’s to stop you from making a six-figure annual decision based on a chart with no axis labels.
There’s also a bidirectional trap worth knowing about. Pages that already rank well get more visibility, more visibility earns more natural links, and more links reinforce the ranking further. A case study showing “we built links and rankings improved” might have the causation backwards: the page could have started climbing for unrelated reasons and only then attracted the links being credited for the climb. This is a known problem in SEO measurement, not a hypothetical one, and it’s a big part of why serious practitioners are cautious about single-variable success stories.
Questions to Ask the Agency Directly
Once you’ve read the case study, take the conversation further. Our guide on questions to ask a marketing agency covers the broader vetting conversation, but for case studies specifically, ask:
- “Can you show me the Search Console data behind this chart, not just the summary graphic?”
- “Was there a core update during this period, and how did you account for it?”
- “What was this client’s monthly retainer, and how does that compare to what you’re proposing for us?”
- “What’s a case study you’d consider a failure, and what did you learn from it?”
That last question is the most revealing one on the list. Every agency that’s been doing this for more than a couple of years has had a client where results were slow or the strategy needed a mid-course correction. An agency that claims a 100% success rate across every case study they’ve ever run is either extremely lucky or being selective about which stories they tell.
Red Flags That Should End the Conversation
Some patterns aren’t worth a follow-up question. They’re worth walking away over:
- The case study shows a graph with no y-axis labels or numbers at all
- The agency refuses to name the industry or business type, making the result impossible to sanity-check against your own market
- Every single case study on their site shows dramatic, uniform growth with no variation, no plateau, and no setback
- The agency can’t or won’t explain what specific actions produced the result, only that “we did SEO and it worked”
If you’re vetting an agency more broadly and want the fuller list of contract and proposal warning signs beyond just case studies, see our breakdown of SEO proposal red flags and our guide to why guaranteed rankings are a red flag in themselves. For a structured way to compare agencies side by side before you commit budget, our piece on how to choose an SEO agency walks through the full evaluation process, and a transparent SEO retainer should be willing to show you real, dated Search Console data rather than a polished slide.
Frequently Asked Questions
Is it reasonable to ask an agency for their raw data, not just a case study slide?
Yes. A credible agency can export Search Console or analytics data with dates intact in a few minutes. If they resist or stall on a direct data request, treat that as a signal, not an inconvenience.
How much traffic growth is realistic to expect from SEO in six months?
It varies enormously by starting point and competition, which is exactly why single-number claims are misleading. Google’s own guidance is that meaningful results typically take four months to a year, so a six-month case study is often showing early-stage movement, not the full picture.
Should I dismiss a case study just because it doesn’t mention a Google update?
Not automatically, but ask about it directly. Three core updates landed in 2025 alone, so any case study spanning several months has a reasonable chance of overlapping with one. A good agency will have an answer ready.
What if the agency’s case studies are all anonymized?
Anonymization for confidentiality reasons is common and not automatically a red flag, especially for larger clients with NDAs. Ask instead for a reference call or a live walkthrough of the anonymized account’s actual data.
Are percentage-based results ever trustworthy on their own?
Rarely useful without the base number. A 500% increase from 10 visitors to 60 visitors and a 500% increase from 10,000 to 60,000 are very different achievements, and only one of them likely matters to your business.
