Claude and AI Assistants: How Sources Get Selected
Selection happens in two stages: retrieval finds relevant accessible candidates, generation picks what it can confidently cite. Why failing either gate means no citation.
AI assistants select sources in two stages: retrieval narrows the whole web down to a set of candidate passages that are semantically relevant and accessible, then generation picks from those candidates the passages it can most confidently ground and attribute an answer with. Understanding this two-stage filter explains why some pages get cited repeatedly while better-known competitors get skipped — you have to pass both gates, and they reward different things.
Key takeaway
- Selection happens in two stages: retrieval finds relevant accessible candidates, generation picks what it can confidently cite.
- Retrieval rewards access, semantic relevance and clean chunking; generation rewards concrete, sourced, self-contained passages.
- Failing either gate means no citation, which is why access problems and vague prose are equally fatal.
Stage one: retrieval
The system converts the user’s question into a semantic representation and searches its index for passages that mean something similar. To be a candidate at this stage you need three things: to be in the index at all (which requires the relevant search crawler to have been allowed and the page to have been indexed), to be genuinely semantically relevant to the question, and to be chunked cleanly enough that a coherent passage can be isolated. Pages that sprawl across topics without clear section boundaries produce muddled chunks that match nothing precisely.
Two gates
Retrieval decides whether you’re a candidate; generation decides whether you’re cited. Access and relevance get you through the first; concrete, sourced, self-contained substance gets you through the second.
Source — retrieval-augmented generation architecture
Stage two: generation
From the retrieved candidates, the model composes an answer and decides what to attribute. Here the criteria shift. It favours passages that answer the question directly rather than circling it, that stand alone without needing surrounding context, that contain something concrete — a specific figure, a named expert‘s statement, a definite claim — and that carry visible sourcing making the claim verifiable. Between two equally relevant candidates, the one with a sourced statistic and a direct opening beats the one with elegant generalities almost every time.
Why known brands still get skipped
Authority helps but doesn’t decide this. A well-known company whose page buries the answer under three hundred words of positioning, or whose relevant section opens with “This is why it matters,” can lose to a smaller site with a direct, self-contained, sourced passage. The generation stage is evaluating passages, not brands. That’s the practical opening for smaller sites, and the reason ranking and citation have decoupled — only around 17% of AI Overview citations came from organic top-ten results in early 2026, down from roughly 76% in mid-2024.
Optimise for both gates deliberately
Treat them as separate checklists. For retrieval: confirm the search crawlers are allowed at robots.txt and CDN, confirm you’re indexed in both Google and Bing, cover the questions genuinely, and structure content into focused sections that chunk cleanly. For generation: open each section with a direct answer, keep passages self-contained by restating nouns instead of using orphan pronouns, and put a sourced statistic or named quotation on every claim that matters. Both matter equally — flawless content behind a blocked crawler is invisible, and perfect access to vague prose earns nothing.
Frequently asked questions
How do AI assistants choose which sources to cite?
In two stages. Retrieval searches the index for passages semantically relevant to the question, requiring that you’re indexed, genuinely relevant and cleanly chunked. Generation then picks from those candidates the passages it can most confidently ground and attribute an answer with — favouring direct, self-contained content with concrete, visibly-sourced claims.
Why do well-known brands sometimes not get cited?
Because the generation stage evaluates passages, not brands. A known company whose relevant section buries the answer in positioning copy, or opens with an orphan pronoun, can lose to a smaller site with a direct, self-contained, sourced passage. This is why ranking and citation have decoupled, with most AI citations now coming from outside the organic top ten.
What matters most for passing retrieval?
Access and structure. The relevant search crawlers must be allowed at both robots.txt and your CDN, and your pages must actually be indexed in Google and Bing. Beyond that, content needs genuine semantic relevance to real questions, and clear section boundaries so coherent passages can be isolated rather than producing muddled chunks that match nothing precisely.
Can I optimise for one gate and not the other?
No — failing either means no citation. Flawless, quotable content behind a blocked crawler is invisible, and perfect crawler access to vague, unsourced prose earns nothing. Treat them as separate checklists: access, indexing and clean chunking for retrieval; direct answers, self-contained passages and concrete sourcing for generation. Both need to be genuinely handled.
The bottom line
Source selection is a two-stage filter. Retrieval asks whether you’re accessible, relevant and cleanly chunked; generation asks whether your passage is direct, self-contained and concretely sourced enough to build an answer on. Authority isn’t the deciding factor at either stage, which is why passage craft can beat brand size — but only if you clear both gates.
We optimise for both retrieval and generation, because failing either means no citation. Part of our AI Visibility service.
