How Llms.txt And Robots.txt Affect AI Crawlers

The instinct to delete legacy pages during a refresh is usually wrong. They are what the existing mentions point at, and removing them severs the connection between old corroboration and the current record.

This claim circulates constantly and it is usually presented with more confidence than the evidence supports. It is also probably directionally true, for reasons that are structural rather than mysterious.

The volumes will be small, so avoid drawing conclusions from a handful of sessions and let it accumulate over a quarter or two. Also compare against your branded organic traffic rather than all organic, since branded search is closer in intent and makes for a fairer comparison.

Deciding Whether to Block Anything There is a legitimate argument for restricting training crawlers, particularly for publishers whose archive is the product. That is a commercial and editorial decision and it deserves a real discussion rather than a default.

If the budget is substantial, add the earned coverage work, which is the slowest and most expensive component and the one you genuinely cannot do quickly on your own. Buying that first, before the cheap fixes are done, is the most common way money gets wasted in this field. ai visibility agency

The condition is that the output has to be yours to keep and act on elsewhere, including the prompt set. An audit that only makes sense inside that agency's retainer is a sales document with a price attached.

You are unlikely to read all of it, and its presence changes the incentives entirely. An agency that knows the raw evidence ships with the report writes a different summary than one that knows it will not be checked.

What the Evidence Actually Is The figure quoted most often comes from Opollo, which reported assistant referred traffic converting at 14.2 percent against 2.8 percent from conventional search. The sample was 312 business to business brands, attributed through UTM parameters, covering the third quarter of 2024 through the first quarter of 2025.

Be prepared for the internal objection that this sends people to competitors. Some of it will, and those are mostly people who would not have bought from you anyway. The trade is that the page becomes usable as an impartial source, which is worth considerably more than the small number of poorly matched prospects it redirects, and the sales team usually agrees once they see which enquiries stop arriving.

Keeping Them Alive Comparison content decays faster than anything else you publish. Prices change, features ship, companies get acquired and a page comparing five options on last year's figures is not just stale, it is wrong.

If you want your own figure, the segment worth building is narrower than most people set up. Compare assistant referrals against branded organic search rather than against all organic, over at least a quarter, and exclude any campaign traffic. It will be a small sample and it will be about your audience, which makes it more useful for your decisions than a published study about somebody else's.

The risk is scope drift into activity that is easy to report and hard to value. The protection is to have the retainer specify countable units: prompt set runs per month, listings audited, corrections submitted, pages published or rewritten, outreach attempts made.

Broad sites are forgiving. A blocked section or a badly rendered template still leaves a hundred other pages describing the organisation. A small site with five pages has no such buffer, which makes the mechanical checks disproportionately important.

The lesson generalises to any brand whose name is short, generic or ambiguous. The correction is not clever, it is repetitive: pick one written form, use it everywhere, and pair it with a descriptive phrase so that a mention alone is never the only clue about what it refers to.

Legacy Content Is an Asset and a Liability An older site carries accumulated mentions, which is genuine value that a new domain does not have. It also carries accumulated inconsistency: superseded pages, old contact details and descriptions that no longer match what the organisation does.

Making Any Model Safe Four clauses do most of the protective work regardless of structure. The prompt set and baseline archive belong to you and leave with you. Raw answers ship with every report. Scope is stated in countable units. And there is a defined review point with agreed criteria before the contract auto renews.

Blocking these is therefore not one decision. Turning away a training crawler is a defensible editorial position. Turning away the agent that fetches pages at answer time removes you from answers entirely, and the two are frequently confused.

The monthly report is where an engagement is either accountable or theatrical, and the difference is visible from the first page. A useful report can be argued with. A padded one cannot, because there is nothing in it specific enough to disagree about.