What robots.txt Actually Does

The robots.txt file sits at the root of your domain and tells compliant crawlers which paths they may request. That is its entire job. It controls crawling, not indexing, and the difference between those two things is responsible for a large share of technical SEO confusion. A URL blocked in robots.txt can still appear in search results if other sites link to it, because the crawler was told not to fetch the page but never told not to list it. Conversely, a page you block cannot have its noindex tag read, so blocking is the wrong tool for removing content from search.

Used correctly, robots.txt is a crawl efficiency instrument. It keeps automated crawlers away from infinite URL spaces, internal tooling and duplicative parameter combinations so that the crawl capacity you do have is spent on pages that can earn traffic.

How We Help: AAMAX.CO SEO Services

Small configuration files cause outsized damage, which is why technical review is a standing part of every engagement at AAMAX.CO. As a full-service digital marketing company delivering web development, digital marketing and SEO worldwide, we regularly find sites losing visibility to a single stray disallow rule left over from a staging deployment. Our SEO services include full crawl and indexation audits where we validate robots.txt against your live URL inventory, confirm nothing valuable is blocked, tighten crawl waste, and monitor index coverage afterwards so configuration drift is caught early rather than discovered in a traffic report.

Syntax Essentials

The file is plain text, served at the root of each host and protocol, and is case sensitive in its paths. It is organised into groups, each beginning with one or more user-agent lines followed by allow and disallow rules. A crawler follows the most specific user-agent group that applies to it and ignores the others, which means a wildcard group is not merged with a named group.

Matching is prefix based. A disallow of a folder path blocks everything beneath it. Two wildcards are supported by major search engines: an asterisk matches any sequence of characters, and a dollar sign anchors the end of a URL. When allow and disallow rules conflict, the more specific rule wins, and if they are equally specific the permissive rule generally prevails. An empty disallow value means nothing is blocked.

A Sensible Baseline

For most content sites, the best robots.txt is nearly empty. A reasonable baseline allows all crawlers everywhere, blocks a small number of genuinely non-public paths such as internal search results, cart and checkout flows, account areas and admin endpoints, and points to the XML sitemap. That is usually the whole file.

The sitemap directive is worth including because it is independent of user-agent groups and helps crawlers discover your canonical URL set quickly. List the absolute URL, and list multiple sitemaps or a sitemap index if you have them.

What You Should Never Block

Blocking CSS, JavaScript or image directories prevents search engines from rendering your pages the way users see them, which can distort mobile usability and layout assessments. This was once common practice and is now a clear mistake. Similarly, do not block URLs that carry canonical or noindex instructions you want honoured, do not block paginated series that provide the only path to deeper content, and never leave a global disallow in place on production. That last error, a single slash after a disallow directive for all agents, has removed entire businesses from search overnight.

Also avoid using robots.txt to hide sensitive information. The file is public, and listing a private path is effectively publishing a map to it. Use authentication for anything confidential.

Managing Crawl Waste

Where robots.txt earns its keep is on large sites with generated URL spaces. Faceted navigation that combines colours, sizes, brands and sort orders can produce millions of near-identical URLs. Calendars can generate infinite date pages. Session identifiers and tracking parameters can duplicate every page many times over.

Blocking these patterns preserves crawl capacity for pages that matter. Use wildcard patterns to target the parameters rather than individual URLs, but be careful: if a parameter combination is genuinely valuable for search, such as a popular category and brand pairing, keep it crawlable and handle duplication with canonical tags instead. The general rule is to use robots.txt for URL spaces that should never be crawled, and canonicalisation for URL spaces that should be crawled but consolidated.

Crawl Delay and Other Directives

Crawl-delay is supported by some crawlers and ignored by others, including Google, which prefers you manage crawl rate through server capacity and its own settings. Noindex directives inside robots.txt are not supported and should never be relied upon. Host directives are legacy and unnecessary for most sites. Keep the file to user-agent, allow, disallow and sitemap lines unless you have a specific reason otherwise.

Managing Aggressive Bots

Beyond search engines, your logs will show scrapers, SEO tools and AI training crawlers. Many of these respect named user-agent rules, so you can slow or exclude them if they consume meaningful bandwidth. Decide deliberately whether to allow AI crawlers: excluding them protects content but may reduce the chance of your brand being surfaced in generative answers. For businesses investing in GEO services, staying accessible to reputable answer engines is usually the better trade, while still blocking abusive scrapers that ignore rules entirely and are better handled at the firewall.

Testing and Maintenance

Always validate changes with a robots.txt tester before deploying, and test a representative sample of real URLs rather than assuming a pattern behaves as intended. Keep the file under version control so changes are reviewable, and treat it as a deployment artefact rather than something edited directly on the server.

Add two monitors: an alert if the file returns anything other than a 200 status, because a 5xx response can cause crawlers to pause crawling entirely, and an alert on content change so an unexpected staging file cannot ship silently. Review the file whenever you launch new sections, change platforms or add parameters, and cross-check it against index coverage reports quarterly.

The Short Version

Keep robots.txt minimal, block only what should never be crawled, never block rendering resources, reference your sitemap, use noindex rather than disallow for removal, and test every change. Done well, nobody notices it. Done badly, it is the cheapest way to lose an entire channel.