Common Robots.txt Configurations & Examples
Ready-to-use robots.txt templates for common technical SEO architectures and crawl scenarios. Copy and customize them for your website.
Allow All Crawlers Everything
Grants unrestricted crawling access across the entire domain to all compliant search engine bots.
User-agent: * Disallow:
Block a Sensitive or Private Directory
Restricts crawlers from spending crawl budget on internal directories like checkouts, accounts, or staging.
User-agent: * Disallow: /admin/ Disallow: /checkout/ Disallow: /private/
Block Googlebot Specifically
Applies a blanket disallow strictly to Googlebot while leaving other crawlers governed by their respective groups.
User-agent: Googlebot Disallow: /
Allow a Public Subdirectory Inside a Disallowed Parent
Disallows a top-level folder but carves out a specific subfolder using Google's longest-match Allow rule.
User-agent: * Disallow: /products/ Allow: /products/public/ Allow: /products/featured/
Block Specific File Extensions via Wildcards
Uses wildcard (*) and end anchor ($) matching to prevent crawlers from indexing raw documents or scripts.
User-agent: * Disallow: /*.pdf$ Disallow: /*.json$ Disallow: /*?*sort=
Standard Production Setup with Sitemap Declaration
The gold-standard baseline for modern corporate, blog, and e-commerce websites.
User-agent: * Disallow: /api/ Disallow: /_next/ Disallow: /admin/ Sitemap: https://example.com/sitemap.xml Sitemap: https://example.com/sitemap-posts.xml
What Is a Robots.txt File?
A robots.txt file is a standardized plain-text document situated at the root directory of a web server. Its primary function is to govern crawl traffic by instructing automated web crawlers—such as Googlebot, Bingbot, and Applebot—which parts of your website they are permitted or prohibited from requesting and downloading.
Originating from the Robots Exclusion Standard established in 1994 and formally codified as RFC 9309 by the Internet Engineering Task Force (IETF) with Google, Microsoft, and modern search architectures, robots.txt serves as the frontline gateway of web crawling etiquette. When a compliant web spider visits your domain, it requests https://yourdomain.com/robots.txt before touching any other page or asset.
If the file is present, the crawler parses its directives to plan its crawl budget efficiently. If the file does not exist (returning an HTTP 404 status), the spider assumes you have declared no crawling barriers and proceeds to explore all discoverable links on your website. For a comprehensive audit of your website's overall health, explore our SolveX SEO Crawler & Site Auditor .
How Does Robots.txt Work?
The operation of robots.txt relies on a cooperative handshake between web servers and compliant automated clients. When a search engine spider identifies a hyperlink pointing to your domain, it cannot simply fetch the target document immediately. Instead, it must first query the host's root for robots instructions.
The crawler parses directives from top to bottom, matches its user-agent identity to the most appropriate directive block, and evaluates candidate paths. Once a matching rule is determined, the spider either adds the target URL to its download queue or discards it from crawling.
Robots.txt Crawler Evaluation Process
The exact 4-step logic sequence search bots run prior to downloading any web page
Fetch /robots.txt
Bot requests /robots.txt at root origin. If HTTP 404, bot assumes zero crawl restrictions.
Select Group
Bot searches for exact User-agent match (e.g. Googlebot). If not found, falls back to *.
Longest Match
Evaluates all matching Allow & Disallow patterns against target path. The rule with the greatest character length wins.
Crawl Execution
If Allowed, spider fetches HTML. If Blocked, spider does not request content, but URL can still be indexed via backlinks!
What Is a Robots.txt Tester?
A Robots.txt Tester is a specialized technical SEO utility that inspects, validates, and simulates search engine crawling permissions on your live website. Rather than manually reading through dozens of lines of regex patterns and wildcard rules, a robots.txt tester acts as an automated sandbox.
The SolveX Robots.txt Tester enables webmasters and SEO engineers to:
- Validate File Reachability: Confirm your web server returns a valid HTTP 200 response rather than an accidental 403 Forbidden or 500 Internal Server Error.
- Simulate Crawler Decisions: Test whether Googlebot, Googlebot-Image, or Bingbot is allowed or blocked on any arbitrary path on your website.
- Debug Specificity Conflicts: Uncover when an unexpected Disallow directive takes precedence over an intended Allow rule due to pattern length.
- Extract XML Sitemaps: Confirm that search spiders can immediately locate your XML sitemap declarations to accelerate indexing.
Whether preparing for a major site migration, restructuring URL taxonomies, or auditing organic search drop-offs, testing your robots file eliminates costly indexing catastrophes.
How to Test a URL Using Robots.txt
Verifying whether a search crawler can access an individual page or directory is a straightforward 4-step process with our tester:
Submit Your Website Root
Enter your homepage URL (e.g., https://example.com). The tool contacts the host server and pulls the live /robots.txt file directly into memory.
Select Your Target Crawler
Choose the search engine bot you want to simulate—such as Googlebot for primary search, Googlebot-Image for visual assets, or * for general spiders.
Enter Path or Paste Batch List
Type the specific URL path (e.g., /products/featured/item-4) or toggle the Batch URL List to paste dozens of internal paths simultaneously.
Analyze the Decision & Matched Rule
Review the verdict badge (ALLOWED vs BLOCKED), examine the exact line number responsible, and read the longest-match specificity note explaining why that directive prevailed.
What Do Allow and Disallow Mean?
At the heart of every robots.txt configuration lie two opposing directives: Disallow and Allow. Understanding how search engines weigh these statements against one another is essential for every technical SEO practitioner.
The Disallow Directive: Tells matching search engine bots not to download or crawl any URL starting with the given path prefix. If a Disallow line contains no value (Disallow:), it acts as an explicit override that clears all prior crawl prohibitions, signaling that the entire site is open for crawling.
The Allow Directive: Introduced by Google to carve out exceptions within broadly disallowed parent directories. For example, if an entire directory is blocked (Disallow: /catalog/), you can selectively permit access to public media (Allow: /catalog/public/).
Rule Specificity: How Longest-Match Resolution Works
Googlebot and modern crawlers determine permissions strictly by pattern length (character count)
/products/archived/item-99/catalog/public/product-a/reports/annual-2026How to Test Robots.txt for Googlebot
Googlebot represents the vast majority of organic search crawling for most commercial websites. However, Googlebot evaluates robots.txt files with several distinct behaviors codified in Google's search developer documentation:
1. Longest-Match Specificity Rule
Googlebot does not rely on directive order or first-come-first-served logic. Instead, it counts the number of characters in the matching pattern. The longest pattern always wins. If an Allow and Disallow directive are of equal length, Google awards the win to Allow.
2. Single Group Exclusivity
If your file contains both a User-agent: Googlebot block and a User-agent: * block, Googlebot matches its dedicated block and completely ignores the wildcard block. It does not merge or inherit directives across blocks.
3. File Size Limit (500 KiB)
Googlebot reads and parses up to 500 kibibytes (approx. 512,000 bytes) of a robots.txt file. Any rules appearing after the 500 KiB threshold are discarded and treated as non-existent.
4. No Crawl-Delay Support
Googlebot ignores the Crawl-delay directive. If your server experiences performance bottlenecks from Googlebot, rate adjustments must be configured via Google Search Console.
What Is a User-Agent in Robots.txt?
In HTTP networking, a User-Agent is an identification string transmitted in headers that informs the server about the client application making the request. In a robots.txt file, the User-agent: directive identifies which automated crawler a subsequent group of Allow and Disallow rules applies to.
Common search engine user-agent tokens include:
Google's general search crawler for web page indexing.
Crawler responsible for Google Images indexing.
Microsoft crawler powering Bing and Copilot search.
Universal wildcard fallback applicable to all spiders.
You can declare multiple User-agent lines in a single group to apply the exact same crawl policies across several spiders simultaneously, minimizing file bloat.
Robots.txt vs XML Sitemap & Noindex Meta Tags
One of the most widespread misunderstandings in technical SEO is conflating crawling controls (robots.txt) with indexing controls (noindex meta tags) and discovery mechanisms (XML sitemaps).
While they all interact with search engines, they operate at completely distinct layers of the search pipeline. Review the architecture comparison matrix below:
Robots.txt vs XML Sitemap vs Noindex Tag
The fundamental roles of search directives and where website owners commonly confuse them
| Technical Mechanism | Primary Purpose | Controls Crawling? | Prevents Indexing? | Security Level |
|---|---|---|---|---|
| Robots.txt (Disallow) | Preserves crawl budget & server load | YES (Blocks bot requests) | NO (Can still index via links) | None (Public plain-text file) |
| XML Sitemap | Assists content discovery & priority | Assists (Does not restrict) | No effect on deindexing | Public inventory of URLs |
| Meta Noindex Tag | Guaranteed removal from search results | NO (Must crawl to read tag) | YES (Guarantees no index) | Requires crawlable HTML |
| Server Authentication (401/403) | Confidential data & admin protection | YES (Refuses bot access) | YES (Content never exposed) | High (Enforced by web server) |
Does Robots.txt Prevent Google From Indexing a Page?
No! Robots.txt Controls Crawling, Not Guaranteed Indexing
If you disallow a URL in robots.txt, Googlebot will obey the rule and refrain from requesting the HTML document from your server. However, if other websites on the internet (or even other pages on your own website) link to that URL with descriptive anchor text, Google can still index the URL based purely on incoming signals!
When Google indexes a blocked page, it typically appears in search results with the notice:
The Paradox of Disallow + Noindex: If you place a <meta name="robots" content="noindex"> tag inside a page, but simultaneously block that page via Disallow: /page/ in robots.txt, Googlebot is blocked from downloading the HTML and will never see your noindex tag!
To guarantee that a page is removed from Google's index, you must allow crawling in robots.txt so that Googlebot can download the page and process the noindex directive. Learn more in our guide on Search Engine Performance Tracking.
Common Robots.txt Mistakes
A single typographical error in robots.txt can catastrophically dismantle your organic search visibility overnight. Here are the most frequent mistakes we uncover during technical audits:
Leaving Disallow: / active after pushing staging code to production. This instructs every search engine spider to halt all crawling across your domain.
Disallowing paths like /assets/ or /static/ prevents Googlebot from rendering pages visually, causing failed mobile-friendliness tests and broken Core Web Vitals scoring.
URL path matching in robots.txt is strictly case-sensitive. Writing Disallow: /Private/ will not block URLs with lowercase letters such as /private/.
Declaring Sitemap: /sitemap.xml violates RFC 9309. Sitemaps must always be absolute URLs including protocol and domain: https://example.com/sitemap.xml.
Should You Block Admin, Login, and Private Pages?
Blocking utility directories such as /admin/, /login/, and /cart/ is a legitimate technical SEO practice for preserving crawl budget. Search bots should not waste server resources repeatedly requesting login forms or temporary checkout carts.
Important Security Distinction:
Robots.txt is a publicly viewable document readable by any human, competitor, or malicious threat actor. Adding Disallow: /secret-admin-portal-v2/ publicly broadcasts the exact URL of your internal dashboard to the entire world.
Never use robots.txt to secure confidential data. Genuine access control must be enforced through HTTP server authentication, VPN gateways, password protection, and strict authorization headers.
How Often Should You Test Robots.txt?
Because robots.txt can profoundly influence organic indexing, you should conduct automated or manual testing under the following scenarios:
- After Site Migrations & CMS Updates: Content management systems (such as WordPress, Shopify, or Webflow) often alter robots.txt configurations automatically when themes or staging environments are deployed.
- When Restructuring URL Taxonomies: Introducing new subfolders or dynamic facet parameters requires testing to ensure crawl restrictions do not accidentally swallow public pages.
- During Organic Traffic Drops: If impressions drop suddenly in Google Search Console, checking for accidental disallow rules should be the first diagnostic step.
Need specialized assistance diagnosing complex crawl issues or optimizing enterprise crawl budget? Explore our professional SolveX SEO Services or review our Complete Guide to Modern SEO.
Frequently Asked Questions About Robots.txt
Clear, technically accurate answers to the most common questions regarding robots.txt configuration, Googlebot testing, and search engine crawl rules.
Ready to Transform Your Marketing?
Join hundreds of businesses that trust SolveX Marketing to drive their growth. Let's create something extraordinary together.