Robots.txt Explained for Beginners (2026 Guide)

Robots.txt is one of the most misunderstood files in SEO. Get it right, and search engines will crawl your site efficiently. Get it wrong, and you might accidentally block Google from your entire website.

In this beginner-friendly guide, you'll learn exactly what robots.txt is, how to create one, and common mistakes to avoid.

What is Robots.txt?

Robots.txt is a simple text file placed in the root directory of your website (e.g., https://smp-web.com/robots.txt) that tells search engine crawlers which pages or sections of your site they should or shouldn't crawl.

It's part of the Robots Exclusion Protocol — an industry standard used by all major search engines including Google, Bing, and Yahoo.

How Robots.txt Works

When a search engine crawler visits your site, it first looks for /robots.txt. It reads the instructions and follows them when crawling your site.

Important: Robots.txt only controls crawling, not indexing. If a page is linked elsewhere, it can still appear in search results even if you block it in robots.txt. To prevent indexing, use the noindex meta tag instead.

Basic Robots.txt Syntax

Robots.txt uses two main directives:

• User-agent: Which crawler the rule applies to (* means all)
• Disallow: Which paths should NOT be crawled
• Allow: Which paths SHOULD be crawled (overrides Disallow)
• Sitemap: Location of your XML sitemap

Example 1: Allow Everything

User-agent: *
Allow: /

Sitemap: https://smp-web.com/sitemap.xml

This tells all crawlers to crawl the entire site. This is the most common setup.

Example 2: Block a Specific Folder

User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://smp-web.com/sitemap.xml

Example 3: Block Everything (Not Recommended)

User-agent: *
Disallow: /

This blocks ALL crawlers from your entire site — meaning you'll disappear from Google. Only use this on staging/development sites.

What Should You Block?

Common things to block in robots.txt:

• Admin panels: /admin/, /wp-admin/
• Login pages: /login, /signin
• Cart/checkout: /cart, /checkout
• Search results: /search?q=
• Duplicate content: /print/, /amp/ (if canonical handles it)
• Internal folders: /tmp/, /cgi-bin/

What NOT to Block

• ❌ CSS and JavaScript files (Google needs them to render pages)
• ❌ Images you want in Google Images
• ❌ Your main content pages
• ❌ Your sitemap.xml

Do You Even Need Robots.txt?

Technically, no. If you don't have a robots.txt file, search engines will crawl your entire site. Many small blogs don't need one.

You should create robots.txt if:

• You have an e-commerce site with cart/checkout pages
• You have admin areas you want to hide
• You have duplicate content issues
• You want to explicitly point to your sitemap

How to Create Robots.txt (Step by Step)

1. Open a plain text editor (Notepad, VS Code)
2. Add your rules (see examples above)
3. Save as robots.txt (not .txt.txt)
4. Upload to your website's root folder (same location as index.html)
5. Test at https://yourdomain.com/robots.txt
6. Verify in Google Search Console → robots.txt Tester

Common Robots.txt Mistakes

• Blocking CSS/JS: Google can't render your pages properly
• Using wrong case: Disallow:/Admin/ won't match /admin/
• Forgetting slashes: Always start paths with /
• Blocking /wp-admin/admin-ajax.php: Breaks WordPress features
• Typo in filename: Must be exactly robots.txt
• Using HTML: It's a text file, not HTML

Testing Your Robots.txt

After uploading, test it:

1. Visit https://yourdomain.com/robots.txt in your browser
2. Use Google's robots.txt Tester
3. Check Google Search Console for crawl errors

Robots.txt vs Noindex

Final Thoughts

Robots.txt is a simple but powerful file. For most websites, a basic "allow everything + sitemap" setup is enough. Only add Disallow rules when you have a specific reason.

Always test after changes, and never block CSS or JavaScript files. Use our free Robots.txt Generator to create a proper file in seconds.