What is robots.txt?

A robots.txt file tells search engine crawlers which URLs they can access on your site. It helps prevent server overload and keeps sensitive pages out of search results.

Place robots.txt in the root directory of your website (e.g., https://example.com/robots.txt). Most search engines (Google, Bing, Yandex) respect robots.txt directives.

• User-agent: specifies which crawler the rules apply to. Use * for all crawlers.
• Disallow: blocks crawlers from specific paths.
• Allow: overrides a Disallow rule for a specific sub-path.
• Sitemap: points search engines to your XML sitemap.

Common Examples

User-agent: *
Disallow: /

User-agent: *
Disallow:

User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Allow: /admin/login.php
Sitemap: https://example.com/sitemap.xml

User-agent: Googlebot-Image
Disallow: /photos/
User-agent: *
Disallow: