{"id":1797,"date":"2026-09-01T16:47:00","date_gmt":"2026-09-01T16:47:00","guid":{"rendered":"https:\/\/rblog.it\/en\/?p=1797"},"modified":"2026-09-01T16:47:00","modified_gmt":"2026-09-01T16:47:00","slug":"robots-txt-guide","status":"publish","type":"post","link":"https:\/\/rblog.it\/en\/seo\/robots-txt-guide\/","title":{"rendered":"robots.txt: A Practical Guide with Examples"},"content":{"rendered":"<p>The <strong>robots.txt<\/strong> file is a small text file that tells search engine crawlers which parts of your site they may and may not crawl. It is simple, it sits at the root of every site, and a single wrong line in it can quietly block your whole site from search. This tutorial explains what robots.txt does, how the syntax works, what to put in it, and the mistakes to avoid \u2014 with examples.<\/p>\n<h2>What robots.txt is<\/h2>\n<p>The robots.txt file lives at the root of a domain \u2014 always at <code>\/robots.txt<\/code>. When a search engine crawler visits, it reads this file first to learn the rules for that site.<\/p>\n<p>It is worth being precise about what the file controls. robots.txt manages <strong>crawling<\/strong> \u2014 whether a crawler is allowed to fetch a URL. It does not directly control <strong>indexing<\/strong> \u2014 whether a page appears in search results. That distinction is the source of the most common and most damaging robots.txt mistakes, so keep it in mind throughout.<\/p>\n<h2>How the syntax works<\/h2>\n<p>robots.txt uses a small set of directives. The main ones:<\/p>\n<ul>\n<li><strong>User-agent:<\/strong> names the crawler the rules apply to. <code>*<\/code> means all crawlers.<\/li>\n<li><strong>Disallow:<\/strong> a path the crawler should not crawl.<\/li>\n<li><strong>Allow:<\/strong> a path the crawler may crawl, used to make an exception inside a disallowed section.<\/li>\n<li><strong>Sitemap:<\/strong> the full URL of your XML sitemap.<\/li>\n<\/ul>\n<p>A basic, sensible robots.txt for many sites looks like this:<\/p>\n<pre><code>User-agent: *\nDisallow: \/wp-admin\/\nAllow: \/wp-admin\/admin-ajax.php\n\nSitemap: https:\/\/example.com\/sitemap_index.xml<\/code><\/pre>\n<p>This tells every crawler to skip the admin area but allow one file inside it that the site needs, and points crawlers to the <a href=\"https:\/\/rblog.it\/en\/seo\/xml-sitemap-best-practices\/\">XML sitemap<\/a>.<\/p>\n<figure class=\"rblog-img-placeholder\">\n<div style=\"background:#f3f4f6;padding:40px;text-align:center;border:2px dashed #999;\"><strong>[IMG da inserire]<\/strong><br \/>a simple robots.txt file open in a text editor with the directives labeled<\/div>\n<\/figure>\n<h2>The crawl-versus-index trap<\/h2>\n<p>Here is the mistake that catches people. To keep a page out of search results, many people add a <code>Disallow<\/code> rule for it in robots.txt. That is the wrong tool.<\/p>\n<p>If you <code>Disallow<\/code> a page, crawlers will not fetch it \u2014 but the URL can still appear in search results, often with no description, if other pages link to it. Worse, because the crawler is blocked, it cannot even see a <code>noindex<\/code> tag on that page.<\/p>\n<p>The rule is simple. To stop a page being <strong>crawled<\/strong>, use robots.txt. To stop a page being <strong>indexed<\/strong>, use a <code>noindex<\/code> tag on the page itself \u2014 and leave it crawlable so the crawler can read that tag. Do not block a page in robots.txt and expect it to disappear from search.<\/p>\n<h2>What to block \u2014 and what not to<\/h2>\n<p>For most sites, robots.txt should stay short. Block only what genuinely should not be crawled: admin areas, internal scripts, and pointless parameter URLs that waste crawl effort.<\/p>\n<p>Be careful what you do <em>not<\/em> block. Never block CSS and JavaScript files \u2014 search engines need them to render and understand your pages. Never block your important content. And do not try to use robots.txt as a security tool: the file is public, so listing a sensitive path in it simply advertises that path to anyone who looks.<\/p>\n<h2>robots.txt and AI crawlers<\/h2>\n<p>Beyond traditional search crawlers, a growing number of AI bots crawl the web to gather training data and to power AI answers. robots.txt can name these crawlers specifically and allow or disallow them.<\/p>\n<p>This is a genuine decision, not a default. Allowing AI crawlers can help your content be surfaced in AI-generated answers; disallowing them keeps your content out of certain AI systems. There is no single correct choice \u2014 it depends on whether you want visibility in AI answers. Decide deliberately rather than copying someone else&#8217;s file blindly.<\/p>\n<h2>robots.txt on WordPress<\/h2>\n<p>WordPress generates a basic virtual robots.txt automatically. Most SEO plugins let you view and edit it from the plugin settings, which is the easiest way to manage it.<\/p>\n<p>For the large majority of WordPress sites, the default plus a sitemap line is enough. Resist the urge to add long lists of rules copied from elsewhere \u2014 a short, correct robots.txt is safer than an elaborate one you do not fully understand. Treat robots.txt review as a standard part of any <a href=\"https:\/\/rblog.it\/en\/seo\/seo-audit-checklist\/\">SEO audit<\/a>.<\/p>\n<h2>Common robots.txt mistakes<\/h2>\n<p>A few errors do real damage. The most severe is <strong>blocking the whole site<\/strong> \u2014 a line like <code>Disallow: \/<\/code> tells crawlers to skip everything, and it is sometimes left over from a staging environment after launch. Next is <strong>blocking CSS and JavaScript<\/strong>, which stops search engines rendering pages correctly. Then <strong>using Disallow to hide a page from search<\/strong>, which does not work and can leave a bare URL in results. And <strong>forgetting the sitemap line<\/strong>, a small, free improvement many sites skip. After any site launch or migration, check robots.txt first.<\/p>\n<h2>FAQ<\/h2>\n<h3>What does robots.txt do?<\/h3>\n<p>robots.txt tells search engine crawlers which parts of a site they may and may not crawl. It controls crawling, not indexing \u2014 it does not directly remove pages from search results. It sits at the root of every domain at \/robots.txt.<\/p>\n<h3>Can I use robots.txt to remove a page from Google?<\/h3>\n<p>No. Blocking a page in robots.txt stops it being crawled but the URL can still appear in search results. To keep a page out of the index, use a noindex tag on the page and leave it crawlable so the tag can be read.<\/p>\n<h3>Should I block CSS and JavaScript in robots.txt?<\/h3>\n<p>No. Search engines need CSS and JavaScript to render and understand your pages correctly. Blocking them can hurt how your pages are evaluated. Keep these resources crawlable.<\/p>\n<h3>Where is the robots.txt file located?<\/h3>\n<p>Always at the root of the domain, at \/robots.txt. On WordPress, a basic version is generated automatically, and most SEO plugins let you view and edit it from their settings without touching files directly.<\/p>\n<h2>Where to go next<\/h2>\n<p>Open your own robots.txt at \/robots.txt and check it. Confirm there is no leftover <code>Disallow: \/<\/code>, that CSS and JavaScript are not blocked, and that your sitemap is listed. Keep the file short \u2014 block only admin areas and genuine crawl waste \u2014 and remember that hiding a page from search needs a noindex tag, not a Disallow line. A correct robots.txt is small, boring, and quietly important.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The robots.txt file is a small text file that tells search engine crawlers which parts of your site they may and may not crawl. It is simple, it sits at the root of every site, and a single wrong line [&hellip;]<\/p>\n","protected":false},"author":8,"featured_media":1796,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_wds_title":"","_wds_metadesc":"","_wds_focus-keywords":"","_wds_canonical":"","sync_status":"none","episode_type":"","audio_file":"","podmotor_file_id":"","podmotor_episode_id":"","castos_file_data":"","cover_image":"","cover_image_id":"","duration":"","filesize":"","filesize_raw":"","date_recorded":"","explicit":"","block":"","wds_primary_category":0,"footnotes":"","_wds_robots-noindex":"","_wds_robots-nofollow":"","_wds_opengraph-title":"","_wds_opengraph-description":"","_wds_twitter-title":"","_wds_twitter-description":""},"categories":[28],"tags":[213,212,159,158],"series":[],"class_list":["post-1797","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-seo","tag-crawling","tag-robots-txt","tag-technical-seo","tag-wordpress"],"episode_featured_image":"https:\/\/rblog.it\/en\/wp-content\/uploads\/2026\/05\/robots-txt-guide.jpg","episode_player_image":"https:\/\/rblog.it\/en\/wp-content\/uploads\/2021\/11\/rblog-podcast.png","download_link":"","player_link":"","audio_player":false,"episode_data":{"playerMode":"dark","subscribeUrls":{"apple_podcasts":{"key":"apple_podcasts","url":"","label":"Apple Podcasts","class":"apple_podcasts","icon":"apple-podcasts.png"},"stitcher":{"key":"stitcher","url":"","label":"Stitcher","class":"stitcher","icon":"stitcher.png"},"google_podcasts":{"key":"google_podcasts","url":"","label":"Google Podcasts","class":"google_podcasts","icon":"google-podcasts.png"},"spotify":{"key":"spotify","url":"https:\/\/open.spotify.com\/show\/5IjCCrGAVU62CiHVLLbjTY","label":"Spotify","class":"spotify","icon":"spotify.png"}},"rssFeedUrl":"https:\/\/rblog.it\/en\/feed\/podcast\/growth-marketing-rblog","embedCode":"<blockquote class=\"wp-embedded-content\" data-secret=\"Xze3vrJ2ju\"><a href=\"https:\/\/rblog.it\/en\/seo\/robots-txt-guide\/\">robots.txt: A Practical Guide with Examples<\/a><\/blockquote><iframe sandbox=\"allow-scripts\" security=\"restricted\" src=\"https:\/\/rblog.it\/en\/seo\/robots-txt-guide\/embed\/#?secret=Xze3vrJ2ju\" width=\"500\" height=\"350\" title=\"&#8220;robots.txt: A Practical Guide with Examples&#8221; &#8212; Rblog\" data-secret=\"Xze3vrJ2ju\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\" class=\"wp-embedded-content\"><\/iframe><script type=\"text\/javascript\">\n\/* <![CDATA[ *\/\n\/*! This file is auto-generated *\/\n!function(d,l){\"use strict\";l.querySelector&&d.addEventListener&&\"undefined\"!=typeof URL&&(d.wp=d.wp||{},d.wp.receiveEmbedMessage||(d.wp.receiveEmbedMessage=function(e){var t=e.data;if((t||t.secret||t.message||t.value)&&!\/[^a-zA-Z0-9]\/.test(t.secret)){for(var s,r,n,a=l.querySelectorAll('iframe[data-secret=\"'+t.secret+'\"]'),o=l.querySelectorAll('blockquote[data-secret=\"'+t.secret+'\"]'),c=new RegExp(\"^https?:$\",\"i\"),i=0;i<o.length;i++)o[i].style.display=\"none\";for(i=0;i<a.length;i++)s=a[i],e.source===s.contentWindow&&(s.removeAttribute(\"style\"),\"height\"===t.message?(1e3<(r=parseInt(t.value,10))?r=1e3:~~r<200&&(r=200),s.height=r):\"link\"===t.message&&(r=new URL(s.getAttribute(\"src\")),n=new URL(t.value),c.test(n.protocol))&&n.host===r.host&&l.activeElement===s&&(d.top.location.href=t.value))}},d.addEventListener(\"message\",d.wp.receiveEmbedMessage,!1),l.addEventListener(\"DOMContentLoaded\",function(){for(var e,t,s=l.querySelectorAll(\"iframe.wp-embedded-content\"),r=0;r<s.length;r++)(t=(e=s[r]).getAttribute(\"data-secret\"))||(t=Math.random().toString(36).substring(2,12),e.src+=\"#?secret=\"+t,e.setAttribute(\"data-secret\",t)),e.contentWindow.postMessage({message:\"ready\",secret:t},\"*\")},!1)))}(window,document);\n\/* ]]> *\/\n<\/script>\n","syncStatus":{"isSynced":false,"status":"none","error":"","message":"Not synced yet.","title":"Not synced"}},"_links":{"self":[{"href":"https:\/\/rblog.it\/en\/wp-json\/wp\/v2\/posts\/1797","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rblog.it\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rblog.it\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rblog.it\/en\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/rblog.it\/en\/wp-json\/wp\/v2\/comments?post=1797"}],"version-history":[{"count":0,"href":"https:\/\/rblog.it\/en\/wp-json\/wp\/v2\/posts\/1797\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rblog.it\/en\/wp-json\/wp\/v2\/media\/1796"}],"wp:attachment":[{"href":"https:\/\/rblog.it\/en\/wp-json\/wp\/v2\/media?parent=1797"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rblog.it\/en\/wp-json\/wp\/v2\/categories?post=1797"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rblog.it\/en\/wp-json\/wp\/v2\/tags?post=1797"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/rblog.it\/en\/wp-json\/wp\/v2\/series?post=1797"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}