How to Edit robots.txt Without Blocking Your Own Site

Your traffic dropped by half over a weekend, and the only change anyone remembers is "a small tweak to the robots.txt file". That is one of the most common ways a small business or community site quietly removes itself from search, and it usually takes a single line to do it.

This guide shows how to edit robots.txt safely: what the file does, which lines are worth having, which ones cause damage, and how to test a change before search engines see it. It is written for WordPress users, but the rules apply to any website.

What robots.txt does and what it does not do

Robots.txt is a plain text file that lives at the root of your domain, so it always sits at /robots.txt. Crawlers from search engines read it before they fetch other pages, and it tells them which paths they may request.

Three points trip people up again and again:

• It controls crawling, not indexing. A page blocked in robots.txt can still appear in search results if other sites link to it, often with no description.
• It is public. Anyone can open the file. Listing a secret admin path in it is an invitation, not a lock.
• It is advice. Reputable crawlers follow it. Scrapers and spam bots often ignore it completely.

So robots.txt is a traffic sign for polite crawlers. It is not security and it is not the right tool for removing a page from search.

Where to find and edit robots.txt in WordPress

WordPress creates a virtual robots.txt automatically if no real file exists. You will not see it in your file manager, but you can open it in a browser at your domain followed by /robots.txt.

You have three ways to change it:

1. Through an SEO plugin. Yoast, Rank Math and All in One SEO each include a file editor under their tools menu. This is the simplest route for most site owners.
2. By uploading a real file. Create a text file called robots.txt and upload it to the public_html folder with your host's file manager or SFTP. A physical file overrides the virtual one.
3. Through your host. Some managed hosts add their own rules for staging sites, so check after any migration. Our guide on moving WordPress to a new host without downtime covers this step.

Before you touch anything, copy the current contents into a note with today's date. If something goes wrong, you can paste it back in under a minute.

A safe robots.txt for a small site

Most small sites need far fewer rules than people assume. A sensible WordPress file has four lines:

• User-agent: * which means the rules below apply to every crawler.
• Disallow: /wp-admin/ which keeps crawlers out of the dashboard area.
• Allow: /wp-admin/admin-ajax.php because some themes and plugins load front end content through that file.
• Sitemap: followed by the full address of your XML sitemap, starting with https.

That is enough. The sitemap line helps crawlers find new pages, and the admin rule saves them from wasting requests on login screens. Everything else on a site of a few hundred pages is usually noise.

Lines that break sites

These are the edits that show up again and again in traffic drop investigations:

• Disallow: / on its own. This blocks the whole site. It is often left over from a staging copy, or set by the WordPress option "Discourage search engines from indexing this site" being ticked during a launch.
• Blocking /wp-content/ or /wp-includes/. This hides your CSS, JavaScript and images, so search engines see a broken page and can rank it lower.
• Wildcards that match too much. A rule like Disallow: /*? meant to block search result pages can also block product filters, pagination or tracking links your visitors use.
• Typos in the path. Paths are case sensitive. /Blog/ and /blog/ are different folders to a crawler.

Why blocking thin pages is often the wrong fix

A lot of SEO checklists tell you to disallow tag archives, author pages, search results and anything that looks thin. On a small site, I usually push back on this.

The reason is simple. If a page is blocked in robots.txt, the crawler cannot read it, which means it also cannot see a noindex tag on that page. The URL can stay in the index with no description for months. If you want a page out of search, the better method is to let crawlers visit it and add a noindex meta tag, which every major SEO plugin can do with one toggle. Use robots.txt for crawl waste on large sites, such as thousands of filter combinations in a shop, not for tidying up a 60 page brochure site.

Crawl budget is a real concern for sites with hundreds of thousands of URLs. For a local bakery or a fan club with 150 posts, search engines will crawl everything they need without help.

How to test a robots.txt change before it goes live

Treat every edit like a code change. A five minute check can save weeks of lost visits.

1. Write down five important URLs. Your homepage, a top category, two popular posts and your contact page.
2. Use the robots.txt report in Search Console. It shows the version search engines last fetched and flags syntax errors.
3. Run the URL Inspection tool on each of your five URLs after the change. Each one should say crawling is allowed.
4. Open the live file in a private browser window to make sure caching or a CDN is not serving an old copy.
5. Check again after seven days. Look at the Pages report for a jump in "Blocked by robots.txt" counts.

If something looks wrong, restore the old version from your note and request a recrawl of the homepage. Recovery is usually quick once the block is lifted, often within a few days for an active site.

Keep the rest of your search setup in step

Robots.txt rarely works alone. If you change URLs, update your redirects and sitemap at the same time, and check that your page titles and snippets still make sense. Our piece on writing meta descriptions that earn clicks is a good next step once crawlers can reach every page again. For more small fixes like this one, browse the web tips collection.

Frequently asked questions

Does every website need a robots.txt file?

No. A site with no robots.txt is crawled normally, and WordPress creates a virtual one anyway. Adding a simple file with your sitemap address is still worth it, because it helps crawlers find new pages faster.

Can robots.txt remove a page from search results?

Not reliably. Blocking a page stops crawling, but the URL can still be indexed from links elsewhere. To remove a page, allow crawling and add a noindex tag, or delete the page and return a 404 or 410 status.

How long does a robots.txt change take to work?

Search engines usually refetch the file within about 24 hours, and you can request a faster recrawl in Search Console. The effect on individual pages depends on how often they are crawled, so expect a few days to a few weeks for full changes.