Decide what search engines crawl and what they show.

Edit your robots.txt without FTP, and set noindex, nofollow and snippet rules for any page, archive or post type. Rilea SEO keeps your sitemap listed and your rules consistent, for free.

What it does

Robots.txt in the dashboard

Write your own robots.txt rules in Rilea SEO's settings. No FTP, no file manager.

Sitemap line added for you

Your sitemap index is listed in robots.txt automatically, unless your custom rules already include it.

Per-page robots meta

Noindex, nofollow, noarchive, nosnippet and noimageindex for any post, page, product or term.

Snippet controls

Set the maximum snippet length and image preview size search engines may show for a page.

Site-wide rules

Noindex whole post types and taxonomies, empty archives, paginated archives and search results.

Kept in step

Anything set to noindex is left out of your sitemaps automatically, so the two never disagree.

Set it up

  1. 1

    Check your current file

    Open yoursite.com/robots.txt in a browser to see what search engines see today.

  2. 2

    Edit if you need to

    In Rilea SEO, then Settings, then General, add custom rules in the robots.txt box, or leave it empty for WordPress's default.

  3. 3

    Set your noindex rules

    In Titles & Meta, choose which post types and archives should be kept out of search.

How it compares

Rilea SEO
Free: robots.txt editor, per-page robots meta and site-wide rules
Yoast SEO
Free: robots.txt file editor and per-page robots settings
Rank Math
Free: robots.txt editor and per-page robots settings
Is there a real robots.txt file on your server? If so, it overrides WordPress and any editor. Delete or rename the physical file to let Rilea SEO's version take over.

Feature details from yoast.com, rankmath.com and the plugins' WordPress.org pages, checked November 2026. Plans change, so check their sites for current details.

Questions

Should I block pages in robots.txt or with noindex?

Use noindex to keep a page out of search results. Robots.txt stops crawling, but a blocked page can still be indexed from links, and search engines can't see a noindex tag on a page they're not allowed to crawl.

Can I block AI crawlers?

Yes. Add rules for their user agents in your custom robots.txt. Well-behaved crawlers follow them; robots.txt is a request, not a lock.

What does WordPress put in robots.txt by default?

It blocks the admin area while allowing the file WordPress uses for background requests. Rilea SEO adds your sitemap line to that.

Two tools for two jobs

Search engines work in two stages. First they crawl: they fetch pages and follow links. Then they index: they decide which pages to store and show in results. You have a separate tool for each stage, and mixing them up is one of the most common technical SEO mistakes.

  • robots.txt controls crawling. It's a plain text file at the root of your site that tells crawlers which paths they may and may not request.
  • The robots meta tag controls indexing. It's a tag in each page's head that tells search engines whether to include the page in results, whether to follow its links, and how much of it to show.

Rilea SEO gives you both, in the WordPress dashboard, without editing files or theme code.

Editing robots.txt

WordPress doesn't create a real robots.txt file. It generates one on request, which is why you can see yoursite.com/robots.txt even though there's no such file in your folders. By default, it blocks the admin area while allowing admin-ajax.php, which some themes and plugins use for front-end features.

In Rilea SEO, then Settings, then General, there's a robots.txt box. Leave it empty and WordPress's default is used. Type your own rules, and they replace the default. Either way, Rilea SEO adds a line pointing to your sitemap index at /sitemap_index.xml, unless your rules already mention it, so every search engine that reads robots.txt can find your sitemap.

A typical robots.txt

Most WordPress sites need very little. A sensible starting point is to block the admin area, allow admin-ajax.php, and let Rilea SEO add the sitemap line. Add more only when you have a specific reason, such as keeping crawlers out of internal search results or a folder of files that shouldn't be crawled.

Common uses

  • Blocking internal search. Search result pages can create endless crawlable URLs. Blocking the search query path saves crawl effort.
  • Blocking parameter URLs. Filtered or sorted URLs on a shop can multiply quickly. Blocking the parameters keeps crawlers on your main pages.
  • Managing AI crawlers. Many AI companies publish the user agent names of their crawlers. Add rules for them if you want to allow or refuse them.
  • Slowing aggressive bots. Some crawlers respect a crawl-delay rule. Google ignores it.

What not to block

  • CSS and JavaScript. Search engines render pages like a browser. Blocking theme files can make your pages look broken to them.
  • Pages you want removed from results. Use noindex instead. A blocked page can still appear in results if other sites link to it, just without a description.
  • Your whole site by accident. A single "Disallow: /" under "User-agent: *" blocks everything. It's often left over from a staging site, so check carefully after a launch.

Robots meta for each page

Open any post, page or product, and in the Rilea SEO panel's advanced settings you'll find the robots controls:

  • Noindex. Keep this page out of search results. Use the default to follow the setting for its post type, or override it either way.
  • Nofollow. Ask search engines not to follow the links on this page.
  • Noarchive. Don't show a cached copy.
  • Nosnippet. Don't show a text snippet under the title in results.
  • Noimageindex. Don't index the images on this page.
  • Max snippet. Limit the snippet to a number of characters.
  • Max image preview. Allow none, standard or large image previews.

When a page is indexable, Rilea SEO tells search engines they may show a large image preview and full video previews by default, which helps your results stand out in Google Discover and image-rich layouts. You can change the image preview for any page.

Categories, tags and other terms have their own noindex setting on their edit screens.

Site-wide indexing rules

Most indexing decisions are better made once for a whole type of page. In Rilea SEO, then Settings, then Titles & Meta, you can set noindex for:

  • Each post type, such as a custom post type used only for internal content.
  • Each taxonomy, such as tags on a site where tag archives are thin.
  • Author and date archives, which often duplicate your blog on single-author sites.
  • Search results, which should almost never be indexed.

In the General settings, you'll also find options to noindex empty category and tag archives, noindex paginated archive pages from page two onwards, and redirect attachment pages to their parent post.

Robots meta and sitemaps work together

A page that's set to noindex shouldn't be in your sitemap: a sitemap says "please index this", and a noindex tag says the opposite. Search Console reports that contradiction as an error. Rilea SEO avoids it automatically. Anything set to noindex, individually or through its post type or taxonomy, is left out of your sitemaps. If you change a page from noindex to index, it appears in the sitemap again.

Checking your setup

  1. Open /robots.txt and check it shows your rules and the sitemap line.
  2. Use the robots.txt report in Google Search Console to see the version Google has fetched and any problems it found.
  3. Run important pages through Rilea SEO's SEO Analyzer, which checks the robots meta tag, robots.txt and the sitemap reference.
  4. Use Search Console's URL Inspection tool on any page you're unsure about, to see whether Google can crawl and index it.
  5. Check the Pages report in Search Console for "Excluded by noindex tag" and "Blocked by robots.txt", and make sure only the pages you intended appear there.

Common mistakes

  • Leaving a staging block in place. WordPress's "Discourage search engines" setting, under Settings, then Reading, adds a noindex to every page. Make sure it's off on your live site.
  • Blocking a page and noindexing it. If robots.txt blocks a page, search engines can't read its noindex tag. To remove a page from results, allow crawling and use noindex.
  • A physical robots.txt file. If your host or an old plugin created a real file, it's served instead of WordPress's version and your edits won't appear. Remove it.
  • Two plugins writing robots meta. Running another SEO plugin alongside Rilea SEO can produce two robots tags. Keep one SEO plugin active.
  • Noindexing pages that earn links. If a page has valuable links pointing to it, consider improving or redirecting it rather than removing it from the index.

Moving from another plugin

The importers bring across each post's noindex, nofollow and advanced robots settings, along with the site-wide noindex choices for post types and taxonomies. If you had a custom robots.txt in your old plugin, copy the rules into Rilea SEO's robots.txt box before you deactivate it, then check /robots.txt in a private browser window.

On Shopify

Shopify generates robots.txt from a theme template rather than a setting. Rilea SEO for Shopify handles robots meta per item instead: you can choose to index a page, to noindex it everywhere using Shopify's own hidden setting, or to use "noindex, follow" while keeping it in the store's sitemap and search, plus nofollow, noarchive, nosnippet and image preview controls.

Which pages should be noindexed?

Every site is different, but some pages almost never belong in search results, while others are often noindexed by mistake.

Usually noindex

  • Thank-you and confirmation pages that people should only reach after a form or purchase.
  • Internal search results, which create endless thin pages.
  • Login, account, cart and checkout pages.
  • Thin tag archives on sites where most tags have one or two posts.
  • Author archives on single-author sites, which duplicate the blog.
  • Date archives, which rarely match anything people search for.
  • Test, draft-like and duplicate landing pages made for ads or experiments.

Usually keep indexed

  • Category pages with a clear focus and an introduction, which often rank well for broad searches.
  • Product category pages on a shop.
  • Paginated pages of important archives, where the content on later pages can't be found any other way. Consider noindexing them only if the archive's posts are well linked elsewhere.
  • Old posts that still get visits, even if they're no longer central to your site.

Snippet controls in practice

Snippet controls are rarely needed, but they're useful in a few situations. Use nosnippet or a short max-snippet length for pages with content you don't want reproduced in search results, such as premium or legal text. Use max-image-preview set to none if you don't want images from a page shown large in results. For almost every other page, leave the defaults, which allow full snippets and large image previews and give your results the best chance of standing out.

Robots.txt and crawl budget

Search engines don't crawl every URL on a site every day. On large sites, especially shops with filters and sort options, the number of possible URLs can run into the millions, most of them near-duplicates. Careful robots.txt rules keep crawlers on the pages that matter: products, categories and content. Combined with a clean sitemap and good internal links, that helps new and updated pages get crawled sooner. Small sites rarely need to worry about this; for them, the default robots.txt and the sitemap line are enough.

Making changes safely

Robots rules are powerful, and a single wrong line can hide a whole site from search. Before changing robots.txt on a live site, copy the current version somewhere safe. Change one thing at a time, check the result with Search Console's robots.txt report, and watch the Pages report over the following weeks. If traffic drops unexpectedly after a change, the robots rules are one of the first places to look.

Rilea SEO keeps the riskiest settings, such as noindex and site-wide rules, behind the advanced and settings permissions in its role manager, so only the people you trust with them can change them.

Put your SEO budget somewhere else.

Install Rilea SEO, import your settings and keep every feature, for free.

Scroll to Top