Robots Meta Tag: noindex, nofollow and Every Rule

November 11, 2026 · 11 min read

Robots Meta Tag: noindex, nofollow, and Every Rule Google Reads

A robots meta tag is a <meta name="robots"> element in a page's head that tells search engines whether to index it, follow its links and show a snippet. Add noindex to keep a page out of results. Below is every rule Google honours today, how conflicts resolve, and how to noindex drafts on docs sites.

What Is a Robots Meta Tag?

Most people who search for meta tags robots want one thing: a way to keep a page out of Google. The tag is a page-level instruction for crawlers. It sits in the <head> of an HTML page, and its content attribute holds one or more rules separated by commas. Here's the version you'll see most often:

<meta name="robots" content="noindex, nofollow">

This asks every crawler that supports these rules not to show the page in search results and not to follow its links. Both the name and content values are case-insensitive, so NOINDEX works too.

Without any tag, the default is all, which means index the page and follow its links. So you only add a robots tag when you want to restrict something. Google says it also respects the tag in the body, but the head is still the right place for it. Our guide to which meta tags matter for SEO covers the other tags that sit beside it.

Every Robots Directive Google Supports, in One Table

Google's robots meta tag specification lists these rules as of November 2026. You can combine them in one comma-separated list or spread them across several tags.

RuleWhat it doesNotes
allNo restrictions on indexing or servingThe default, so listing it has no effect
noindexKeeps the page out of search resultsGoogle must crawl the page to see it
nofollowAsks Google not to follow links on the pageGoogle's docs call it a hint
noneSame as noindex, nofollowA shorthand
nosnippetNo text snippet or video previewAlso stops the text being used as direct input for AI Overviews and AI Mode
indexifembeddedAllows indexing when the page is embedded through an iframeOnly works together with noindex
max-snippet:[number]Caps the text snippet at that many characters0 equals nosnippet; -1 lets Google choose
max-image-preview:[setting]Caps the image preview sizeAccepts none, standard or large
max-video-preview:[number]Caps video previews at that many seconds0 allows a static image only; -1 means no limit
notranslateNo translation offer in search resultsOtherwise Google may translate the title link and snippet
noimageindexImages on the page aren't indexedThe page's text can still be indexed
unavailable_after:[date/time]Drops the page from results after that dateAccepts formats such as RFC 822, RFC 850 or ISO 8601

Older guides still list three rules that Google now ignores:

  • noarchive controlled the cached link, a feature Google has removed.
  • nocache isn't used by Google Search at all.
  • nositelinkssearchbox controlled the sitelinks search box, which no longer exists.

Google's spec also warns that other search engines may treat these rules differently, so check each engine's own docs if you care about Bing or others. To hide just part of a page from snippets, put the data-nosnippet attribute on a span, div or section element. Google ignores it on any other element.

How Do Noindex Meta Tags Work?

Noindex meta tags work only after Googlebot crawls the page and reads the rule. Per Google's noindex documentation, the page then drops out of results entirely, even if other sites link to it. The rule doesn't stop crawling, and anyone with the URL can still open the page.

<meta name="robots" content="noindex, follow">

This keeps the page out of results while Google may still follow its links. We prefer plain noindex, though. Following links is already the default, so follow adds nothing except one more word for the next maintainer to second-guess.

Removal isn't instant. A page stays listed until Google recrawls it, and Google says that can take months for pages it considers less important. The URL Inspection tool in Search Console lets you request a recrawl and shows whether Googlebot actually saw your noindex.

Targeting One Crawler and Resolving Conflicting Rules

Swap robots for a crawler's user agent token to aim rules at that crawler alone. Google accepts two tokens in the meta tag: googlebot for text results and googlebot-news for news results. It ignores any other value in the name attribute.

When two rules conflict, Google applies the more restrictive one. A page carrying both max-snippet:50 and nosnippet gets no snippet at all. When separate tags target different crawlers, Google adds up the negative rules:

<meta name="robots" content="nofollow">
<meta name="googlebot" content="noindex">

Googlebot treats this page as noindex, nofollow. A crawler that only reads the robots tag sees just nofollow.

Page-level nofollow isn't the same as rel="nofollow" on one link. The meta rule covers every link on the page, while the attribute marks a single link. Google treats nofollow as a hint rather than a strict command, so don't rely on it to keep a linked page out of the index.

X-Robots-Tag: The Header Version for PDFs and Other Files

An X-Robots-Tag HTTP response header carries the same rules as the meta tag. Any rule from the table works there. It's also how you apply these rules to PDFs, images and videos, which have no head to hold a tag.

X-Robots-Tag: noindex
X-Robots-Tag: googlebot: nofollow

The first line applies to every crawler. The second, prefixed with a user agent, applies to Googlebot only. To noindex every PDF on an Apache site, Google's docs give this rule for .htaccess or httpd.conf:

<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>

On NGINX, the same rule goes in the site's .conf file:

location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}

So which takes precedence when a page sends both the header and a meta tag? Google's spec doesn't rank one source above the other. It says conflicting rules resolve to the more restrictive one, so a noindex in either place keeps the page out.

Is Noindex the Same as Disallow in robots.txt?

No. A Disallow line in robots.txt stops crawling, while noindex stops indexing. Google says a page it can't crawl may still appear in results if other pages link to it.

The trap is using both on the same URL. Google can't read a noindex on a page it isn't allowed to fetch, so the rule never takes effect. With these lines in robots.txt, a noindex tag on any page under /drafts/ goes unread:

User-agent: *
Disallow: /drafts/

Leave the page crawlable so Google can keep reading the noindex. Google doesn't support noindex inside robots.txt either.

Noindexing Draft Pages in Docusaurus, MkDocs, and Next.js

Markdown docs sites usually set per-page options in front matter, the YAML block at the top of a file. If that format is new to you, start with our Markdown front matter guide. Docusaurus and MkDocs read robots settings from it, while Next.js uses a metadata object in code. Our editor's preview doesn't render front matter, so check the robots tag in your built site, not in a Markdown preview.

Docusaurus

Docusaurus docs pages accept two front matter flags. draft: true keeps a page out of production builds entirely, so there's nothing to index. unlisted: true builds the page but hides it: the docs say it isn't indexed, stays out of sitemaps and opens only by direct link.

---
title: Release checklist (internal)
unlisted: true
---

For an explicit tag on a single page, add a head block to the Markdown file. Docusaurus merges its contents into the page's real head:

<head>
  <meta name="robots" content="noindex" />
</head>

For a staging copy of a whole site, set noIndex: true in docusaurus.config.js. Per the Docusaurus config reference, that adds noindex, nofollow to every page. Our Docusaurus Markdown guide covers the rest of its syntax.

MkDocs with Material

MkDocs passes front matter to templates as page.meta, and Material for MkDocs provides an empty extrahead block for custom meta tags. First point custom_dir at an overrides folder in mkdocs.yml:

theme:
  name: material
  custom_dir: overrides

Then create overrides/main.html, which extends the theme and prints a robots tag only when a page asks for one:

{% extends "base.html" %}
{% block extrahead %}
  {% if page.meta and page.meta.robots %}
    <meta name="robots" content="{{ page.meta.robots }}">
  {% endif %}
{% endblock %}

Now any page with this front matter gets noindex in its head when MkDocs builds the site:

---
robots: noindex
---

Work in progress: the new API reference.

Our MkDocs Markdown guide walks through installing and configuring MkDocs itself.

Next.js App Router

In Next.js, export a metadata object from the page or layout:

export const metadata = {
  robots: { index: false },
}

Next.js renders that as <meta name="robots" content="noindex" />. Add follow: false to get noindex, nofollow. The Next.js metadata reference lists the other robots fields, including a separate googleBot object.

Do You Need a Robots Meta Tag on Every Page?

No. Pages you want in search need no robots tag at all, because indexing and link following are the defaults. Add one only where a page shouldn't show up in results:

  • Internal site search results and filtered listings
  • Thank-you and order confirmation pages
  • Staging and preview copies of your site
  • Thin tag or archive pages
  • Drafts that must be live for review

Keep two cautions in mind. The tag is a request that well-behaved crawlers honour, not access control, so anything private needs a password. And watch staging configs: a site-wide noindex copied to production can quietly pull an entire site out of search. After each launch, check the meta tags and response headers on a live page.

Writing a head block by hand? Our meta tag generator has index and follow toggles that write the robots line next to your description and Open Graph tags. You paste its output into your own template.

Robots Meta Tag FAQ

The robots meta tag is small, but a single wrong word can drop a page or a whole site from search. Use noindex for pages that shouldn't rank, the X-Robots-Tag header for files, and never block those URLs in robots.txt. Draft the Markdown for your docs pages in our free editor, with no account needed, and let your site generator add the tag at build time.