Robots Meta Tag: noindex, nofollow and Every Rule
November 11, 2026 · 11 min read
Robots Meta Tag: noindex, nofollow, and Every Rule Google Reads
A robots meta tag is a <meta name="robots"> element in a page's head that tells search engines whether to index it, follow its links and show a snippet. Add noindex to keep a page out of results. Below is every rule Google honours today, how conflicts resolve, and how to noindex drafts on docs sites.
What Is a Robots Meta Tag?
Most people who search for meta tags robots want one thing: a way to keep a page out of Google. The tag is a page-level instruction for crawlers. It sits in the <head> of an HTML page, and its content attribute holds one or more rules separated by commas. Here's the version you'll see most often:
<meta name="robots" content="noindex, nofollow">
This asks every crawler that supports these rules not to show the page in search results and not to follow its links. Both the name and content values are case-insensitive, so NOINDEX works too.
Without any tag, the default is all, which means index the page and follow its links. So you only add a robots tag when you want to restrict something. Google says it also respects the tag in the body, but the head is still the right place for it. Our guide to which meta tags matter for SEO covers the other tags that sit beside it.
Every Robots Directive Google Supports, in One Table
Google's robots meta tag specification lists these rules as of November 2026. You can combine them in one comma-separated list or spread them across several tags.
| Rule | What it does | Notes |
|---|---|---|
all | No restrictions on indexing or serving | The default, so listing it has no effect |
noindex | Keeps the page out of search results | Google must crawl the page to see it |
nofollow | Asks Google not to follow links on the page | Google's docs call it a hint |
none | Same as noindex, nofollow | A shorthand |
nosnippet | No text snippet or video preview | Also stops the text being used as direct input for AI Overviews and AI Mode |
indexifembedded | Allows indexing when the page is embedded through an iframe | Only works together with noindex |
max-snippet:[number] | Caps the text snippet at that many characters | 0 equals nosnippet; -1 lets Google choose |
max-image-preview:[setting] | Caps the image preview size | Accepts none, standard or large |
max-video-preview:[number] | Caps video previews at that many seconds | 0 allows a static image only; -1 means no limit |
notranslate | No translation offer in search results | Otherwise Google may translate the title link and snippet |
noimageindex | Images on the page aren't indexed | The page's text can still be indexed |
unavailable_after:[date/time] | Drops the page from results after that date | Accepts formats such as RFC 822, RFC 850 or ISO 8601 |
Older guides still list three rules that Google now ignores:
noarchivecontrolled the cached link, a feature Google has removed.nocacheisn't used by Google Search at all.nositelinkssearchboxcontrolled the sitelinks search box, which no longer exists.
Google's spec also warns that other search engines may treat these rules differently, so check each engine's own docs if you care about Bing or others. To hide just part of a page from snippets, put the data-nosnippet attribute on a span, div or section element. Google ignores it on any other element.
How Do Noindex Meta Tags Work?
Noindex meta tags work only after Googlebot crawls the page and reads the rule. Per Google's noindex documentation, the page then drops out of results entirely, even if other sites link to it. The rule doesn't stop crawling, and anyone with the URL can still open the page.
<meta name="robots" content="noindex, follow">
This keeps the page out of results while Google may still follow its links. We prefer plain noindex, though. Following links is already the default, so follow adds nothing except one more word for the next maintainer to second-guess.
Removal isn't instant. A page stays listed until Google recrawls it, and Google says that can take months for pages it considers less important. The URL Inspection tool in Search Console lets you request a recrawl and shows whether Googlebot actually saw your noindex.
Targeting One Crawler and Resolving Conflicting Rules
Swap robots for a crawler's user agent token to aim rules at that crawler alone. Google accepts two tokens in the meta tag: googlebot for text results and googlebot-news for news results. It ignores any other value in the name attribute.
When two rules conflict, Google applies the more restrictive one. A page carrying both max-snippet:50 and nosnippet gets no snippet at all. When separate tags target different crawlers, Google adds up the negative rules:
<meta name="robots" content="nofollow">
<meta name="googlebot" content="noindex">
Googlebot treats this page as noindex, nofollow. A crawler that only reads the robots tag sees just nofollow.
Page-level nofollow isn't the same as rel="nofollow" on one link. The meta rule covers every link on the page, while the attribute marks a single link. Google treats nofollow as a hint rather than a strict command, so don't rely on it to keep a linked page out of the index.
X-Robots-Tag: The Header Version for PDFs and Other Files
An X-Robots-Tag HTTP response header carries the same rules as the meta tag. Any rule from the table works there. It's also how you apply these rules to PDFs, images and videos, which have no head to hold a tag.
X-Robots-Tag: noindex
X-Robots-Tag: googlebot: nofollow
The first line applies to every crawler. The second, prefixed with a user agent, applies to Googlebot only. To noindex every PDF on an Apache site, Google's docs give this rule for .htaccess or httpd.conf:
<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>
On NGINX, the same rule goes in the site's .conf file:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}
So which takes precedence when a page sends both the header and a meta tag? Google's spec doesn't rank one source above the other. It says conflicting rules resolve to the more restrictive one, so a noindex in either place keeps the page out.
Is Noindex the Same as Disallow in robots.txt?
No. A Disallow line in robots.txt stops crawling, while noindex stops indexing. Google says a page it can't crawl may still appear in results if other pages link to it.
The trap is using both on the same URL. Google can't read a noindex on a page it isn't allowed to fetch, so the rule never takes effect. With these lines in robots.txt, a noindex tag on any page under /drafts/ goes unread:
User-agent: *
Disallow: /drafts/
Leave the page crawlable so Google can keep reading the noindex. Google doesn't support noindex inside robots.txt either.
Noindexing Draft Pages in Docusaurus, MkDocs, and Next.js
Markdown docs sites usually set per-page options in front matter, the YAML block at the top of a file. If that format is new to you, start with our Markdown front matter guide. Docusaurus and MkDocs read robots settings from it, while Next.js uses a metadata object in code. Our editor's preview doesn't render front matter, so check the robots tag in your built site, not in a Markdown preview.
Docusaurus
Docusaurus docs pages accept two front matter flags. draft: true keeps a page out of production builds entirely, so there's nothing to index. unlisted: true builds the page but hides it: the docs say it isn't indexed, stays out of sitemaps and opens only by direct link.
---
title: Release checklist (internal)
unlisted: true
---
For an explicit tag on a single page, add a head block to the Markdown file. Docusaurus merges its contents into the page's real head:
<head>
<meta name="robots" content="noindex" />
</head>
For a staging copy of a whole site, set noIndex: true in docusaurus.config.js. Per the Docusaurus config reference, that adds noindex, nofollow to every page. Our Docusaurus Markdown guide covers the rest of its syntax.
MkDocs with Material
MkDocs passes front matter to templates as page.meta, and Material for MkDocs provides an empty extrahead block for custom meta tags. First point custom_dir at an overrides folder in mkdocs.yml:
theme:
name: material
custom_dir: overrides
Then create overrides/main.html, which extends the theme and prints a robots tag only when a page asks for one:
{% extends "base.html" %}
{% block extrahead %}
{% if page.meta and page.meta.robots %}
<meta name="robots" content="{{ page.meta.robots }}">
{% endif %}
{% endblock %}
Now any page with this front matter gets noindex in its head when MkDocs builds the site:
---
robots: noindex
---
Work in progress: the new API reference.
Our MkDocs Markdown guide walks through installing and configuring MkDocs itself.
Next.js App Router
In Next.js, export a metadata object from the page or layout:
export const metadata = {
robots: { index: false },
}
Next.js renders that as <meta name="robots" content="noindex" />. Add follow: false to get noindex, nofollow. The Next.js metadata reference lists the other robots fields, including a separate googleBot object.
Do You Need a Robots Meta Tag on Every Page?
No. Pages you want in search need no robots tag at all, because indexing and link following are the defaults. Add one only where a page shouldn't show up in results:
- Internal site search results and filtered listings
- Thank-you and order confirmation pages
- Staging and preview copies of your site
- Thin tag or archive pages
- Drafts that must be live for review
Keep two cautions in mind. The tag is a request that well-behaved crawlers honour, not access control, so anything private needs a password. And watch staging configs: a site-wide noindex copied to production can quietly pull an entire site out of search. After each launch, check the meta tags and response headers on a live page.
Writing a head block by hand? Our meta tag generator has index and follow toggles that write the robots line next to your description and Open Graph tags. You paste its output into your own template.
Robots Meta Tag FAQ
The robots meta tag is small, but a single wrong word can drop a page or a whole site from search. Use noindex for pages that shouldn't rank, the X-Robots-Tag header for files, and never block those URLs in robots.txt. Draft the Markdown for your docs pages in our free editor, with no account needed, and let your site generator add the tag at build time.