Skip to content
SEO Worth
[ META TAGS ]

Robots Meta Tag: Noindex, Nofollow & Every Directive

What the robots meta tag does, every directive Google supports, the noindex vs nofollow vs disallow difference, and recipes that do not backfire.

Robots Meta Tag: Noindex, Nofollow & Every Directive

The robots meta tag is a single line in a page’s <head> that tells search engines how to treat that one page: index it or not, follow its links or not, and how the result may look. It looks like this:

<meta name="robots" content="noindex, nofollow" />

It is the most consequential meta tag on the web, because it is the only one that can make a page disappear. This is the full reference: every directive Google supports, the three-way confusion between noindex, nofollow and robots.txt, the recipes worth copying, and the one mistake that quietly keeps unwanted pages in the index for months.

Syntax and placement

The tag goes in the <head>, before any content, and applies only to the page that contains it.

<head>
    <meta name="robots" content="noindex" />
</head>

Three rules cover the syntax:

  • name selects the audience. robots means every crawler that honours the standard. Replacing it with a specific crawler token targets one of them: googlebot, googlebot-news, bingbot, AdsBot-Google.
  • content holds the directives, comma-separated, case-insensitive. NOINDEX, NOFOLLOW and noindex, nofollow are the same instruction.
  • Most restrictive wins. If a page carries several robots meta tags, or a meta tag and an X-Robots-Tag header, crawlers combine them and the strictest directive applies. noindex in one tag and index in another resolves to noindex.

One exception matters: if a page has both a generic robots tag and a googlebot tag, Google obeys the googlebot one and ignores the generic tag entirely. It does not merge them.

Every directive Google supports

DirectiveWhat it does
allNo restrictions. The default, same as index, follow
indexIndex this page. Default, so writing it changes nothing
followFollow links on this page. Default, so writing it changes nothing
noindexDo not show this page in search results
nofollowDo not follow the links on this page
noneShorthand for noindex, nofollow
noarchiveDo not offer a cached copy of the page
nosnippetNo text snippet and no video preview in the result
max-snippet:[n]Cap the text snippet at n characters. 0 means none, -1 means no cap
max-image-preview:[setting]none, standard or large
max-video-preview:[n]Cap video preview at n seconds. 0 means a static image, -1 no cap
noimageindexDo not index images hosted on this page
notranslateDo not offer a translation of this result
unavailable_after:[date]Stop showing the page after a date (RFC 822, RFC 850 or ISO 8601)
indexifembeddedWith noindex, still allow indexing when embedded via iframe

The four max-* and no* display directives are the underused half of this list. They do not affect whether you rank, only how much of your page Google is allowed to show. max-snippet:0 on a paywalled article, or max-image-preview:large on a recipe site that wants the big thumbnail in Discover, are both real, specific choices most sites never make.

noindex vs nofollow vs disallow

This is where almost every robots mistake starts. The three controls answer different questions and live in different places.

ControlWhere it livesWhat it controlsPage still crawled?
noindex meta tagThe page’s <head>Whether the page is indexedYes, it must be to be read
nofollow meta tagThe page’s <head>Whether its outbound links countYes
Disallow: in robots.txt/robots.txtWhether the URL is crawledNo
X-Robots-Tag headerHTTP responseSame as the meta tagYes

Read the last column twice. noindex requires a crawl. It is an instruction Google can only obey after fetching the page and reading the <head>.

The correct removal sequence is the opposite of most people’s instinct. Add noindex, leave the page crawlable, wait until Search Console confirms the pages have dropped out, and only then add a robots.txt disallow if you also want to save the crawl budget.

A related note: Google stopped honouring noindex: directives inside robots.txt on 1 September 2019. If you inherited a robots.txt that still contains them, they have been doing nothing for years.

Targeting a specific crawler

Swap the name value for a crawler token to give one bot different rules.

<meta name="robots" content="noindex" /> <meta name="googlebot" content="all" />

Those two lines together mean: every crawler except Google should skip this page, and Google should index it normally. Google reads its own tag and ignores the generic one.

name valueApplies to
robotsEvery crawler that respects the standard
googlebotGoogle Search’s main crawler
googlebot-newsGoogle News only
bingbotBing
AdsBot-GoogleGoogle Ads landing page quality checks

The common real use is <meta name="googlebot-news" content="noindex"> on evergreen pages of a news site, so they stay in web search but out of the News surface.

When to use X-Robots-Tag instead

The meta tag only works in HTML. If you need to control a PDF, an image, a CSV export or an entire directory, use the X-Robots-Tag HTTP response header, which accepts exactly the same directives.

X-Robots-Tag: noindex, noarchive

Two cases where the header is the only option: non-HTML files, and rules you want applied in bulk at the server or CDN layer rather than edited into thousands of templates. The trade-off is visibility. A header is invisible in “view source”, which makes it a frequent cause of “why is this page not indexing and there is no noindex anywhere”. Check the headers before you conclude a page is clean.

Recipes worth copying

SituationTag
Thank-you / confirmation page<meta name="robots" content="noindex, follow" />
Internal search results page<meta name="robots" content="noindex, follow" />
Staging or preview environment<meta name="robots" content="noindex, nofollow" />
Paginated archive pagesLeave default; canonicalize, do not noindex
Filtered or faceted listing<meta name="robots" content="noindex, follow" />
Expiring job or event listing<meta name="robots" content="unavailable_after: 2026-12-31T23:59:59+00:00" />
Paywalled article<meta name="robots" content="max-snippet:0" />
Gallery page whose images belong to you<meta name="robots" content="noimageindex" />
Author or tag archive with one post<meta name="robots" content="noindex, follow" />

Notice how often noindex, follow appears. It is the workhorse combination: keep the page out of results, but let the link equity and the crawl path flow through it to the pages you do want indexed. Reaching for none or noindex, nofollow on an internal page usually throws away crawl paths for no benefit.

Server racks in a data centre with status lights, representing crawler requests hitting a web server
Robots directives are read on every crawl, so a change takes effect on the next fetch, not the moment you deploy.

How to check a page for noindex

  1. View source and search the <head> for name="robots" and name="googlebot". Look at the rendered DOM too if the site is JavaScript-heavy, since a framework can inject or remove the tag after load.
  2. Check the response headers for X-Robots-Tag. curl -sI https://example.com/page/ takes two seconds and catches the invisible case.
  3. Run URL Inspection in Search Console. It reports the exact indexing decision Google made, which beats guessing from the HTML. This is also where “Excluded by ‘noindex’ tag” shows up in the Pages report.
  4. Check robots.txt separately. If the URL is disallowed, the noindex question is academic: Google cannot read the tag at all.
  5. Crawl the site. One template bug can noindex a whole section, and a full crawl is the only way to see the pattern rather than one symptom.

Step 4 catches the most expensive version of this problem, which is a site that was noindexed during a rebuild and never un-noindexed. It is worth adding a check for noindex in the production <head> to your deploy pipeline, alongside whatever else you assert about the head tags.

Where it sits among the other meta tags

The robots tag is the only meta tag that can cost you all of your traffic, which is why it belongs at the top of any technical review. The rest of the head is lower stakes: the title tag decides which queries you compete for, the meta description influences click-through rate, and the canonical resolves duplicates.

For the priority order across every head-level signal, see meta tags for SEO. For the syntax of every tag that belongs in a <head>, including charset, viewport and Open Graph, see the meta tags in HTML reference.

The bottom line

The robots meta tag controls one page, and it only works if that page can be crawled. Use noindex, follow for pages that should not rank but should still pass crawl paths, never pair a noindex with a robots.txt disallow, drop the pointless index, follow your CMS writes everywhere, and check the X-Robots-Tag header before you conclude a page is clean. Then put the rest of the head in order with the on-page SEO basics checklist.

Want technical SEO references like this in your inbox? Join the newsletter for concise, practical SEO you can apply the same day.

[ FAQ ]
What does the robots meta tag do?
It tells search engine crawlers how to treat a single page: whether to index it, whether to follow the links on it, and how the result may be displayed. It lives in the page's `<head>` and applies only to that page. Sitewide rules belong in robots.txt or in the X-Robots-Tag HTTP header.
What is the difference between noindex and nofollow?
`noindex` means do not put this page in search results. `nofollow` means do not follow the links on this page. They are independent: a page can be indexed with nofollowed links, or excluded from the index while its links are still followed. `noindex, follow` is the combination most people actually want on paginated and filtered pages.
Should I use noindex or robots.txt disallow?
Use noindex when the page must not appear in search results. Use robots.txt disallow when you want to save crawl budget on pages you do not care about either way. Never use both on the same URL: a disallowed page cannot be crawled, so Google never reads the noindex and the URL can stay in the index as a bare link.
Do I need to add index, follow to my pages?
No. `index` and `follow` are the default behaviour and Google ignores them. A tag that only says `index, follow` has no effect. It is harmless, but it is also noise, and it frequently hides the fact that a templating bug is writing the same tag on pages that should be noindexed.
How do I check whether a page is noindexed?
View the page source and search for `name="robots"` and `name="googlebot"`, then check the HTTP response headers for `X-Robots-Tag`, which is easy to miss because it never appears in the HTML. Confirm with the URL Inspection tool in Search Console, which reports the exact indexing decision Google made and why.
How long does it take for noindex to remove a page from Google?
It takes effect the next time Google crawls the page, which can be days or weeks depending on how often that URL is fetched. If you need it gone faster, request removal in Search Console for a temporary block (about six months) while the noindex does the permanent work.

[ READ NEXT ]