What is the Robots Meta Tag?
The robots meta tag is an HTML element placed in the head of a page that gives search engines instructions about how to handle that specific page — most importantly, whether to index it and whether to follow its links. It is the page-level counterpart to the site-wide robots.txt file, and it is the standard way to control indexing of individual pages. Used correctly, the robots meta tag is essential for managing exactly which pages appear in search; used carelessly, it is one of the most common causes of pages silently disappearing from Google.
What does the robots meta tag control?
The robots meta tag controls two main behaviours through its directives: whether a page is indexed and whether its links are followed. The index or noindex directive tells Google whether to include the page in its index, and the follow or nofollow directive tells Google whether to follow the links on the page and pass authority through them. These combine to give precise control over a single page's treatment.
The most consequential directive is noindex, which removes a page from search results. When a page carries a noindex directive, Google keeps it out of the index entirely, so it cannot appear for any query. This is the intended tool for keeping specific pages — thank-you pages, internal search results, low-value pages — out of search while leaving them accessible to users.
Additional directives control finer behaviours. Directives can tell Google not to show a cached version, not to use a snippet, or to limit how content is displayed in results. These are less commonly needed but useful in specific cases, and together with the core index and follow directives they make the robots meta tag a flexible, page-level control covered as part of technical SEO.
How does the robots meta tag differ from robots.txt?
The robots meta tag and robots.txt are often confused but do different things. Robots.txt is a single site-wide file that controls crawling — whether Google is allowed to access pages and directories. The robots meta tag is placed on individual pages and primarily controls indexing — whether a page that has been crawled is included in the index. One governs access at the site level; the other governs indexing at the page level.
The crucial interaction is that a page blocked in robots.txt cannot have its robots meta tag read. If robots.txt prevents Google from crawling a page, Google never sees the page's content, including its noindex directive, so the noindex is never applied. This is a common and damaging mistake: blocking a page in robots.txt to keep it out of search can actually leave it indexed without a snippet, because Google cannot read the noindex.
The correct approach depends on the goal. To keep a page out of the index, it must be crawlable so Google can read its noindex directive — so it should not be blocked in robots.txt. To save crawl budget by preventing crawling entirely, robots.txt is the tool, but it does not guarantee the page stays out of the index. Understanding this distinction is essential to using both tools correctly, as the Wix not indexed by Google guide explains.
What are the common robots meta tag mistakes?
The most damaging mistake is an accidental noindex on important pages. A noindex directive left over from development, applied site-wide by mistake, or added to the wrong template can silently remove valuable pages from search without breaking anything visible. Because the page still works for users, the problem often goes unnoticed until rankings and traffic drop, which makes accidental noindex one of the first things to check when pages vanish from search.
Conflicting directives cause confusion. A page with a noindex robots meta tag that is also blocked in robots.txt creates the contradiction described above, where the noindex cannot be read. Similarly, a noindex directive combined with a canonical URLs tag pointing elsewhere sends mixed signals. Keeping the directives consistent and coherent is essential to predictable behaviour.
Carrying staging directives to production is a frequent migration error. Development and staging sites are often set to noindex site-wide to keep them out of search, and if that directive carries over when the site goes live, the entire production site becomes non-indexable. Verifying that the live site does not carry a site-wide noindex is a standard post-launch check, part of the care any website migration requires.
How do you use the robots meta tag correctly?
Using the robots meta tag correctly starts with a clear decision about which pages should and should not be indexed. Valuable pages that should rank are left to index normally; low-value pages that should stay out of search — internal search results, thank-you pages, thin utility pages — receive a noindex directive. This deliberate index management keeps the index clean and focused on pages worth ranking.
Pages to be noindexed must remain crawlable. Because Google has to read the page to see its noindex directive, the page must not be blocked in robots.txt. Leaving noindexed pages crawlable, with the follow directive intact so their links still pass authority, is the standard configuration for keeping a page out of search while preserving its role in the site's link structure.
Verification through Google Search Console confirms the directives work as intended. The URL Inspection tool shows whether Google sees a page as indexable or noindexed, which catches both accidental noindex on valuable pages and missing noindex on pages that should be excluded. Checking the directives after any significant site change is part of the technical hygiene covered in the Wix SEO diagnostic checklist.
When does the robots meta tag matter most?
The robots meta tag matters most for managing the index on sites with pages that should not appear in search. Any site with utility pages, thank-you pages, internal search results, or low-value pages benefits from deliberate noindex directives that keep these out of the index while leaving them accessible to users. This index management becomes more important as a site grows and accumulates pages of varying value.
It is critical during and after migrations and launches. The risk of an accidental site-wide noindex carried over from staging makes verifying the robots meta tags one of the highest-priority post-launch checks. A single misplaced directive can take an entire site out of search, so confirming the live site indexes correctly is essential whenever a site is launched or migrated.
For ongoing SEO, the robots meta tag is checked whenever pages unexpectedly appear in or disappear from search. A valuable page missing from the index, or a low-value page cluttering it, often traces to a robots meta tag directive. A free SEO scan can establish whether a site's indexing directives are configured correctly and whether any valuable pages are being accidentally excluded.
On this page
Do you need help managing indexing?
A misplaced robots meta tag can silently remove your pages from Google, or leave low-value pages cluttering search. We Optimizz audits and configures indexing directives across Wix Studio, WordPress, Framer, Webflow, and Shopify. 894 websites delivered across 35+ countries.
