Meta Rules for Web Robots Explained

Published on 10 May 2012 at 8:19 pm.
Filed under Google,Informative,Search Engine Optimization.

Controlling what pages search engines index on your site is a critical part of SEO. So how can meta tags help give you this control?

Search engines know that web authors want control over what is displayed in search results. There are a few ways in which author’s can carry this out. One of the most well-known is the famous robots.txt file. A robots.txt file is a quick way to tell search engines to not crawl or index any content listed in said file, but it might still index the URL for the page if it is linked on another website. To entirely prevent the page’s content from displaying on the web you are going to need to implement the robots meta tag.

Robots Meta Tag: Noindex

The noindex value for the robots meta tag tells search engines not to index the content on any of the pages. Depending on the situation, the noindex meta tag route might be more useful than editing your robots.txt file if the page is in a section that you might normally want indexed. It also provides a fallback solution if you forgot to add the rule to your robots.txt file.

The syntax for this meta tag looks as follows:

<meta name="robots" value="noindex" />

How Twitter should implement nofollow

Robots Meta Tag: Nofollow

The nofollow value for the robots meta tag tells search engines not to follow any of the links that appear on that page. It tells search engines not to distribute any of the value of your page’s rank to said links. This does not mean that the content that is linked to is hidden because other pages on the web that do not use nofollow may link to them.

The syntax for this meta tag looks as follows:

<meta name="robots" value="nofollow" />

Robots Meta Tag: Noarchive

The noarchive value for the robots meta tag tells search engines not to cache the page’s content. This is useful when you have content that is updated several times a day and a cached copy is likely to be out of date. Telling a crawler not to keep a copy ensures that the user will click on your site and not the index and thus they will see the most up to date information.

The syntax for this meta tag looks as follows:

<meta name="robots" value="noarchive" />

Robots Meta Tag: Nosnippet

The nosnippet value for the robots meta tag tells search engines not to display a text exert under the page title on the search engine results page. Disabling a snippet in Google will also tell Google to not display a text link to an Instant Preview, nor cache the that page.

The syntax for this meta tag looks as follows:

<meta name="robots" value="nosnippet" />

The meta robots tag can only be used once on a page. If you would like to implement multiple values for the commands above you can do so by editing the “value” attribute above and placing multiple keywords in there, separated by a comma. For example, if you would like to implement noindex and nofollow you would use:

<meta name="robots" value="noindex,nofollow" />

Now that you know how to prevent search engines from indexing the wrong parts of your site you may start to see your site’s ranking improve. Go forth and take control of your website.

This post was originally published as Meta Rules for Web Robots Explained for The BrandBuilder Company.