X-Robots-Tag in HTTP Header

Robots Meta tag is implemented to control the crawling and indexing of that particular page. But in some cases, the implementation of a Robots Meta tag is not possible. The reasons can be

  • a) The file type is non-HTML. Meta Robots tag can’t be implemented .doc, .pdf, .xls, video or audio files.
  • b) where the editing of the code or content of an HTML file is not possible.
  • c) Sites with large amounts of changing content, where updating individual files is too hard or expensive.

In such a case, X-Robots-Tag is the solution. An X-Robots-Tag is an element in HTTP header which can contain same values as the Robots META tags.
This can be done using by sever-side tools so that this information is generated by web server.
Given below is the example to avoid indexing .doc files through .htaccess.

Header set X-Robots-Tag “NOINDEX, NOFOLLOW”

Want to boost your online visibility? Contact SEO India Company, Profit By Search, Today!


Speak with us

USA: +1-888-322-7617
  • UK: +44-161-870-6437
  • AU: +61-261-452-356
  • In: +91 (0)1204338720
Get a Free Audit