#robotstxt

Prevent pages from being indexed: robots.txt vs noindex