SEO Robots file

A text file placed in a site's root directory, built with specific syntax, instructing some or all spiders to ignore certain pages or directories. Some spiders respect these instructions, others disregard them.