Google Wants to Establish an Official Standard for Using Robots.txt via @MattGSouthern
Google has proposed an official internet standard for the rules included in robots.txt files.Those rules, outlined in the Robots Exclusion Protocol (REP), have been an unofficial standard for the past 25 years.While the REP has been adopted by search engines…
