Robots.txt - SEO Word of the Day

Google updated the webmaster guidelines, following a comment Pat of Feedthebot.com made during the You & A with Matt Cutts at SMX Advanced in Seattle this week.

This comes a few months after Barry Schwartz of Search Engine Roundtable discovered a new guideline back in March.

What was the new Google webmaster guideline?
Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.

This is really interesting to me, because I’ve heard Google mention the poor quality of search result pages on several occasions. It looks like what’s good for the user is NOT always good for the engines. Still, I get the point, the last thing someone wants when they’re searching for something is to search again once they think they’ve found what they were originally searching for. Like always, there are exceptions to the rule. If the search results pages have unique content and strong categories or other added value, they might actually rank well, so take everything (even the guidelines) with a grain of salt.

What sparked the most recent Google webmaster guidelines update?
Clarification of the guidelines stemmed from Pat’s question: Why are Google guidelines so brief and what does the future hold? Matt Cutts’ response quickly lost the audience by referencing piano’s axioms and group theory in mathematics. The bottom line, Google’s philosophy is to only provide a minimal set of information to enable users to make educated inferences based on their unique situation. Concise guidelines also provides greater flexibility for Google as they do not have to micro-manage each item.

Obviously, Google felt they could throw the SEOs a bone and flushed the guidelines out just a bit more.

For those that were really curious about Piano axioms (or maybe that was just me):

“The seven Piano axioms of integer arithmetic correctly describe an infinite cardinality of integers. Only the integers that are beyond infinity fall in the collection of under specified integers. If we are willing to limit ourselves to an infinite number of integers that are less than infinity, we can establish the requirements for integer arithmetic in a finite manor. Though the requirements may not theoretically describe that system, the coverage may be quite satisfactory.”

(Ronald LeRoi Burback, Stanford University)

For complete coverage of the organic sessions check out my posts on Search Engine Journal.

Share This