Robots.txt – SEO Word of the Day

Google updated the webmaster guidelines, following a comment Pat of made during the You & A with Matt Cutts at SMX Advanced in Seattle this week.

This comes a few months after Barry Schwartz of Search Engine Roundtable discovered a new guideline back in March.

What was the new Google webmaster guideline?
Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.

This is really interesting to me, because I’ve heard Google mention the poor quality of search result pages on several occasions. It looks like what’s good for the user is NOT always good for the engines. Still, I get the point, the last thing someone wants when they’re searching for something is to search again once they think they’ve found what they were originally searching for. Like always, there are exceptions to the rule. If the search results pages have unique content and strong categories or other added value, they might actually rank well, so take everything (even the guidelines) with a grain of salt.

What sparked the most recent Google webmaster guidelines update?
Clarification of the guidelines stemmed from Pat’s question: Why are Google guidelines so brief and what does the future hold? Matt Cutts’ response quickly lost the audience by referencing piano’s axioms and group theory in mathematics. The bottom line, Google’s philosophy is to only provide a minimal set of information to enable users to make educated inferences based on their unique situation. Concise guidelines also provides greater flexibility for Google as they do not have to micro-manage each item.

Obviously, Google felt they could throw the SEOs a bone and flushed the guidelines out just a bit more.

For those that were really curious about Piano axioms (or maybe that was just me):

“The seven Piano axioms of integer arithmetic correctly describe an infinite cardinality of integers. Only the integers that are beyond infinity fall in the collection of under specified integers. If we are willing to limit ourselves to an infinite number of integers that are less than infinity, we can establish the requirements for integer arithmetic in a finite manor. Though the requirements may not theoretically describe that system, the coverage may be quite satisfactory.”

(Ronald LeRoi Burback, Stanford University)

For complete coverage of the organic sessions check out my posts on Search Engine Journal.

Be Sociable, Share!


  1. avatar pat Says:

    Wow, you are quick. Why is every SEO hottie either married or about to be? Snif.
    I think I will be going to Toronto, you going?

  2. avatar pat Says:

    Hello, I am afraid I steered you wrong, I was not the discoverer of this, Barry Schwartz at Search Engine Roundtable noted it when it first came out, in (this hurts) mid March. I really havent heard any news or discussion of it though, so I hope it is still a reminder to those who haven’t seen it before.

  3. avatar Carsten Cumbrowski Says:

    Tss. so they don’t want to be a bit more specific that maybe some people outside Google would be able to interpret it, because they think that if they start getting a bit more specific that they will have to be specific to every single theoretically possible combination involving every theoretically possible case?!

    That’s what you get with mathematicians logic.

    It’s like saying that we can’t build the house, because the measuring sticks are only accurate up to a millimeter, but the computer generated drawings are actually accurate up to 1/1000 of a millimeter and the workmen will not be able to measure things precise enough and thus not be able to finish the house to be precisely as specified in the drawings.

    The mason and the carpenter would look at each other and think you are crazy.

    Thanks god does the world not depend on mathematicians to get stuff done and problems solved in the real world :)

  4. avatar Carsten Cumbrowski Says:

    One more quick note: That reminds me of the prophecies by Nostradamus. He wrote them as vague as he did for the same reasons :)

    I am in the wrong business. I should have become a fortune teller instead :)

  5. avatar Rhea Drysdale Says:

    Carsten, thanks for the responses! I understand the logic flaw, but can’t fault them. My grandpa happens to be a rocket scientist, so I kind of sympathize on where they’re coming from. The rules only exist so we can break them. :)

  6. avatar AussieWebmaster Says:

    I thought I had studied piano axioms in my fourth year of piano lessons….

RSS feed for comments on this post. TrackBack URL

Got something to say?