A couple months ago Google gave a way to do Googlebot verification. Basically, it involves two steps: a reverse DNS lookup followed by a forward DNS lookup. Now Microsoft has implemented the same DNS policies, so you can use the same method to verify MSNBot.
Good for them. Who will be next to implement this open method to authenticate search engine bots: Yahoo!, Ask, or Exalead? Place your bets.
Update: w00t! I’m digging out from comments and just saw this comment by Peter Linsley of Ask: “The issue of whether Ask could support bot authentication was raised in Las Vegas at PubCon and we’ve updated the webmaster FAQ. No surprises here, it’s the old DNS roundtrip lookup trick; all our crawlers are under the ask.com domain:
http://about.ask.com/en/docs/about/webmasters.shtml”
So it looks like Ask wins the prize! Here’s the specific entry in Ask’s documentation:
Good for them. Who will be next to implement this open method to authenticate search engine bots: Yahoo!, Ask, or Exalead? Place your bets.
Update: w00t! I’m digging out from comments and just saw this comment by Peter Linsley of Ask: “The issue of whether Ask could support bot authentication was raised in Las Vegas at PubCon and we’ve updated the webmaster FAQ. No surprises here, it’s the old DNS roundtrip lookup trick; all our crawlers are under the ask.com domain:
http://about.ask.com/en/docs/about/webmasters.shtml”
So it looks like Ask wins the prize! Here’s the specific entry in Ask’s documentation:
Q. How do I authenticate the Ask Crawler?
A: A User-Agent is no guarantee of authenticity as it is trivial for a malicious user to mimic the properties of the Ask Crawler. In order to properly authenticate the Ask Crawler, a round trip DNS lookup is required. This involves first taking the IP address of the Ask Crawler and performing a reverse DNS lookup ensuring that the IP address belongs to the ask.com domain. Then perform a forward DNS lookup with the host name ensuring that the resulting IP address matches the original.
1 comments:
We rank the best antivirus software by comparisons. Check out in-depth product reviews, articles. Compare features, prices and browse hundreds of screenshots to help you make the most informed decision as to which antivirus software is right for you.. http://bit.ly/93bfZU
Post a Comment