Saturday, 24 August 2013

How to block Yandex and Baidu IPs in GAE?

How to block Yandex and Baidu IPs in GAE?

In robots.txt I can put:
#Baiduspider
User-agent: Baiduspider
Disallow: /
#Yandex
User-agent: Yandex
Disallow: /
to tell the search engines to stop crawling my site. But how to block them
by IPs in GAE?

No comments:

Post a Comment