Distributed location aware web crawling
Date
2004ISBN
1-58113-844-XSource
Thirteenth International World Wide Web Conference Proceedings, WWW2004Thirteenth International World Wide Web Conference Proceedings, WWW2004
Pages
1200-1201Google Scholar check
Keyword(s):
Metadata
Show full item recordAbstract
Distributed crawling has shown that it can overcome important limitations of the today's crawling paradigm. However, the optimal benefits of this approach are usually limited to the sites hosting the crawler. In this work, we propose a location-aware method, called IPMicra, that utilizes an IP address hierarchy, and allows crawling of links in a near optimal location aware manner.