Real-time web crawler detection
Date
2011ISBN
978-1-4577-0024-8Source
2011 18th International Conference on Telecommunications, ICT 20112011 18th International Conference on Telecommunications, ICT 2011
Pages
428-432Google Scholar check
Keyword(s):
Metadata
Show full item recordAbstract
In this paper we present a methodology for detecting web crawlers in real time. We use decision trees to classify requests in real time, as originating from a crawler or human, while their session is ongoing. For this purpose we used machine learning techniques to identify the most important features that differentiate humans from crawlers. The method was tested in real time with the help of an emulator, using only a small number of requests. Our results demonstrate the effectiveness and applicability of our approach. © 2011 IEEE.