Real-time web crawler detection
Dikaiakos, Marios D.
Source2011 18th International Conference on Telecommunications, ICT 2011
2011 18th International Conference on Telecommunications, ICT 2011
Google Scholar check
MetadataShow full item record
In this paper we present a methodology for detecting web crawlers in real time. We use decision trees to classify requests in real time, as originating from a crawler or human, while their session is ongoing. For this purpose we used machine learning techniques to identify the most important features that differentiate humans from crawlers. The method was tested in real time with the help of an emulator, using only a small number of requests. Our results demonstrate the effectiveness and applicability of our approach. © 2011 IEEE.