Comparison between web Robot Request Detection Techniques on Web Server Log in Data Mining
Authors: Nitika Kadam
Published in: IJRRETAS, Volume 3, Issue 9, pp. 1-3 (01 Sep 2017)
Impact Factor: 3.5 (2017)
License:
CC BY 4.0 (Open Access)
Certificate: View Certificate
Abstract
Web robots are software programs which automatically traverse through hyperlink structure of Web to retrieve Web resources. Robots can be used for variety of tasks such as crawling and indexing information for search engines, offline browsing, shopping comparison and email collectors. Apart from that robots can also be used for some malicious purposes like sending spam mails, stealing business intelligence etc. It is necessary to detect robots due to privacy, security and performance of server related issues. Several well-known techniques to detect robots are : robots.txt check, known robot’s IP address, User agent mapping, keywords matching in User agent field, browsing speed, unassigned referrer etc. In this paper we have discussed as well as implemented various robot identification techniques on real server log data and compared their performance for a given dataset.
Keywords
- Robot detection
- Web server log
- Web usage mining
- Data extraction
References
- Tan, Pang-Ning, and Vipin Kumar. "Discovery of web robot sessions based on their navigational patterns." Intelligent Technologies for Information Analysis. Springer Berlin Heidelberg, 2004. 193-222.
- N. Tyagi, A. Solanki, S. Tyagi, “An Algorithmic Approach to Data Preprocessing in Web Usage Mining”, International Journal of Information Technology and Knowledge Management 2 (2) (2010) 279–283.
- Ling Zheng Hui Gui. Feng Li, “Optimized Data Preprocessing Technology for Web Log Mining”,International Conference On Computer Design And Applications (ICCDA 2010).
- Michal Munk, Jozef Kapustaa, Peter Šveca*, “Data Preprocessing Evaluation for Web Log Mining:Reconstruction of Activities of a Web Visitor” International Conference on Computational Science, ICCS 2011 P5ocedia Computer Science 1 (2012).
- Kwon, Shinil, Young-Gab Kim, and Sungdeok Cha. "Web robot detection based on pattern-matching technique." Journal of Information Science 38.2 (2012): 118-126.
- Lu, Wei-Zhou, and Shun-zheng Yu. "Web robot detection based on hidden Markov model." 2006 International Conference on Communications, Circuits and Systems. 2006.
- Doran, Derek, and Swapna S. Gokhale. "Web robot detection techniques: overview and limitations." Data Mining and Knowledge Discovery 22.1-2 (2011): 183-
- Sardar, Tanvir Habib, and Zohreh Ansari. "Detection and confirmation of web robot requests for cleaning the voluminous web log data." IMpact of E-Technology on US (IMPETUS), 2014 International Conference on the. IEEE, 2014.
- Srivastava, Mitali, Rakhi Garg, and P. K. Mishra. "Preprocessing techniques in web usage mining: A survey." International Journal of Computer Applications 97.18 (2014).
- Srivastava, Mitali, Rakhi Garg, and P. K. Mishra. "Analysis of Data Extraction and Data Cleaning in Web Usage Mining." Proceedings of the 2015 International Conference on Advanced Research in Computer Science Engineering & Technology (ICARCSET 2015). ACM, 2015.
- Koster, M. 1994a. Guidelines for robot writers. Htp://info.webcrawler.com/mak/projects/robots/guidelines. html
How to Cite
More from this Issue
- A REVIEW ON ATM FRAUD DETECTION TECHNIQUE USING IMAGE PROCCESING IN MATDeepika Khatwa, Rahul Kaul
More by these Authors
- Role of AI and ChatGPT in Education with Impact on other field2024 • Vol. 10, Issue 6
- “An Improved Hybrid Filtering Approach for Recommender System”2016 • Vol. 2, Issue 3