An Ontology Based Web Crawler with a Near-Duplicate Detection System to Improve the Performance of a Web Crawler
DOI:
https://doi.org/10.47604/ijts.2984Keywords:
Ontology Based Web Crawler, Near-Duplicate Detection System, PerformanceAbstract
Purpose: The aim of the study is to examine how an ontology-based web crawler with a near-duplicate detection system improves the performance of a web crawler.
Methodology: The experiment was carried out using secondary data from a sample web site which was used since crawling is an endless process. Using these two approaches, the ontology web crawler would search for relevant searches according to the search query of the user while the near-duplicate detection system would eliminate redundant data.
Findings: It was observed that ontology web crawler performed better and faster than a normal crawler. It takes less execution time to search the web than other web crawlers. This is due to the fact that web documents are being filtered by the ontology web crawler such that only relevant web documents are retrieved according to the search query of the user. The relevant documents are further filtered by a near-duplicate detection system by removing web pages that are duplicates of each other and also remove near-duplicate web documents. This further reduces the number of web pages retrieved by the web crawler. This model saves on storage space because of the reduced number of web pages retrieved as it takes care of irrelevant and redundant web pages searched.
Unique Contribution to Theory, Practice and Policy: The study recommends that the model can be improved to be dynamic by adding new relations that is the crawler should search for web pages related to the search even if they don’t contain the keywords searched. Domains and concepts should be added when visiting new web pages. Standardization of weights needs to be done because as of now experts assign weights to terms according to the area of expertise and knowledge.
Downloads
References
Arun Pr, Sumesh Ms. (2015). Near-duplicate web page detection by enhanced TDW and simHash technique. International Conference on Computing and Network Communications.
Komal , Dr. Ashutosh Dixit. (2016). Design Issues in Web Crawlers and Review of Parallel Crawlers
Lawankar, A., & Mangrulkar, N. (2016, February). A review on techniques for optimizing web crawler results. In 2016 World Conference on Futuristic Trends in Research and Innovation for Social Welfare (Startup Conclave) (pp. 1-4). IEEE.
Pranav, A., & Chauhan, S. (2015). Efficient focused web crawling approach for search engine. International Journal of Computer Science and Mobile Computing, 4(5), 545-551.
Saini, A. K., & KumarKhurana, V. (2016, March). ICT based communication systems as enabler for technology transfer. In 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom) (pp. 90-99). IEEE.
Sharma, G., Sharma, S., & Singla, H. (2016). Evolution of web crawler its challenges. Int J Comput Technol Appl, 9, 53-57.
Sharma, N., & Devi, V. S. (2016). An Efficient SmartCrawler for Harvesting Web Interfaces of a Two-Stage Crawler. i-Manager's Journal on Information Technology, 5(4), 20.
Suryawanshi, P., & Patil, D. V. (2015). An Overview of Approaches Used In Focused Crawlers. International Research Journal of Engineering and Technology (IRJET) Volume, 2.
Udapure, T. V., Kale, R. D., & Dharmik, R. C. (2014). Study of web crawler and its different types. IOSR Journal of Computer Engineering, 16(1), 01-05.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Ngulamu Daines Walowe, Dr. Michael Kimwele , Dr. Ann Kibe
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution (CC-BY) 4.0 License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
