1 / 8

Web Information Retrieval with Hadoop MapReduce: A Study on Inverted Index Evaluation and Spam Classification

This research project utilized MapReduce algorithms to process ClueWeb09, a 25 TB corpus of web pages, focusing on the first 50 million English documents. Inverted index was evaluated using TREC measures and BM25 and Waterloo spam scores were used for spam classification. A 14-node Hadoop cluster with 112 cores and 336 GB of RAM was employed to create the inverted index, taking 2 hours and 24 minutes. Batch evaluation was conducted with 50 topics and graded relevance judgments given by TREC, following an evaluation plan that was affordable, repeatable, insightful, and understandable. Future work includes integrating PageRank model and utilizing BM25F retrieval model for enhanced performance.

edgarr
Download Presentation

Web Information Retrieval with Hadoop MapReduce: A Study on Inverted Index Evaluation and Spam Classification

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ClueWeb09 Corpus Information Retrieval using Hadoop MapReduce

  2. Research Problem • Used MapReduce algorithms to process a corpus of web pages and develop required index files • Inverted Index evaluated using TREC measures • Used Hadoop and Ivory

  3. Dataset • ClueWeb09 Collection – 25 TB of uncompressed documents. • Project focusses on first 50 million English documents • For data verification, Document Record counters and Checksum values are present

  4. System Designed • Inverted Index created using MapReduce • BM25 and Waterloo spam scores used to classify documents as spam or ham • 14 node Hadoop cluster • Inverted Index was created using 112 cores and 336GB of RAM • Creation of Inverted Index took 2 hours 24 minutes and 228.7 GB space

  5. Batch Evaluation • 50 topics given by TREC • Graded relevance judgments given by TREC • The evaluation plan had the following features: - Affordable - Repeatable - Insightful - Understandable

  6. BM25 Parameters

  7. Results and Conclusions k1= 1.5 and b=0.3

  8. Future Work • PageRank model can also be included with Waterloo spam scores • BM25F retrieval model would perform better as it takes into account the web page features like headings, anchor text etc.

More Related