1 / 29

SLASHPack: Collector Performance Improvement and Evaluation

Rudd Stevens CS 690 Spring 2006. SLASHPack: Collector Performance Improvement and Evaluation. Outline. 1. Introduction, system overview and design. 2. Performance modifications, re-factoring and re-structuring. 3. Performance testing results and evaluation. Outline.

nibaw
Download Presentation

SLASHPack: Collector Performance Improvement and Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Rudd Stevens CS 690 Spring 2006 SLASHPack: CollectorPerformance Improvement andEvaluation SLASHPack Collector - 5/4/2006

  2. SLASHPack Collector - 5/4/2006 Outline 1. Introduction, system overview and design. 2. Performance modifications, re-factoring and re-structuring. 3. Performance testing results and evaluation.

  3. SLASHPack Collector - 5/4/2006 Outline 1. Introduction, system overview and design. 2. Performance modifications, re-factoring and re-structuring. 3. Performance testing results and evaluation.

  4. SLASHPack Collector - 5/4/2006 Introduction SLASHPack Toolkit (Semi-LArge Scale Hypertext Package) Sponsored by Prof. Chris Brooks, engineered for initial clients Nancy Montanez and Ryan King. Collector component Framework for collecting documents. Evaluate and improve performance.

  5. SLASHPack Collector - 5/4/2006 Contact and Information Sources Contact Information: Rudd Stevens rstevens (at) cs.usfca.edu Project Website: http://www.cs.usfca.edu/~rstevens/slashpack/collector/ Project Sponsor: Professor Christopher Brooks Department of Computer Science University of San Francisco cbrooks (at) cs.usfca.edu

  6. SLASHPack Collector - 5/4/2006 Stages Addition of protocol module for Weblog data set. Performance testing using the Weblog and HTTP modules. Identify problem areas. Modify Collector to improve scalability and performance. Repeat performance testing and evaluate performance improvements.

  7. SLASHPack Collector - 5/4/2006 Implementation Language: Python Platform: Any Python supported OS. Python 2.4 or later (Developed and tested under Linux.) Progress: Fully built, newly re-factored for performance and usability.

  8. SLASHPack Collector - 5/4/2006 High level design SLASHPack designed as a framework. Modular components, that contain sub-modules. Collector pluggable for protocol modules, parsers, filters, output writers, etc.

  9. SLASHPack Collector - 5/4/2006 High level design (cont.)

  10. SLASHPack Collector - 5/4/2006 Outline 1. Introduction, system overview and design. 2. Performance modifications, re-factoring and re-structuring. 3. Performance testing results and evaluation.

  11. SLASHPack Collector - 5/4/2006 Performance Testing Large scale text collection. Weblog data set. Long web crawls. Performance testing monitoring Python Profiling. Integrated Statistics. Functionality Testing Python logging. Functionality test runs.

  12. SLASHPack Collector - 5/4/2006 UrlSentry Urls filtered using robots: 38 Urls filtered for depth: 9 Urls Processed: 5881 Urls filtered using filters: 165 UrlBookkeeper Duplicate Urls: 1557 Urls recorded: 4104 Collector Runtime Statistics UrlFrontier Url Frontier size, current number of links: 3465 Urls requested from frontier: 659 Url Frontier, current number of server queues: 78 Urls delivered from frontier: 639 Collector Documents per second: 3.70328865405 Total runtime: 2 Minutes 31.4869761467 Seconds

  13. SLASHPack Collector - 5/4/2006 Collector Runtime Statistics DocFingerprinter Documents Written: 386 Average Document Size (bytes): 20570 HTTP Status Responses: 200: 394 204: 10 301: 8 302: 25 404: 91 403: 7 401: 1 400: 24 500: 1 Duplicate Documents: 51 Total Documents Collected: 561 Documents by mimetype: text/xml: 1 image/jpeg: 1 text/html: 451 image/gif: 1 text/plain: 106 application/octet-stream: 1

  14. SLASHPack Collector - 5/4/2006 Challenges Large text (XML) files 21 1 GB XML files. ~450,000 files per XML file. ~10 Million files, after processing. Memory/Storage Disk space. Memory usage during processing. (XML)

  15. SLASHPack Collector - 5/4/2006 Weblog raw data <post> <weblog_url> http://www.livejournal.com/users/chuckdarwin </weblog_url><weblog_title> ""Evolve!"“ </weblog_title><permalink>http://www.livejournal.com/users/chuckdarwin/1001264.html </permalink> <post_title> Flickr </post_title> <author_name> Darwin (chuckdarwin) </author_name> <date_posted> 2005-07-09 </date_posted> <time_posted> 000000 </time_posted> <content> &lt;html&gt;&lt;head&gt;&lt;meta content="text/html; charset=UTF-8" http-equiv="Content-Type"/&gt;&lt;title&gt;""Evolve!""&lt;/title&gt;&lt;/head&gt;&lt;body&gt; &lt;div style="text-align: center;"&gt;&lt;font size="+1"&gt;&lt;a href="http://www.nytimes.com/2005/07/09/arts/09boxe.html?ei=5088&amp;amp;en=61cfcd5835008b1a&amp;amp;ex=1278561600&amp;amp;partner=rssnyt&amp;amp;emc=rss&amp;amp;pagewanted=print"&gt;7/7 and 9/11?&lt;/a&gt;&lt;/font&gt;&lt;/div&gt;&lt;/body&gt;&lt;/html&gt; </content> <outlinks> <outlink> <url> http://www.nytimes.com/2005/07/09/arts/09boxe.html </url> <site> http://www.nytimes.com </site> <type> Press </type> </outlink> </outlinks> </post>

  16. SLASHPack Collector - 5/4/2006 Weblog processed data <spdata> <url>http://www.livejournal.com/users/chuckdarwin</url><date>20060212</date> <crawlname>WeblogPosts20050709</crawlname> <weblog> <weblog_title>”"Evolve!""</weblog_title> <permalink>http://www.livejournal.com/users/chuckdarwin/1001264.html</permalink> <post_title>Flickr</post_title> <author_name>Darwin (chuckdarwin)</author_name> <date_posted>2005-07-09</date_posted> <time_posted>000000</time_posted> <outlinks> <outlink> <type>Press</type> <url>http://www.nytimes.com/2005/07/09/arts/09boxe.html</url> <site>http://www.nytimes.com</site> </outlink> </outlinks> </weblog> <tags></tags> <size>493</size> <mimetype>text/plain</mimetype> <fingerprint>9949bba4ac535d18c3f11db66cdb194e</fingerprint> <content>Jmx0O2h0bWwmZ3Q7CiZsdDtoZWFkJmd0OwombHQ7bWV0YSBjb250ZW50P ….</content> </spdata>

  17. SLASHPack Collector - 5/4/2006 Original Design

  18. SLASHPack Collector - 5/4/2006 Problems to Address Overall collection performance Streamline processing. Robot file look up Incredibly slow and inefficient. (Not mine!) Thread interaction Efficient use of threads and queues to process data. Inefficient code Python code not always the fastest. miniDom XML parsing. Faster data structures Re-work collection protocols, DNS prefetch. Re-structure URL Frontier, URL Bookkeeper.

  19. SLASHPack Collector - 5/4/2006 New Design

  20. SLASHPack Collector - 5/4/2006 Performance Modifications Structure Re-design (threading) More queues, more independence. Robot Parser String creation, debug calls. URL Frontier More efficient data structures. Protocol Modules More efficient data structures. Re-factoring for reliable collection. XML parsing Switch to faster parser, removal of DOM parser. DNS Pre-fetching More efficient structuring.

  21. SLASHPack Collector - 5/4/2006 New data structures Dictionary fields for Base data type. (Must be implemented by any data protocol). Now passed in dictionary to storage component. Key Value Type datatype user defined datatype name string status HTTP document status string url URL of document string date collection date string crawlname name of current crawl string size byte length of content string mimetype mime type of document string fingerprint md5sum hash of content string content raw text of document string

  22. SLASHPack Collector - 5/4/2006 Outline 1. Introduction, system overview and design. 2. Performance modifications, re-factoring and re-structuring. 3. Performance testing results and evaluation.

  23. SLASHPack Collector - 5/4/2006 Performance Comparison Initial Results: Weblog data set w/o parsing, robots: 161 doc/s, 50 min. w/ parsing, robots: 3.9 doc/s, 162 min. (killed) HTTP Web crawl 100 docs w/ parsing, robots: 0.2 doc/s,16 min:13s 150 docs w/ parsing, robots: 0.3 doc/s, 21min:3s Modified Results: Weblog data set w/o parsing, robots: 170 doc/s, 42 min. w/ parsing, robots: 186 doc/s, 63 min. HTTP Web crawl 100 docs w/ parsing, robots: 2.2 doc/s, 1min:10s 150 docs w/ parsing, robots: 2.9 doc/s, 1min:14s

  24. SLASHPack Collector - 5/4/2006 Performance Comparison (cont.) Hardware considerations HTTP web crawl for 500 documents Pentium 4 2.4GHz 1 GB RAM 3.7 doc/s 3min:18s, 728 docs total (faster connection) Pentium 4 2.0GHz, 1GB RAM 3.7 doc/s 4min:25s, 725 docs total Pentium 4 3.2GHz HT, 2GB RAM 4.3 doc/s 2min:47s, 717 docs total (faster connection)

  25. SLASHPack Collector - 5/4/2006 Performance Comparison (cont.) • Comparison to other web crawlers (published results, 1999) • Google: 33.5 doc/s • Internet Archive: 46.3 doc/s • Mercator: 112 doc/s • Consideration of functionality • More than just a web crawler. • Mime types.

  26. SLASHPack Collector - 5/4/2006 Available Documentation Pydoc API Generated with Epydoc. Use and configuration guide (README). Quick start guide. Full Report Full specification of Collector, use, configuration and development background.

  27. SLASHPack Collector - 5/4/2006 Future Work Addition of pluggable modules. Improved fingerprint sets. Improved Python memory management and threading.

  28. SLASHPack Collector - 5/4/2006 References Allan Heydon and Marc Najork. Mercator: A scalable, extensible web crawler. http://research.compaq.com/SRC/mercator/papers/www/paper.pdf Soumen Chakrabati, Mining the Web, 2002. Ch. 2, pages 17-43. Heritrix, Internet Archive. http://crawler.archive.org/ Python Performance Tips http://wiki.python.org/moin/PythonSpeed/PerformanceTips Prof. Chris Brooks and the SLASHPack Team.

  29. SLASHPack Collector - 5/4/2006 Conclusion • Four stages: • Addition of protocol module for Weblog data set. • Performance testing and identifying problem areas. • Modify Collector to improve scalability and performance. • Repeat performance testing and evaluate performance improvements. • Results: • Expanded functionality for data types. • Modifications improved performance. • More stable and flexible design.

More Related