1 / 26

Brass: A Queueing Manager for Warrick

Brass: A Queueing Manager for Warrick. Frank McCown, Amine Benjelloun, and Michael L. Nelson Old Dominion University Computer Science Department Norfolk, Virginia, USA IWAW 2007 Vancouver, BC June 23, 2007. Agenda. Dangers facing website Web-repository crawling

kaiya
Download Presentation

Brass: A Queueing Manager for Warrick

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Brass: A Queueing Manager for Warrick Frank McCown, Amine Benjelloun, and Michael L. Nelson Old Dominion UniversityComputer Science DepartmentNorfolk, Virginia, USAIWAW 2007 Vancouver, BCJune 23, 2007

  2. Agenda • Dangers facing website • Web-repository crawling • Comparing web crawling with web-repository crawling • All about Brass • Alternate Warrick deployments

  3. Black hat: http://img.webpronews.com/securitypronews/110705blackhat.jpgVirus image: http://polarboing.com/images/topics/misc/story.computer.virus_1137794805.jpg Hard drive: http://www.datarecoveryspecialist.com/images/head-crash-2.jpg

  4. A couple weeks ago I… accidentally deleted my entire database of about 30 articles. After I finished berating myself for being so stupid, I realized that my hosting company would have a backup, so I sent an email asking them to restore the database. Their reply stated that backups were “coming soon”…OUCH! So right after I signed up with a better hosting company I had to figure out a plan B.

  5. Crawling the Crawlers

  6. McCown, et al., Brass: A Queueing Manager for Warrick, IWAW 2007. • McCown, et al., Factors Affecting Website Reconstruction from the Web Infrastructure, ACM IEEE JCDL 2007. • McCown and Nelson, Evaluation of Crawling Policies for a Web-Repository Crawler, HYPERTEXT 2006. • McCown, et al., Lazy Preservation: Reconstructing Websites by Crawling the Crawlers, ACM WIDM 2006. Available at http://warrick.cs.odu.edu/

  7. Cached Image

  8. Cached PDF http://www.fda.gov/cder/about/whatwedo/testtube.pdf canonical MSN version Yahoo version Google version

  9. Examples of Lost Websites Recovered with Warrick

  10. Web Crawler

  11. Web-Repository Crawler

  12. Web crawling Limit hit rate per host Websites periodically unavailable Portions of website off-limits (robots.txt, passwords) Deep web Spam Duplicate content Flash and JavaScript interfaces Crawler traps Web-repo crawling Limit hit rate per repo Limited hits per day (API query quotas) Repos periodically unavailable Flash and JavaScript interfaces Can only recover what repos have stored Lossy format conversions (thumb nail images, HTMLlized PDFs, etc.) Issues

  13. Problems with Warrick • Requires user to download, install, and run from the command line warrick.pl –d –r –o log.txt –c –wr ia http://foo.org/ • Google API keys are no longer available • Screen-scrapes Google’s web user interface which can cause Google to black-list an IP address

  14. Solution: Brass • Queueing system using ODU nodes, so API query limits can be spread across several machines • Uses Google API keys which we obtained before they were no longer made available • Easy-to-use web interface utilizing email to notify user when reconstructions are complete

  15. Warrick Brown Captain Jim Brass http://www.cbs.com/primetime/csi/bios/index.php?cast_member=gary

  16. Brass Architecture

  17. Other Warrick Deployments • GUI interface for client executable • Installation difficulties • Lack of Google API keys • Web interface along with client application which makes queries • Browser plug-in, Flash, or applet • Must manage Google API keys • Browser must be left open and continued Internet access

  18. Conclusions • Warrick interface is almost ready for the public • Web interface will likely greatly increase Warrick usage • Collection of usage data will allow us to better understand what kinds of websites the public is interesting in recovering

  19. And that’s everything there is to know about Brass! Thanks, Dad, but I just wanted to know when you were going to change my diaper… Frank McCownfmccown@cs.odu.edu

More Related