1 / 73

Forward in Reverse

Forward in Reverse. A Gentle Overview Of Forward System Architecture Eric, Mike & Steve – WiLSWorld 2010. Outline. Intro to Forward with Demo Batch Processing (Backend) Web Application (Frontend) Challenges Q&As throughout. Intro & Demo. http://forward.library.wisconsin.edu.

Olivia
Download Presentation

Forward in Reverse

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Forward in Reverse A Gentle Overview Of Forward System Architecture Eric, Mike & Steve – WiLSWorld 2010

  2. Outline • Intro to Forward with Demo • Batch Processing (Backend) • Web Application (Frontend) • Challenges • Q&As throughout

  3. Intro & Demo

  4. http://forward.library.wisconsin.edu

  5. Batch Processing

  6. We have gobs & gobs of data.

  7. 1) Extract it

  8. 1a) ILS Data

  9. Sort, Deduplicate, Merge

  10. Antique Style Key By Stars*Go*Blue http://www.flickr.com/photos/artbydebora/1406682449/

  11. Common Identifier = OCLC Number

  12. Catalog Extract Processing Details • 14 Voyager Instances • 13M MARC bibliographic records extracted • Approximately 14 hours • Local C code • Sorted, deduplicated and merged output: • 8M records • 10GB Raw MARC data

  13. Why Merge? • URLs • Formats • Holdings

  14. 1b) Digital Collection Data

  15. Fedora Extract Processing Details • 1 Fedora Repository • 13K “First Class” XML Objects extracted • Approximately 4 hours • Repository query language • XML output: • METS XML package • Structural XML • MODS Bibliographic XML • 41MB XML data

  16. 2) Index it

  17. We take raw library data and process it with MARC/XML parsing tools and local parsing rulesin order to build a Solr search index.

  18. Raw data (MARC & METS XML) • Parsing libraries (Java code: marc4j, SAXParser) • Local code that defines parsing rules • Solr index

  19. 1. Raw data

  20. LEADER 02000cam a22003734a 45 0 001 6939454 005 20051208125417.0 008 051104s2004 enka $b 001 0 eng 010 $a 2003045349 035 $a (OCoLC)ocm52165958 040 $aDLC $c DLC $d XMA $d BAKER $d UKM 015 $a GBA430162 $2 bnb 016 7$a 012906573 $2 Uk 020 $a 0754605175 (alk. paper) 024 $a 99811375970 042 $a pcc 049 $a GZMA 050 00$a B3376.W564 $b W55355 2004 082 00 $a 111/.85/092 $2 21 245 00 $a Wittgenstein, aesthetics, and philosophy / $c edited by Peter B. Lewis. 260 $a Aldershot, Hants, England ; $a Burlington, VT : $b Ashgate, $c c2004. 300 $a xii, 255 p. : $b ill. ; $c 24 cm. 440 0 $a Ashgate Wittgensteinian studies 505 0$a Wittgenstein and the aesthetic domain / Kjell S. Johannessen -- 2. Wittgenstein, anti-essentialism and the definition of art / Terry Diffey -- 3. Rules, creativity and pictures : Wittgenstein's Lectures on aesthetics / David Novitz -- 4. Criticism without theory / Mark W. Rove -- 5. On aesthetic reactions and changing one's mind / Lars Hertzberg -- 6. Wittgenstein and the arts : understanding and performing / Graham McFee -- 7. Wittgenstein's music / R.A. Sharpe -- 8. Wittgenstein on music and language / Oswald Hanfling -- 9. Ethics and aesthetics are one / Carolyn Wilde -- 10. Fiction and reality in the arts / Ilham Dilman -- 11. Literature, human understanding and morality / Ben Tilghman -- 12. 'The self, thinking' : Wittgenstein, Augustine and the autobiographical situation / Garry L. Hagberg 504 $a Includes bibliographical references (p. 235-247) and index.

  21. 02000cam a22003734a 45 001000800000005001700008008004100025010001700066035002300083040003000106015001900136016001800155020002800173024001600201042000800217049000900225050002800234082002000262245007400282260006800356300003400424440003600458505081100494504006401305600005001369700002501419938007101444945001901515946003001534946001301564947002101577948001601598994001201614693945420051208125417.0051104s2004 enka b 001 0 eng a 2003045349 a(OCoLC)ocm52165958 aDLCcDLCdXMAdBAKERdUKM aGBA4301622bnb7 a0129065732Uk a0754605175 (alk. paper) a99811375970 apcc aGZMA00aB3376.W564bW55355 200400a111/.85/09222100aWittgenstein, aesthetics, and philosophy /cedited by Peter B. Lewis. aAldershot, Hants, England ;aBurlington, VT :bAshgate,cc2004. axii, 255 p. :bill. ;c24 cm. 0aAshgate Wittgensteinian studies0 aWittgenstein and the aesthetic domain / Kjell S. Johannessen -- 2. Wittgenstein, anti-essentialism and the definition of art / Terry Diffey -- 3. Rules, creativity and pictures : Wittgenstein's Lectures on aesthetics / David Novitz -- 4. Criticism without theory / Mark W. Rove -- 5. On aesthetic reactions and changing one's mind / Lars Hertzberg -- 6. Wittgenstein and the arts : understanding and performing / Graham McFee -- 7. Wittgenstein's music / R.A. Sharpe -- 8. Wittgenstein on music and language / Oswald Hanfling -- 9. Ethics and aesthetics are one / Carolyn Wilde -- 10. Fiction and reality in the arts / Ilham Dilman -- 11. Literature, human understanding and morality / Ben Tilghman -- 12. 'The self, thinking' : Wittgenstein, Augustine and the autobiographical situation / Garry L. Hagberg aIncludes bibliographical references (p. 235-247) and index.10aWittgenstein, Ludwig,d1889-1951xAesthetics.1 aLewis, Peter,d1947- aBaker & TaylorbBKTYc99.95d99.95i0754605175n0004227086sactive c1d89087961587 a714694b2005-11-23c81.86 c99.95d1 aHEUR 4801bm,stk aSCNd348032 a92bGZM

  22. 2. MARC/XML parsing libraries

  23. 02000cam a22003734a 45 001000800000005001700008008004100025010001700066035002300083040003000106015001900136016001800155020002800173024001600201042000800217049000900225050002800234082002000262245007400282260006800356300003400424440003600458505081100494504006401305600005001369700002501419938007101444945001901515946003001534946001301564947002101577948001601598994001201614693945420051208125417.0051104s2004 enka b 001 0 eng a 2003045349 a(OCoLC)ocm52165958 aDLCcDLCdXMAdBAKERdUKM aGBA4301622bnb7 a0129065732Uk a0754605175 (alk. paper) a99811375970 apcc aGZMA00aB3376.W564bW55355 200400a111/.85/09222100aWittgenstein, aesthetics, and philosophy /cedited by Peter B. Lewis. aAldershot, Hants, England ;aBurlington, VT :bAshgate,cc2004. axii, 255 p. :bill. ;c24 cm. 0aAshgate Wittgensteinian studies0 aWittgenstein and the aesthetic domain / Kjell S. Johannessen -- 2. Wittgenstein, anti-essentialism and the definition of art / Terry Diffey -- 3. Rules, creativity and pictures : Wittgenstein's Lectures on aesthetics / David Novitz -- 4. Criticism without theory / Mark W. Rove -- 5. On aesthetic reactions and changing one's mind / Lars Hertzberg -- 6. Wittgenstein and the arts : understanding and performing / Graham McFee -- 7. Wittgenstein's music / R.A. Sharpe -- 8. Wittgenstein on music and language / Oswald Hanfling -- 9. Ethics and aesthetics are one / Carolyn Wilde -- 10. Fiction and reality in the arts / Ilham Dilman -- 11. Literature, human understanding and morality / Ben Tilghman -- 12. 'The self, thinking' : Wittgenstein, Augustine and the autobiographical situation / Garry L. Hagberg aIncludes bibliographical references (p. 235-247) and index.10aWittgenstein, Ludwig,d1889-1951xAesthetics.1 aLewis, Peter,d1947- aBaker & TaylorbBKTYc99.95d99.95i0754605175n0004227086sactive c1d89087961587 a714694b2005-11-23c81.86 c99.95d1 aHEUR 4801bm,stk aSCNd348032 a92bGZM

  24. 3. Local code

  25. 4. http://lucene.apache.org/solr/

  26. What is Solr? An XML API over a Lucene search index.

  27. Access to Raw Formats Raw MARC stored for Merged record Live calls made to Fedora web services

  28. Data Refresh Bibliographic: weekly Circulation status: nightly

  29. For more information, see http://sdg.library.wisc.edu/blog/2010/03/03/solr-marc-indexing-based-on-diffs/

  30. Web Application

  31. Frontend? • (X)HTML • JavaScript • Cascading Style Sheets • Design • Information Architecture • User experience • Chrome (images, icons, pretty)

  32. Forward Colophon • ActiveRecordBaseWithoutTable (Rails plugin) • Apache • Blacklight (Rails plugin) • Blueprint CSS • Bookreader (jQuery) • Capistrano • Crontab • Engines (Rails plugin) • Fedora • Freebase API • GeoIP (Ruby gem) • Google Books API • Haml (Rails plugin) • Happymapper (Ruby gem) • HathiTrust API • jQuery • Ken (Ruby gem) • LowPro (Prototype JS) • MARC4J • Passenger (modrails) • Prototype JS • PostgreSQL • Raphael • Ruby on Rails • Shibboleth • Subversion • Solr / Lucene • Summon (Ruby gem) • UW-Madison Libraries Staff Directory API • UWDC (Rails plugin) • Voyager API • Tender love and attention

  33. Campus Affiliation Users localize to a school, allows us scope many features to their campus. GeoIP RubyGem Match IP addresses with physical locations. Raphaël—JavaScript Library “Small JavaScript library that should simplify your work with vector graphics on the web”.

  34. Raphaël SVG elements, like the circles and squares in the Forward splash page, can be treated as XHTML elements allowing us to manipulate them with JavaScript and CSS. http://raphaeljs.com/

More Related