1 / 26

Converting Local Portal Data to MySpace

Converting Local Portal Data to MySpace. Anne L. Highsmith, Consortia Systems Coordinator & Prabha Acharya, Programmer Analyst II Texas A&M University. Points to Cover. Background Data Conversion Issues Testing on a Cloned Database Conversion Process Results and Future Plans.

booker
Download Presentation

Converting Local Portal Data to MySpace

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Converting Local Portal Data to MySpace Anne L. Highsmith, Consortia Systems Coordinator & Prabha Acharya, Programmer Analyst II Texas A&M University

  2. Points to Cover • Background • Data Conversion Issues • Testing on a Cloned Database • Conversion Process • Results and Future Plans Converting Local Portal Data to MySpace ELUNA 2007

  3. Background (1) • TAMU had a local Electronic Resources Locater (ERL) function to list ejournals and databases. • Users searched ERL and clicked on links to access eresources • Users could store the links in their MyPortal record on the library’s website Converting Local Portal Data to MySpace ELUNA 2007

  4. Background (2) • Metalib went into production at TAMU in January 2006 as Search Now. • TAMU had discussions with Ex Libris staff as to how MyPortal links could be converted to Metalib • First proposal, for Ex Libris staff to do a major conversion, not feasible • Decided to wait for Metalib 4.0 and X-Portal APIs to store data to MySpace Converting Local Portal Data to MySpace ELUNA 2007

  5. Rude awakening! • Fall of 2006, Ex Libris staff informed TAMU there would be no API to store to MySpace • Alternatives • Margaritas? • Convert them ourselves? Converting Local Portal Data to MySpace ELUNA 2007

  6. Conversion options • Assumed conversion would have to be: • User-initiated • Title by title • Discovered a way do the conversion directly, without involving the end user Converting Local Portal Data to MySpace ELUNA 2007

  7. Data Conversion Issues • Patrons • Journals • Databases Converting Local Portal Data to MySpace ELUNA 2007

  8. Data Conversion Issues - Patrons • How to match the MyPortal patron with the Metalib user? • MyPortal patrons identified by uin (University ID Number) • Our Metalib implementation uses uin as user login • Needed to match the uin with the z312_source_id Converting Local Portal Data to MySpace ELUNA 2007

  9. Data Conversion Issues - Journals • How to match MyPortal journals with Metalib? • Basic journal data stored in z11 table • Wrote Perl script (journals.pl) to extract journal identifier and issn from z11 • Used MS Access to match MyPortal data, keyed by uin & issn, with Metalib data • User journal data stored in z312_jnl_001 • Wrote Perl script (journals_by_patron.pl) to extract z312_source_id (uin) and journal identifier from Metalib user record in z312 table Converting Local Portal Data to MySpace ELUNA 2007

  10. Data Conversion Issues - Databases • How to match MyPortal databases with Metalib? • During creation of Metalib IRDS, stored the ERL key in the IRD cataloger note • Loaded IRD extract into MS Access • Matched Metalib user extract with IRD extract • Basic database data stored in z122 table • Wrote Perl script (databases_by_patron.pl) to extract z312_source_id (uin) and Metalib identifier (TEX #) Converting Local Portal Data to MySpace ELUNA 2007

  11. Data Conversion Issues - Final • As a result of matching ERL data and Metalib data, created a file that contained 1 or 2 entries for each user whose data had to be converted: • Tab-separated record beginning with uin and list of database id numbers • Tab-separated record beginning with uin and list of journal id numbers Converting Local Portal Data to MySpace ELUNA 2007

  12. Testing on a Cloned Database Converting Local Portal Data to MySpace ELUNA 2007

  13. Testing – Cloning the database (1) • TAMU has a test Metalib DB on separate server • Cloned production DB to test DB so could test in a realistic environment • mpsync - documented in Appendix D in the System Administrators Guide & Knowledge Base entry #1240. • exp_current_library & imp_current_library, documented in 7.8 in the SysAdmin Guide • Further doc – “Working with Test/Failover Installations of MetaLib” Converting Local Portal Data to MySpace ELUNA 2007

  14. Cloning – Export from Production • Created files on prod for export to test • Run mpsync to copy IRDs • TAMU has approx. 4500 IRDS • Elapsed time – 58 sec. • Run export current library for vir00 • TAMU has 32,000+ user records • Elapsed time – 1 hr. • Run export current library for jnl01 • TAMU has approx. 44,000 journal records • Elapsed time – 2:25 min. Converting Local Portal Data to MySpace ELUNA 2007

  15. Cloning – Import into Test • Import IRDS created by mpsync • Elapsed time – 3 hrs. 688 new records added • Matched existing keys. Appeared to create some duplicates • Import User Data • Elapsed time – 40 min. • Drops & re-adds tables • Import Journal Data • Elapsed time – 3 min. • Drops & re-adds tables Converting Local Portal Data to MySpace ELUNA 2007

  16. Conversion Process • Quality control • File preparation • Loading new patrons • Convert databases and ejournals Converting Local Portal Data to MySpace ELUNA 2007

  17. Conversion – Quality Control • Wrote a Perl script (pre_migration.pl) to collect stats and set up QC measures. For each user it reports: • Users who don’t have Metalib record • # of databases and journals already in Metalib • # of databases and journals to convert • # of users that have more than maximum number of databases or ejournals Converting Local Portal Data to MySpace ELUNA 2007

  18. Conversion – File Preparation • Testing demonstrated that It takes about 3 seconds per resource to be migrated. • With about 13000 resources that would be about 10 hours • So, wrote Perl script (breakup_myportal_file.pl) to break the input file into several files of about 250 records each. Converting Local Portal Data to MySpace ELUNA 2007

  19. Conversion – Loading new patrons • Since TAMU uses external authentication rather than patron load, some older MyPortal patrons not in Metalib database • Using file created during QC stage, create_xml_patron_load.pl formats user records for Metalib patron load utility • Set expiry date to 1 year in future • Based on a review of records created on that date, approx. 65 have since logged in. Converting Local Portal Data to MySpace ELUNA 2007

  20. Conversion - Convert databases and ejournals (1) • Run databases_by_patron.pl to get current lists from user records for comparison Converting Local Portal Data to MySpace ELUNA 2007

  21. Conversion - Convert databases and ejournals (2) • Using save_to_bookbag.cgi, for each user • Log in to Metalib • Send request to Metalib pds to get pds handle • Send request to Metalib, using just obtained pds handle, to get session id • Construct url of the form • http://metalibhost.edu/V/sessionid?pds_handle=pdshandle&func=find-db-add-res&resource=resourceid Converting Local Portal Data to MySpace ELUNA 2007

  22. Conversion - Convert databases and ejournals (3) • Set session id in cookie • For each database • Check to see if it duplicates db already stored • If not, store db to user’s clipboard • For each journal • Store to user’s record Converting Local Portal Data to MySpace ELUNA 2007

  23. Conversion - Convert databases and ejournals (4) • Do QC to make sure that all of the appropriate databases and ejournals are stored • Exit the session • Error logging revealed that for some users, 1 database or ejournal was not stored – these were re-run as an added input file Converting Local Portal Data to MySpace ELUNA 2007

  24. Final results • Processing all files took most of the week, off and on • Each file, limited to 250 records, took about 1 hour to process • Only post-production problem had nothing to do with conversion – ezproxy error. • Once users have logged into library portal, can now go directly to My Databases or My Ejournals list within Metalib Converting Local Portal Data to MySpace ELUNA 2007

  25. Future plans • Conversion of Voyager My Bookbag to e-shelf Converting Local Portal Data to MySpace ELUNA 2007

  26. Contact & download info Anne L. Highsmith Consortia Systems Coordinator Texas A&M University hismith@tamu.edu URL to download perl code, cgi, etc. http://libraryasp.tamu.edu/users/ahighsmi/presentations/save_to_bookbag.zip Converting Local Portal Data to MySpace ELUNA 2007

More Related