Cornell's Innovative B atch P rocessing T echniques P ut I nto User-Friendly P ractice

1. Cornell's Innovative Batch Processing Techniques Put Into User-Friendly Practice Boaz Nadav-Manes Director, Acquisitions and Automated Technical Services Philosophy Selector Cornell University Library Library Technical Services

2. Background: Cornell Technical Services and IT • "Technical Services" vs. "Automated Technical Services“ (04/05 onwards). • History of innovative co-development. • Reporting lines (not much "red tape"). • Staff reductions through attrition, early retirements and some layoffs. • Transition to electronic. • 2CUL (combining Technical Services with Columbia’s – 2015)

3. TS Staff and Automation • Not everything in TS can be automated. • Ask: what is big enough and/or what will happen a lot? • TS Staff are enthusiastic about automatic tools if they are the ones to contribute in developing them and they are the ones to operate them. • Institutional history and “traditional” experience count! As you move forward emphasize more your core values of service and train (not necessarily hire) for higher level positions. • You don’t need to be a code expert as long as you know how to logically describe and write a scope for the solution. • Scalability and Flexibility are keys for good solutions. • Work collaboratively – invite others to contribute and to lead projects. • Most of the automatic solutions are not "ground breaking" from the start - they relieve some marginal FTE counts that add up later on.

4. Batch Your Mind! Have a users’group - teach each other = BPEG (Batch Processing Exploratory Group) Introduction to Web Service API's using PHP and HTML General Training for all staff on how to use and design MS-Access reports to streamline TS processes: • Automatic claims (is there a note in PO?) • Open purchase orders • How much you spent with whom? • Who has done what and when?

5. In the ecosystems business... Work on solutions that have value beyond TS: http://www.oclc.org/worldcat-selection.en.html http://poof.library.cornell.edu/content/about Patron Driven Acquisitions in electronic and in print http://2cul.org/ http://www.slideshare.net/boaz22/ala-2011-pda-preconference

6. Maximize simple building blocks (metadata) 1. SSH-Secure Shell using Pre-written Scripts: Commands perform tasks quickly, but offer limited flexibility for record quality and script customization. 2. LS-Tools Web Interface Greater flexibility for accommodating new vendor plans and modifications of records. 3. Combinations of above, adding Perl scripts, OCLC Batch-searching, Z39.50 protocol; and/or MARC-Edit Tool More complex operation, but results in better quality of loaded records.

7. Friendly Interfaces allow non-experts to participate: Locally designed “LS-Tools” Web Interface Consists of 3 main steps: • General Import of record files to LS-Tools • Modification of files to suit our needs (adding appropriate 948 fields, deleting unneeded fields, etc.) • Bulk Import of newly modified records, to be loaded into Voyager in SSH. • Benefits over basic SSH commands: not programmer dependent for new vendor’s files; greater control with a “do-it-yourself” approach.

8. Functional Approach to Tasks

9. Assessment Incorporated: Variety of Users

10. Heaviest User (non specialist!)

11. Power User

12. Structure and Permissions

13. Case Study: ISBN extract from YBP approvals

15. If/then Logic Runs In The Background (invisible and flexible)

16. Checks and Balances: ********************************************************************** Cornell University Library Batch Processing Yankee Approval Metadata run on Mon Oct 7 17:36:04 EDT 2013 Who and Where: apache on lstools01.library.cornell.edu Home directory: /libweb/sites/lstools.library.cornell.edu/htdocs Working directory: /lstools/batch_data/YankAppr.280.1736 ********************************************************************** Netid: btn3 Voyager Site: Test Output Log File: /lstools/users/btn3/Jobs/vendor.yankee.1381181763.log E-mail notification: ph44@cornell.edubtn3@cornell.edu ********************************************************************** ACTION: Misc FTP BULK: YANKEE CATNAME: Yankee EMAIL: ctsbulk-l FTPGET: yankee INFILE: 13300927.mrc OUTF: /lstools/users/btn3/Marc/13300927.mrc PATH: POFLAG: no SCRIPT: yankee.pl TITLE: Yankee Approvals VENDOR: yankee VNAME: Yankee_Appr ********************************************************************** STEP 0.1: 183 initial load records STEP 1.2: 183 input marc records with ISBNs. STEP 1.2: 183 unique ISBNs. STEP 2.1: 183 records written FINAL: 1 vendor specific file prepared for load into Voyager ********************************************************************** SUMMARY FOR PROJECT MANAGEMENT ********************************************************************** Packaged for bulkimport using YANKEE: 183 records filename: /lstools/users/btn3/Marc/13300927.mrc ********************************************************************** Job completed on 2069-12-31 at 19:00:00 RC=0

19. Usage Statistics

20. Use pre-existing tools: Opens MARC files into temporary “MRK” files that can be edited (then compiled back into MARC) Using MarcEdit for a more ‘surgical’ approach http://people.oregonstate.edu/~reeset/marcedit/html/about.html

21. Mix and match tools and approaches: Using ISBN Extract with Batch Searching in OCLC After retrieving a MARC file from vednor’s FTP, ISBN-Extract command creates a “.txt” file of ISBNs in a list, and sends it to a designated folder ISBN File is batch-loaded into OCLC to be searched & exported to Local files.

22. Cultivate outside the box thinking: Using PERL Scripts when MARC files are unavailable Script is designed to navigate to Website, enter password, and extract lists of LCCNs from a set of framed pages. Extracting LCCN lists as “.txt” for all un-received LC Jakarta shipments…

23. SHARE (FOLDERS) LCCN Text files are sent directly to relevant vendor folder, pre sorted by country (IO, MY, PH, BR, TH). Allow experts to review exceptions in rotation. Communal deposit of itterative procedures.

24. smile Contact information: btn3@cornell.edu