A Strand of Perls:

1. A Strand of Perls: Some Home Grown Utilities

2. Syllabus Our New Books List Call Number Sorting Getting Operator Profiles QPID – Quick Patron Information Dump (cupid…)

3. Our New Books List Call Number Sorting Getting Operator Profiles QPID – Quick Patron Information Dump (cupid…)

4. Why present another new books list? Different strokes for different folks… • Given: • professors don’t care about call numbers; they just want to go to their area and see what’s new – information sorted by department

5. Why present another new books list? Different strokes for different folks… • Given: • professors don’t care about call numbers; they just want to go to their area and see what’s new – information sorted by department • information needed on a monthly basis

6. Why present another new books list? Different strokes for different folks… • Given: • professors don’t care about call numbers; they just want to go to their area and see what’s new – information sorted by department • information needed on a monthly basis • can go back through data for several previous months

7. Why present another new books list? Different strokes for different folks… • Given: • professors don’t care about call numbers; they just want to go to their area and see what’s new – information sorted by department • information needed on a monthly basis • can go back through data for several previous months • here’s an overview…

8. New Books List Process at WMU get last month’s acquisitions

9. New Books List Process at WMU get last month’s acquisitions break up by department

10. New Books List Process at WMU get last month’s acquisitions Department A break up by department Department B Department C Department X

11. New Books List Process at WMU get last month’s acquisitions Department A break up by department Department B Department C one text output file Department X

12. New Books List Process at WMU get last month’s acquisitions Department A break up by department Department B Department C one text output file Department X ftp to Batch PC

13. New Books List Process at WMU get last month’s acquisitions Department A break up by department Department B Department C one text output file Department X ftp to Batch PC put on library LAN for the Web Office

14. New Books List Process at WMU Our production-type jobs get the database password from a file, for easy maintenance. Then use DBI to set up access to the database.

15. New Books List Process at WMU The query (sprintf wrapper removed for clarity)

16. New Books List Process at WMU Get data from the query in a loop and put in an array

17. New Books List Process at WMU Get rid of headphones!

18. New Books List Process at WMU Create sort vector and put in array

19. New Books List Process at WMU Got the deduping code from one of the O’Reilly Perl books. Data will implicitly be in call number order due to sort vector structure. …line noise…?

20. Digression… Speaking of line noise… Broken up for clarity This puts the line count of a MARC file into a shell script.

21. New Books List Process at WMU Now we need to get the results classified by department, going by call number ranges. raw ranges file…

22. New Books List Process at WMU The call number range specifications are normalized in the same manner used for sorting.

23. New Books List Process at WMU The call number range specifications are normalized in the same manner used for sorting. Great for the computer, not so easy for us humans. Created a utility to make a human-readable version. formatted ranges file…

24. New Books List Process at WMU The range data is read into arrays. (If a syntactic error was found, the program stops and shows where it is.)

25. New Books List Process at WMU The range data is read into arrays. (If a syntactic error was found, the program stops and shows where it is.) Then loop for each department. If the current call number falls within the current range, it goes into the current department file.

26. New Books List Process at WMU The output files are sorted. For our final processing, we loop through each of these sorted files of raw data. We ignore the call number chunks created during the normalization process. The desired fields are concatenated and line-wrapped. field1 | field2 | etc. how this was done…

27. New Books List Process at WMU As we loop through the contents of each departmental file: We split up the sort vector, and store the output fields with a vertical bar in between.

28. New Books List Process at WMU Some additional processing is done, including the always visually entertaining regular expression manipulations.

29. New Books List Process at WMU The output is line-wrapped prior to writing to the file.

30. New Books List Process at WMU The output is line-wrapped prior to writing to the file. You’ll need some initial setup for the above wrap to work.

31. New Books List Process at WMU Now we have our output file… When we first implemented our list, this was the whole process. The file was handed off to the library, where staff separated the departmental data out of the file, added the HTML, and put it on our web site. It took several hours to do this!

32. New Books List Process at WMU Once I knew this, I looked into further automation. Now we have an additional Perl script that takes care of the rest of the story. I looked at the new books’ web pages the library had created and figured out that I could break out three sections of static html.

33. New Books List Process at WMU We read the previous output file, paying attention to which department we’re “in”.

34. New Books List Process at WMU We read the previous output file, paying attention to which department we’re “in”. Next, we create a separate .html file for each department, incorporating the static HTML sections, adding date information where necessary.

35. New Books List Process at WMU We read the previous output file, paying attention to which department we’re “in”. Next, we create a separate .html file for each department, incorporating the static HTML sections, adding date information where necessary. Finally, these files are put on the library LAN and a reminder email is sent out.

36. New Books List Process at WMU get last month’s acquisitions static HTML Department A break up by department static HTML Department B static HTML Department C static HTML ftp to Batch PC Department X put on library LAN for the Web Office separate HTML file for each department, ready to be incorporated into the library web pages

37. New Books List Process at WMU See the results at: http://www.wmich.edu/library/newbooks/index.html

38. Our New Books List Call Number Sorting Getting Operator Profiles QPID – Quick Patron Information Dump (cupid…)

39. Call Number Sorting Seems right to call it sorting, but it’s really in the normalization process that the “magic” occurs.

40. Call Number Sorting Seems right to call it sorting, but it’s really in the normalization process that the “magic” occurs. Uses intelligent parsing, not a quick regular expression implementation.

41. Call Number Sorting Seems right to call it sorting, but it’s really in the normalization process that the “magic” occurs. Uses intelligent parsing, not a quick regular expression implementation. Designed with LC call numbers in mind, but pretty much handles everything, including locally generated call numbers.

42. Call Number Sorting Seems right to call it sorting, but it’s really in the normalization process that the “magic” occurs. Uses intelligent parsing, not a quick regular expression implementation. Designed with LC call numbers in mind, but pretty much handles everything, including locally generated call numbers. Resulting sorts appear to be about 99% accurate (my estimate).

43. Call Number Sorting Seems right to call it sorting, but it’s really in the normalization process that the “magic” occurs. Uses intelligent parsing, not a quick regular expression implementation. Designed with LC call numbers in mind, but pretty much handles everything, including locally generated call numbers. Resulting sorts appear to be about 99% accurate (my estimate). The algorithm divides call numbers into chunks, based on separators.

44. Call Number Sorting Explicit separators: colon (:)

45. Call Number Sorting Explicit separators: colon (:) semicolon (;)

46. Call Number Sorting Explicit separators: colon (:) semicolon (;) comma (,)

47. Call Number Sorting Explicit separators: colon (:) semicolon (;) comma (,) period (.)

48. Call Number Sorting Explicit separators: colon (:) semicolon (;) comma (,) period (.) space ( )

49. Call Number Sorting Explicit separators: colon (:) semicolon (;) comma (,) period (.) space ( ) forward slash (/)

50. Call Number Sorting Explicit separators: colon (:) semicolon (;) comma (,) period (.) space ( ) forward slash (/) Implicit separators: transitions: alpha->numeric numeric->alpha