A Research Program for Information Science with the NSDL as an Example

1. A Research Program for Information Sciencewith the NSDL as an Example William Y. Arms September 26, 2002

2. A Scenario A faculty member wished to find a paper for students to read in a class. He began by asking an expert. She suggested the original research paper as suitable. Later, he typed a few terms into Google, browsed the hits, selected one that led to ResearchIndex, found the paper, and downloaded a PDF version from the author's web site.

3. Internet Web Google ResearchIndex PDF Computer Science Computer Science

4. Computer Science HCI Human Computer Interaction Browsing Searching User interface design

5. HCI: Eye Tracking

6. Computer Science HCI Cognitive Studies Roles of expert/instructor/student Cognitive psychology Linguistics Natural language processing Cognitive Studies

8. Organizational change Economics Ethics Social culture Law Computer Science Society HCI Society Cognitive Studies

9. Information Science Computer Science Cognitive Studies Applications Society HCI

10. Open Access to Scientific, Scholarly and Professional Information

11. Before the Web Access to scientific, medical, legal information In the United States: excellent if you belonged to a rich organization (e.g, a major university) very poor otherwise In many countries of the world: very poor for everybody

12. Some Light Reading William Y. Arms, "Economic models for open-access publishing." iMP, March 2000. http://www.cisp.org/imp/march_2000/03_00arms.htm William Y. Arms, "Automated digital libraries." D-Lib Magazine, July/August 2000. http://www.dlib.org/dlib/july20/07contents.html William Y. Arms, "What are the alternatives to peer review? Quality control in scholarly publishing on the web." Journal of Electronic Publishing, 8(1), August 2002. http://www.press.umich.edu/jep/08-01/arms.html

13. Research Libraries are Expensive staff library materials buildings & facilities

14. Baumol's Cost Disease Price Labor-intensive services Bundle of goods and services Manufactured goods 2050 1950 2000 1900 Year

15. Baumol's Cost Disease Price Labor-intensive services Moore's Law Bundle of goods and services Manufactured goods 2050 1950 2000 1900 Year

16. Brute Force Computing Few people really understand Moore's Law Computing power doubles every 18 months Increases 100 times in 10 years Increases 10,000 times in 20 years Simple algorithms plus immense computing power can outperform human intelligence

17. Example: Catalogs and Indexes Cost disease: catalogs and indexes Catalog, index and abstracting records are very expensive when created by skilled professionals Moore's Law: automatic indexing of full text Retrieval effectiveness using automatic indexing can be at least as effective as manual indexing with controlled vocabularies (Cleverdon 1967, reporting on experiments by Salton)

18. Resistance to Change "I used to be a heavy user of INSPEC. Now I use Google instead."

19. Information Discovery:1992 and 2002 1992 2002 Content print digital Computing expensive inexpensive Choice of content selective comprehensive Index creation human automatic Frequency one time monthly Vocabulary controlled not controlled Query Boolean ranked retrieval Users trained untrained

20. Brute Force Computing:Substitutes for Human Intelligence Automated algorithms for information discovery Similarity of two documents Vector space and statistical methods (Salton, Sparc Jones, et al.) Importance of digital object Rank importance of web pages by analysis of the graph of web links (Kleinberg, Page, et al.)

21. Brute Force Computing: Automated Metadata Extraction Informedia (Carnegie Mellon) Automatic processing of segments of video, e.g., television news. Algorithms for: dividing raw video into discrete items generating short summaries indexing the sound track using speech recognition recognizing faces (Wactlar, et al.)

23. Computer Science HCI Low Cost Information Simple algorithms plus immense computing power plus the intelligenceof the user can replace labor-intensive services Cognitive Studies

24. The National Science Digital Library (NSDL)

25. 25 Scope All digital information relevant to any level of education in any branch of science. Scientific and technical information Materials used in education Materials tailored to education

26. 26 How Big might the NSDL be? All branches of science, all levels of education, very broadly defined: Five year targets • 1,000,000 different users • 10,000,000 digital objects • 10,000 to 100,000 independent sites

27. 27 Resources Integration team Budget $4-6 million Staff 25 - 30 Management Diffuse How can a small team, without direct management control, create a very large-scale digital library?

28. 28 Philosophy It is possible to build a very large digital library with a small staff. But ... • Every aspect of the library must be planned with scalability in mind. • Some compromises will be made.

29. 29 Basic Assumptions The integration team will not manage any collections The integration team will not create any metadata

30. 30 The Integration Task ... ... to provide a coherent set of collections and services across great diversity

31. 31 Interoperability The Problem Conventional approaches require partners to support agreements (technical, content, and business) But NSDL needs thousands of very different partners ... most of whom are not directly part of the NSDL program The challenge is to create incentives for independent digital libraries to adopt agreements

32. 32 Function Versus Cost of Acceptance Cost of acceptance Few adopters Many adopters Function

33. 33 Example: Textual Mark-up Cost of acceptance SGML XML HTML Function ASCII

34. 34 The Spectrum of Interoperability Level Agreements Example Federation Strict use of standards AACR, MARC (syntax, semantic, Z 39.50 and business) Harvesting Digital libraries expose Open Archives metadata; simple metadata harvesting protocol and registry Gathering Digital libraries do not Web crawlers cooperate; services must and search engines seek out information

35. 35 Searching What to Index? Full text indexing is excellent, but full text indexing is not possible for all materials (non-textual, no access for indexing). Comprehensive metadata is an alternative, but available for very few of the materials. What Architecture to Use? Few collections support an established search protocol (e.g., Z39.50).

36. 36 Broadcast Searching does not Scale Collections User interface server User

37. 37 The Metadata Repository Services The metadata repository is a resource for service providers. It holds information about every collection and item known to the NSDL. Users Metadata repository Collections

38. 38 Search Architecture Metadata repository Portal OAI SDLIP Search andDiscoveryServices Portal http Portal Collections James Allan,Bruce Croft (University of Massachusetts, Amherst)

39. 39 Other Topics User interfaces: data driven portals using a channel architecture Selection: selective web crawling, machine learning Quality measures: ???

40. 40 The Mortal behind the Portal [This space left intentionally blank.]

41. 41 Acknowledgement The NSDL is a program of the National Science Foundation's Directorate for Education and Human Resources, Division of Undergraduate Education. The NSDL Core Integration is a collaboration between the University Center for Atmospheric Research (Dave Fulker), Columbia University (Kate Wittenberg) and Cornell University (Bill Arms). The Technical Director is Carl Lagoze (Cornell University).

42. Information Science Computer Science Cognitive Studies Applications Society HCI