How to Find a Needle in the Haystack

1. How to Find a Needle in the Haystack Adrian StevensonLearning Technology ServicesUniversity of Manchester Institutional Web Management Workshop 2005Parallel Session4pm - 5.30pm, Wednesday 6th July 2005 How to Find a Needle in the HaystackIWMW 2005

2. Overview • Introduction to Cross searching / metasearch • The Problem – why metasearch? • JISC Information Environment • Quick introduction to XML and Web Services • Metasearch Technologies • Z39.50, SRU/SRW, OAI • Metasearch issues • NISO Metasearch Initiative How to Find a Needle in the HaystackIWMW 2005

3. Cross Searching • Cross searching has many names: • Metasearch • Distributed search • Parallel search • Federated search • Broadcast search • Cross-database search • Common theme of allowing search and retrieval to span multiple databases, sources, platforms, protocols, and vendors at once How to Find a Needle in the HaystackIWMW 2005

4. The Problem • Web users such as researchers or tutors frequently require information from a variety of different sources • User required to search many different service interfaces, each with a different look and feel, metadata and subject classifications. • The results are almost always supplied in HTML, which makes them difficult to merge. • Users search many services and portals such as the RDN, zetoc and COPAC, image resources, e-prints, learning objects, external and internal resources. • If a user wants to obtain a local copy of the range of search results, they often have to merge the results themselves, for example by creating a text file. How to Find a Needle in the HaystackIWMW 2005

5. JISC Information Environment • Cross searching is at the core of the JISC IE • JISC notes that considerable investment has been made to provide high-quality digital information resources • But students, lecturers and researchers are faced with a vast and sometimes bewildering range of sources of electronic information. • Each source has its own name, interface, features and search facilities. • Users remain unaware of their existence or fail to discover their value for their own learning, teaching or research. • A key challenge is therefore to achieve a managed, coherent and shared information environment that will overcome these obstacles How to Find a Needle in the HaystackIWMW 2005

6. JISC: Helping Users find digital information • Being able to cross-search will considerably simplify users’ interactions with online resources. • This should encourage take-up and greatly improve means of accessing these resources. • Institutions will be able to incorporate these services within their own institutional online environments, presenting local content alongside nationally provided resources. • A second aspect relates to making the Information Environment actually work. • Making the Information Environment work requires the implementation of a range of commonly-agreed technical standards and protocols How to Find a Needle in the HaystackIWMW 2005

7. JISC IE Technical Architecture • “The JISC Information Environment technical architecture specifies a set of standards and protocols that support the delivery of integrated networked services that allow the end-user to discover, access, use and publish digital and physical resources”

8. Metasearch Technologies • Two main approaches: • Real-time cross searching • Z39.50 • Search and Retrieve URL / Web Service - SRU/SRW • Harvesting • Open Archives Initiative Protocol for Metadata Harvesting – OAI-PMH How to Find a Needle in the HaystackIWMW 2005

9. Metasearch Technologies • Other approaches: • Hybrid • Combination of Z39.50, SRU/W, and OAI and .. • Screen scraping • parsing the HTML to find patterns or parts of content. • Screen scraping is an ad-hoc technique that is dependent on a consistent format for the data being scraped • Regular expressions used for screen scraping. Perl has strong support for regular expressions – grep • Difficult, unreliable and laborious How to Find a Needle in the HaystackIWMW 2005

10. Z39.50 • ANSI/NISO Z39.50 - 2003 Information Retrieval : Application Service Definition & Protocol Specification • The National Information Standards Organization (NISO) is an American National Standards Institute (ANSI) accredited standards developer that serves the library, information, and publishing communities How to Find a Needle in the HaystackIWMW 2005

11. Z39.50 • Z39.50 is designed to enable communication between computers, typically those used to manage library catalogues • A portal can send a real-time query to a number of Z39.50 enabled content providers and a results set is returned to the user • The AHDS Gateway, physically based in London, uses Z39.50 to query five different databases containing information on archaeology (York), history (Colchester), the performing arts (Glasgow), the visual arts (Newcastle), and textual studies (Oxford) • They are driven by different database management software and run on a variety of hardware platforms. Z39.50 enables searches across the five sites. • Library OPAC and desktop applications such as EndNote can also be used to search Z targets How to Find a Needle in the HaystackIWMW 2005

12. AHDS

13. Z39.50 • Z39.50 employs a client/server model • One computer, the client or, in Z39.50 terms, the ‘Origin’, submits a request to another computer, the server or ‘Target’ which then services the request and returns an answer • Queries can be sent to multiple databases simultaneously to cross search • Records can be returned in a number of formats or ‘syntaxes’ as requested by the client. These typically include: • MARC (Machine Readable Cataloging ) • SUTRS (Simple Unstructured Text Record Syntax) • Raw ASCII text file • XML (eXtensible Markup Language) How to Find a Needle in the HaystackIWMW 2005

14. What is XML? Some possible definitions? • a technology for the management, display and organisation of data • a programming language • a markup language • a markup language used to describe the structure of data • not really a language • a standard for creating languages that meet the XML criteria How to Find a Needle in the HaystackIWMW 2005

15. XML: elements <tag> content </tag> <language> English </language> How to Find a Needle in the HaystackIWMW 2005

16. XML must be well formed • a root element is required <ead> …..all your tags and content… </ead> • closing tags are required How to Find a Needle in the HaystackIWMW 2005

17. XML must be well formed • a root element is required <ead> …..all your tags and content… </ead> • closing tags are required • Tags must be properly nested • Case matters How to Find a Needle in the HaystackIWMW 2005

18. Valid XML • Valid XML provides consistency and facilitates the exchange of data • XML must conform to a Document Type Definition (DTD) or Schema to be valid • Schemas and DTDs specify the elements and attributes and defines how they can be used: • Sequence of elements • Maximum and minimum values • People can agree to use a common Schema for interchanging data • e-learning: IEEE Learning Object Metadata Schema (LOM) How to Find a Needle in the HaystackIWMW 2005

19. Some Valid XML - EAD (Encoded Archival Description) <archdesc level="fonds"> <did> <repository>John Rylands University Library of Manchester</repository> <unitid countrycode="GB" repositorycode="0133">GB 0133 NCN</unitid> <unittitle>Papers of Norman Nicholson</unittitle> <unitdate normal="1899-1987">1899-1987</unitdate> <physdesc> <extent>0.44 cu.m; 1,201 items</extent> </physdesc> <langmaterial> <language langcode="eng">English</language> </langmaterial> <origination>Nicholson, Norman Cornthwaite, 1914-1987</origination> <note>Created by the John Rylands Library archivist</note> </did> …..</archdesc> How to Find a Needle in the HaystackIWMW 2005

20. Something to remember about XML • XML does not do anything itself. It is pure information wrapped in XML tags. • You must use other means to send, receive or display the data Display here like this XML technologies is used by to.. Display there like that XML extract this data for this purpose extract that data for that purpose How to Find a Needle in the HaystackIWMW 2005

21. Why Use XML? • Because everyone else is! • International standard, supported by the W3C • XML is open, licence free and platform neutral • XML is human and machine readable • XML documents are text documents How to Find a Needle in the HaystackIWMW 2005

22. Why Use XML? • Separation of content and presentation • With proprietary systems content is inextricably bound up with format • XML does not determine the presentation of the data • You can use CSS (stylesheets) or XSLT (Extensible Style Sheet Language for Transformations) to present XML data • The flexibility of XML enables the presentation of merged search results to the user. How to Find a Needle in the HaystackIWMW 2005

23. Web Services • A Web Service is an online application that can be accessed by other applications in machine to machine (m2m) interactions • Web services use XML to achieve this interoperability • SOAP • WSDL: Web Services Description Language • UDDI: Universal Description, Discovery and Integration How to Find a Needle in the HaystackIWMW 2005

24. What is a Web Service • A Web Service is a process of some kind, some functionality, for example: • A search and retrieve procedure • A conversion process • Fahrenheit to Centigrade • MARC record to Dublin Core record • LCSH subject headings to Dewey Decimal Classification numbers How to Find a Needle in the HaystackIWMW 2005

25. Publicly available Web Services • Google’s ‘similar pages’ • Amazon’s book connections: ‘customers who bought this also bought this’ • These services can be used in other applications • Xmethods website has a list of some experimental services – http://www.xmethods.net How to Find a Needle in the HaystackIWMW 2005

26. Creating a Web Service • Web services can be built for existing applications, or created from scratch • A key element of a Web Service is an XML file with details of how to interact with the service – the WSDL (Web Services Description Language) file How to Find a Needle in the HaystackIWMW 2005

27. Zetoc WSDL extract http://zetoc.mimas.ac.uk/soap/zetocsoap.wsdl … <complexType name="JournalRequest"> <sequence> <element ref="srw:startRecord" minOccurs="1" maxOccurs="1"/> <element ref="bath:any" minOccurs="1" maxOccurs="1" nillable="true"/> <element ref="dc:title" minOccurs="1" maxOccurs="1" nillable="true"/> <element ref="dc:creator" minOccurs="1" maxOccurs="1" nillable="true"/> <element ref="oujnl:jtitle" minOccurs="1" maxOccurs="1" nillable="true"/> <element ref="oujnl:issn" minOccurs="1" maxOccurs="1" nillable="true"/> <element ref="oujnl:volume" minOccurs="1" maxOccurs="1" nillable="true"/> <element ref="oujnl:issue" minOccurs="1" maxOccurs="1" nillable="true"/> <element ref="oujnl:spage" minOccurs="1" maxOccurs="1" nillable="true"/> <element ref="dcterms:issued" minOccurs="1" maxOccurs="1" nillable="true"/> </sequence> </complexType> How to Find a Needle in the HaystackIWMW 2005

28. Interacting with a Web Service • Once the client application knows how to interact with the service, the client and service communicate using messages encoded in XML • These messages are frequently expressed in SOAP • These messages are generally passed over HTTP (but they don’t have to be) How to Find a Needle in the HaystackIWMW 2005

29. SOAP • A way of packaging XML information and passing it from one system to another • Allows one system to make requests of another and to process the reply • Systems can be completely different, running on different software, hardware How to Find a Needle in the HaystackIWMW 2005

30. SOAP request <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"> <soap:Body> <zetoc:JournalRequest> <dc:creator>apps</dc:creator> <oujnl:title>materialia</oujnl:title> <oujnl:issn>1359-6462</oujnl:issn> <oujnl:volume>48</oujnl:volume> … </zetoc:JournalRequest> </soap:Body> </soap:Envelope> How to Find a Needle in the HaystackIWMW 2005

31. SOAP response HTTP/1.1 200 OK Content-Type: text/xml <soap:Envelope > <soap:Body> <zetoc:IdentifierSearchResponse > <srw:numberOfRecords>1</srw:numberOfRecords> <dc:identifier>RN125218404</dc:identifier> <zetoc:type>J</zetoc:type> <dc:title>Phase compositions in magnesium-rare earth alloys containing yttrium, gadolinium or dysprosium</dc:title> … </zetoc:IdentifierSearchResponse > </soap:Body> </soap:Envelope> How to Find a Needle in the HaystackIWMW 2005

32. To recap … • SOAP is a standard used for wrapping XML messages • The XML that is sent and returned within the SOAP wrapper is determined by the WSDL file for any particular Web Service • This is all done on a machine-to-machine level – you should never have to see a SOAP message • However we can demonstrate with XML SPY editor so we can see the SOAP messages [demo] How to Find a Needle in the HaystackIWMW 2005

33. Search Retrieve URL / Web Service (SRU/SRW) • Takes the core of Z39.50 and re-implements as Web Service • SRU and SRW are XML based protocols designed to be a low barrier to entry solutions for performing searches and information retrieval operations across the internet. • The protocol has two ways that it can be carried: • via SOAP – Search Retrieve Web Service • as parameters in a URL. - SRU – Search/Retrieve by URL • The primary function of SRU/SRW is to allow a user to search a remote database of records. • This is done via the searchRetrieve operation: • the client sends a searchRetreiveRequest and • the server responds with a searchRetrieveResponse How to Find a Needle in the HaystackIWMW 2005

34. Example SRW request • Most important part is the ‘query’. It contains a Common Query Language (CQL) string: • The request contains other parameters, all of these are optional except for ‘version’ How to Find a Needle in the HaystackIWMW 2005

35. Example SRW response • Response must contain ‘version’ and ‘number of records’ How to Find a Needle in the HaystackIWMW 2005

36. Some SRU Requests • SRU requests are URL with query string • ‘Explain’ request:http://z3950.loc.gov:7090/voyagerDescribes the database/index and functionality • A simple search for the term "dinosaur“:http://z3950.loc.gov:7090/voyager?version=1.1& operation=searchRetrieve&query=dinosaur • And the first of these records:http://z3950.loc.gov:7090/voyager?version=1.1&operation=searchRetrieve&query=dinosaur&maximumRecords=1 How to Find a Needle in the HaystackIWMW 2005

37. Open Archives Initiative (OAI) • The Open Archives Initiative (OAI) provides is a mechanism for sharing metadata records based on HTTP and XML • Enables metadata records about resources to be ‘harvested’ from multiple distributed services, typically into a central database (which itself may be a Z39.50 target) • Records harvested periodically e.g.. Once a day, hour etc. • Generally considered to be an elegant, simple and efficient protocol • 6 requests types or ‘verbs’ - GetRecord, Identify, ListIdentifier, ListMetadataFormats, ListRecords and ListSets. • JORUM Learning Object Repository Service OAI interface at:http://repository.jorum.ac.uk/intralibrary/IntraLibrary-OAI?verb=identify How to Find a Needle in the HaystackIWMW 2005

38. Real Time Cross Searching VS. Harvesting • Delays occur with real time cross-searching • The response time for searches sent to multiple search targets tends to be limited by the worst performing target or intervening network delays. • Very difficult to build flexible browse interfaces based on a distributed set of gateway databases. • OAI harvesting periodic so search results may not be accurate and up to date How to Find a Needle in the HaystackIWMW 2005

39. OAI - Connect Portal • Connect’ Learning & Teaching Portal • http://www.connect.ac.uk • Connect is a HE Academy project (used to be the LTSN – Learning and Teaching Support Network) • Connect harvests in records from HE Academy subject centres around the UK • Records harvested by server at Rutherford Appleton Labs How to Find a Needle in the HaystackIWMW 2005

40. Connect Portal

41. Connect Portal

42. Connect Portal

43. Connect Portal

44. Connect Portal

45. Metasearch issues: Metadata • Format • As users searching cross-domain, it makes sense to use a cross-domain metadata schema. Dublin Core is a good contender for this and is required for use of OAI-PMH. • However, domains will use their own metadata schemas, such as the IEEE-LOM for learning objects. • Mappings required to enable cross searching, but some of the semantic richness of the original resource may be lost. • Common Meaning – Semantic issues • There needs to be agreement amongst content providers about the meaning of terms such as ‘title’, ‘article’, ‘research paper’, ‘learning object’ • There will inevitably be difficulties in reaching agreement about the meaning of metadata elements, as they are used differently in different contexts. How to Find a Needle in the HaystackIWMW 2005

46. Metasearch issues: Metadata • Political • The decision to make resources more widely available has implications for the organisations concerned: • It may be seen as a loss of control or ownership • staff may not possess the skills required to support more complex systems • Legal • legal requirements of Freedom of Information Legislation in several countries a significant factor in the dissemination of public sector resources. • The Intellectual Property Rights (IPR) of those providing sources may need to be protected. How to Find a Needle in the HaystackIWMW 2005

47. Why not just use Google? • Its content is limited to the visible Web • Limited search functionality • Can’t search by specific criteria (metadata) such as ‘publication date’, ‘author’, ‘educational level’ • Little quality control • Google Scholar? • Still a web crawl • Evidence that gives unreliable results How to Find a Needle in the HaystackIWMW 2005

48. NISO Metasearch Initiative • This NISO MetaSearch initiative is trying to bring the area of metasearching together around a NISO standard. • “Best Practices for Metasearch” document due out June 15th 2005 • http://www.niso.org/committees/MetaSearch-info.html How to Find a Needle in the HaystackIWMW 2005

49. Overview • Introduction to Cross searching / metasearch • The Problem – why metasearch? • JISC Information Environment • Quick introduction to XML and Web Services • Metasearch Technologies • Z39.50, SRU/SRW, OAI • Metasearch issues • NISO Metasearch Initiative How to Find a Needle in the HaystackIWMW 2005

50. Contact Adrian StevensonLearning Technology ServicesInternet ServicesUniversity of Manchester adrian.stevenson [at] manchester.ac.ukTel: +44 (0) 161 306 3109 How to Find a Needle in the HaystackIWMW 2005