DiGIR Di stributed G eneric I nformation R etrieval

1. DiGIRDistributed Generic Information Retrieval Stan Blum, Dave Vieglais, P.J. Schwartz DiGIR

2. Project Goals • To define a protocol for retrieving structured data from multiple, heterogeneous databases • To build a reference implementation of said protocol DiGIR

3. Design Goals • To use open protocols and standards, such as HTTP, XML, and UDDI to leverage existing and emerging technologies • To de-couple the protocol, software and semantics • To automate the establishment of a new data provider as much as possible DiGIR

4. High-level Architecture • Protocol • Provider • Portal • Registry DiGIR

5. Protocol • Defines request and response message formats for communication between Provider and Portal • Assumes Providers conform to a known federation schema • Remains flexible to allow for federation schema pluggability DiGIR

6. Provider • Makes structured data available to portals • Communicates via protocol compliant messaging only • Complies with a known federation schema • Supplies meta-data to describe data classification and availability DiGIR

7. Portal • The entry point for a “user” • Can make requests of N number of providers • Communicates via protocol compliant messaging only • Queries registry for available providers • Can determine, based on provider meta-data, whether a provider should be queried DiGIR

8. Project Information • The DiGIR project is a collaborative effort • DiGIR is currently established as an open source project on SourceForge (http://sourceforge.net). • Further documentation is available on the SourceForge site. • Please join us in collaborating! DiGIR

9. Protocol Details DiGIR

10. Protocol Details • Specified in an XML Schema (.xsd) • Intended to work in conjunction with federation schemas, also expressed as XML Schemas • Actual request and response documents are instance documents conforming to both the protocol schema and a federation schema DiGIR

11. <request xmlns="http://www.namespaceTBD.org/digir" xmlns:darwin="http://www.namespaceTBD.org/darwin" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.namespaceTBD.org/digir digir.xsd http://www.namespaceTBD.org/darwin darwin.xsd"> <header> <requestType>search</requestType> </header> <search> <dbName>myDiggableBipesDB</dbName> <filter> <and> <in> <list xsi:type=“darwin:list”> <darwin:Month>11</darwin:Month> <darwin:Month>12</darwin:Month> </list> </in> <equals> <darwin:Genus>Bipes</darwin:Genus> </equals> </and> </filter> <records start=“0” count=“50”> </search> </request> DiGIR

12. Request Explanation • Composed of elements from the protocol namespace (default) and the schema namespace • <header> contains information about the payload • <search> contains dbName, filter, and record specification (will also specify result format) • <filter> is effectively an XML representation of a SQL where clause • This search request is for the first 50 specimen records that are genus Bipes and were found in the months of November or December. DiGIR

13. LOPs (logical operators) <and> <or> <andNot> <orNot> Can be nested COPs (comparison ops) <equals> <lessThan> <lessThanOrEquals> <notEquals> <greaterThan> <greaterThanOrEquals> <like> <in> (multi value) Filter Building DiGIR

14. What “binds” the schemas? • The protocol schema defines various abstract types and elements: <xsd:element name="searchCondition" abstract="true"> <xsd:element name="alphaSearchCondition" abstract="true“ substitutionGroup="searchCondition"> <xsd:complexType name="listType" abstract="true" /> <xsd:complexType name="numericListType" abstract="true" /> • A federation schema must define searchable concepts, or groups of them, as substitutable for these abstract elements or extensions of the abstract types <xsd:element name="Species" type="xsd:string“ substitutionGroup="digir:alphaSearchCondition" /> DiGIR

15. <xsd:complexType name="list <xsd:complexContent> <xsd:extension base="digir:listType"> <xsd:sequence> <xsd:choice> <xsd:element ref="ScientificName" maxOccurs="unbounded"/> <xsd:element ref="Kingdom" maxOccurs="unbounded" /> <xsd:element ref="Phylum" maxOccurs="unbounded" /> <xsd:element ref="Class" maxOccurs="unbounded" /> <xsd:element ref="Order" maxOccurs="unbounded" /> <xsd:element ref="Family" maxOccurs="unbounded" /> <xsd:element ref="Genus" maxOccurs="unbounded" /> <xsd:element ref="Species" maxOccurs="unbounded" /> <…> </xsd:choice> </xsd:sequence> </xsd:extension> </xsd:complexContent> </xsd:complexType> DiGIR

16. Why “bind” like this? • To provide data-typing (string, numeric, etc.) for various concepts within operators at an abstract level (e.g. LIKE only valid for string data; IN allows for multiples, but in a controlled fashion) • To allow for federation schemas to simply classify data as types without having to redefine/extend operators DiGIR

17. Request Issues • Do we need another abstract element such as dateSearchCondition? • What information will be useful in the header? • How should we specify the format of the results? What standard formats should be offered (I.e. brief, full?). • Will tblName be part of the meta-data required of providers? • What concepts of Darwin Core 2 are searchable? DiGIR

18. Response Prototype <response xmlns="http://www.namespaceTBD.org/digir" xmlns:darwin="http://www.namespaceTBD.org/darwin" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.namespaceTBD.org/digir digir.xsd http://www.namespaceTBD.org/darwin darwin.xsd"> <header> <!-- contents TBD --> </header> <content> <record> </record> </content> <diagnostics> </diagnostics> </response> DiGIR

19. Response Issues • How do we format and validate the response content? • What elements are needed for the <header>, if any? • Do we always have diagnostics, or only if there is an error? • Should a finite set of diagnostics be created and maintained in its own XML Schema? Will there ever be a diagnostic that is specific to a federation schema? DiGIR

20. Provider Details DiGIR

21. Provider Details • Implemented as a web application that answers questions • Interface is not specific to a particular information domain • No state information is recorded • Each request is treated as unique and uninfluenced by previous requests • Must always generate a valid response • Consists of four key components • Request handler • Filter handler • Result set cache • Response generator DiGIR

22. Request Handler • Receives XML document • Validates document • Generates internal structures for further processing DiGIR

23. Filter Handler • Internal structural representation of filter (query) structure • Responsible for generating a native query string for querying the database • Communicates with UDDI to obtain standard database definition • Custom configured to work with specific database implementation DiGIR

24. Result Set Cache • Contains the results of applying a query • Responsible for generating the response records in the requested format • Somewhat directly integrated with the response generator DiGIR

25. Response Generator • Generates the response XML document • Serializes the response header information • Serializes diagnostic information • Serializes the requested subset of records DiGIR

26. Provider Configuration DiGIR

27. Portal Details DiGIR

28. Portal Details • Divided into two distinct components: a presentation layer and PortalServices • The presentation layer supports the UI and translates requests (HTTP requests from forms or links) into protocol compliant XML requests • The presentation layer also handles all display issues involving the responses, such as format, sorting, collating, etc… • The presentation layer is envisioned to be an application server/web server implementation DiGIR

29. Portal Details • PortalServices handles all external network activity (UDDI calls, provider calls, etc) • PortalServices limits provider calls to those necessary based on provider meta-data • PortalServices threads provider calls for increased performance (I.e. response time) • PortalServices is envisioned to be a webapp and supporting classes running within an application server, such as TomCat DiGIR

30. PortalServices • RegistryAccess • ProviderCache • PortalConfig • PortalServlet • PortalRequestHandler • ProviderFilterer • Marshallers DiGIR

31. Portal Issues • What information will be stored in UDDI about a provider? • What information will be known for communicating with a Provider (I.e. IP address, port, etc…?) • What meta-data will be provided and what are the rules for using such data for provider filtering? • What requirements are there for logging and monitoring? DiGIR