Overview of TIO-index implementations

2. Overview of TIO-index implementations The DAG, GIDS and Desire TIO/LDAP index servers Henny Bekker

3. Overview of TIO-index implementations Agenda • General overview of LDAP/TIO-indexes • What are TIO indexes • The generic model • Some specific implementations • The generic Desire TIO index server • The Ericsson DAG server • The GIDS server • Open Issues • The scope and communication between LDAP/TIO index servers • Exchanging TIO’s • Local access policy • Access restrictions • Security requirements • Senario’s What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

4. General overview of LDAP/TIO-indexes Tagged Index Object’s A TIO consists of: • Meta information such as • A mime header defining the object • An object type identifier that uniquely identifies the subtree and scope • One or more URI's that will form the base of the created referrals • The security options and credentials such as a PGP or S/MIME key • The update type indicating the type of TIO (e.g. full or incremental) • The payload • The tokenization types headers (e.g. Full, Token, RFC822 etc) • Indicating which information is ‘tokenizated’ and which delimiters to use • The TAG list • Containing multiple consecutive tags which might be grouped using a dash. What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

5. General overview of LDAP/TIO-indexes Content-Type: application/index.obj.tagged; dsi="1.3.6.1.4.1.5062.1.99.1.114"; base-uri= "weetmuts.surfnet.nl:389/o=SURFnet, c=NL" Content-Length:6219 version: x-tagged-index-1 updatetype: total thisupdate: 950688539 BEGIN IO-Schema sn: FULL cn: FULL . o: TOKEN END IO-Schema BEGIN Index-Info sn: 22/Arends -6/Bezemer -4/Bos -8/Neggers . -2-3,5-9,11,14-15,18-19/+31 302 305 305 -12/030-2305327 o: 1/SURFnet END Index-Info What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

6. General overview of LDAP/TIO-indexes Tagged Index Object’s (cont.) What is it used for: • Provide pointers to servers which most likely contains the requested information • The number of false hits is depending on the choice of attribute tokenization types • Performing phrase searches is depending on the tokenization of the fields • Features a full or incremental update (which uses potentially less bandwidth) What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

7. General overview of LDAP/TIO-indexes The generic model • A TIO interface • For importing, deleting and in some cases exporting TIO’s from the index • Implementing authentication control • A TIO searchable index • For searching the index on referrals to other information services • Accessible through the TIO query interface • The LDAP query interface • Containing a LDAP gateway to the query interface of the TIO index • Can act as an LDAPv2 chaining server or as an LDAPv3 referral server What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

8. Some specific implementations The generic Desire TIO index server • Sponsored by the European Community and build by SURFnet & DFN in cooperation with Dante. • The server consists of: • The TIO index server • Using the MySQL database engine for storing and searching the TIO’s • Containing a TIO push/pull interface and a database for storing TIO’s. • An HTTP frontend for direct access to the TIO index server by the NPS. • A Native Protocol Server (NPS) for access using the LDAP protocol • For connecting clients using specific communication protocols such as LDAPv2, LDAPv3 or WhoIS++. • For connecting directory servers with a specific communication protocols such as LDAPv2, LDAPv3 or WhoIS++. • An LdapCrawler for gathering and converting LDIF files to TIO’s • Currently no encryption of TIO’s implemented • Currently only support for LDAPv2. (no characterset conversion problem) What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

9. Some specific implementations The generic Desire TIO index server (cont.)

10. Some specific implementations The Desire LDAP/TIO index server (cont.)

11. Some specific implementations The Desire LDAP/TIO index server (cont.) • Unfortunately we don’t have yet any performance figures  • The package is on the brink of being completed  • Presumably the GIDS index server will be faster • The generic MySQL engine is slow compared to a dedicated TIO database. • Current implementation • Available on Linux and (hopefully) on Digital Unix • The source code and executable for Linux of • The LdapCrawler with an integrated LDIF2TIO converter • The TIO index (using MySQL v3.23.6) • The LDAP NPS implemented using the Open-LDAP v1.2.10 with an API to the TIO index What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

12. Some specific implementations The Ericsson DAG server • Offspring of the TISDAG project • Aimed to provide a solution for an uniform telephone directory containing numbers without a centralized database • The server consists of: • The DAG (Directory Access Gateway) index server • Implemented using the TimesTen “In-Memory” database engine for storing and searching the TIO’s. • One or more CAP (Client Access Point) modules • For connecting clients using specific communication protocols such as LDAPv2, LDAPv3 or WhoIS++. • One or more SAP (Server Access Point) modules • For connecting directory servers with a specific communication protocols such as LDAPv2, LDAPv3 or WhoIS++. What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

13. Some specific implementations The Ericsson DAG server (cont.)

14. Some specific implementations The Ericsson DAG server (cont.) • Unfortunately we aren’t allowed to present exact figures  • The next version is said to be much faster  • Performance figures • Response times • Use LDAPv3 referral requests to measure the response time of the referral server without doing chaining or following referrals. • The mean response time related to the number of parallel search queries. (measured with a large number of queries) • Number of queries/second (or minute??) • The number of parallel requests • Related to the response time • Maximum number of entries in the TIO index • Bounded by the memory size and the algorithm used to search the index What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

15. Some specific implementations The GIDS server • Offspring of the TISDAG project • Second implementation of the TISDAG TIO index server • The server consists of: • An index server • Using a dedicated database engine for storing and searching the TIO’s. • Is using a dedicated communication protocol (analogous with LDAP) to communicate with the CAP and SAP modules. • One or more CAP (Client Access Point) modules • For connecting clients using specific communication protocols such as LDAPv2, LDAPv3 and HTTP • One or more SAP (Server Access Point) modules • For connecting directory servers with a specific communication protocols such as LDAPv2 and LDAPv3 • An LdapCrawler for gathering and converting LDIF files to TIO’s • With support for LDAPv2 and LDAPv3 and character-set conversion What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

16. Some specific implementations The GIDS server (cont.) • Performance figures • Response times • Measured with LDAPv3 requests (an LDAPv3 bind, sending the query, receiving the message, doing an unbind operation) • With one sequence of LDAPv3 requests, a mean time of approximately 23 msec. per LDAPv3 request. • With 10 simultaneously LDAPv3 requests approximately 150 msec per LDAPv3 request. • Maximum number of queries/second • Approximately 65 LDAPv3 queries/second • With LDAPv2 the number will be lower because the server has to do chaining. • Maximum number of entries in the TIO index • Bounded by the memory size and the algorithm used to search the index • Current demo implementation (CH, DE, NL, NO & SE) 120K tokens of 450 different data sets. (which consumes about 35-Mbyte of memory). What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

17. Open issues The scope and communication between TIO index servers • Scope • Centralized versus distributed LDAP/TIO engines • Location of the TIO/LDAP-index server • Located close by (in network terms) the end users to minimize the RTT • Located close by the referred LDAP servers to minimize the RTT related to LDAPv2 chaining • Exchanging TIO’s • Global TIO collection versus distributed collections on country level • Distributed to country level • Knowledge base or ‘where to find what’?? • Encrypted transport via HTTP • Push or pull ?? What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

18. Open issues Local access policy • Security requirements • Personal data is subjected to privacy legislation • For public data other security requirements might be imposed • No unauthorized access to local directory servers • Only accessible by local inhabitants and peer countries • All applications able to access the index should be known • Only a limited number of referrals might be returned • No ‘access denied’ messages • Don’t show entries which are not accessible • Access restrictions • Restrict access to the TIO/LDAP-index server • Restrict access to the LDAP servers containing the information • Chaining versus LDAPv3 referral • HTTP access control versus LDAP access control • Access via HTTP proxies versus LDAP proxies What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

19. Open issues Senario’s • Create trusted relation between country level TIO servers • Only peers will communicate with each other • Besides the local LDAP clients • A peer will enforce their own local access rules • The TIO index server should only be accessible by known clients • The LDAP query will be chained to the remote peers • The TIO objects of the peer country should deliver referrals which will point to a known access point e.g. an LDAP proxy or the FLDSA • An LDAP search requests from a known LDAP client must be chained to the known access point. • The number of known access points should be limited • The TIO objects cannot be duplicated between the peers

20. Open issues Senario’s (cont.)

21. Discussion..