1 / 31

Lavoisier 2.0

2.0. Lavoisier 2.0. Tsukuba, KEK, 21 December 2010 Sylvain Reynaud. Why Lavoisier ?. Initially developed for the operation portal of EGEE, which aggregates data from many remote data sources these data sources use heterogeneous (and sometimes changing) technologies

sukey
Download Presentation

Lavoisier 2.0

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 2.0 Lavoisier 2.0 Tsukuba, KEK, 21 December 2010 Sylvain Reynaud

  2. Why Lavoisier ? • Initially developed for the operation portal of EGEE, which aggregates data from many remote data sources • these data sources • use heterogeneous (and sometimes changing) technologies • can be unavailable and/or have high latency • requirements may change => need a framework to enable aggregating data easily, efficiently and reliably • Now re-used in project EGI

  3. RDBMS WS RESTful LDAP What is Lavoisier ? • Lavoisier is a web service… • extensible • providing a unified view • of data coming from heterogeneous data sources plug-in plug-in plug-in XML 01011 11010 00111

  4. RDBMS WS RESTful LDAP How to build your own data view ? • Lavoisier is a web service… • extensible • providing a unified view • of data coming from heterogeneous data sources plug-in plug-in plug-in XML 01011 11010 00111

  5. RDBMS WS RESTful LDAP 1) Check if techno is supported 01011 11010 00111

  6. RDBMS WS RESTful LDAP 2) Declare the data views 01011 11010 00111

  7. XML RDBMS WS RESTful LDAP XML XML connector XML 3) Declare the plug-ins to use • Each data view is composed of… • plug-ins • 1 connector • collects data from • external data sources • other data views • can be configured • statically • with another data view • with user query XML 01011 11010 00111

  8. XML RDBMS WS RESTful LDAP trigger trigger connector XML 3) Declare the plug-ins to use • Each data view is composed of… • plug-ins • 1 connector • [ 0-N transformers ] • [ 0-1 cache ] • [ 0-N cache refresh triggers ] • period ("cron-like") • access to expired data • cascading cache refresh • … XML cache transformer transformer XML 01011 11010 00111

  9. XML RDBMS WS RESTful LDAP trigger trigger connector XML 4) Configure each data view • Each data view is composed of… • plug-ins • configuration • data validation (at each step) • data expiration • timeout for input data retrieval • error management • tolerance • fallback rules XML cache transformer transformer XML 01011 11010 00111

  10. RDBMS WS RESTful LDAP 5) Add serializers XML HTML YAML JSON • Each serializer is composed of… • plug-ins • 1 serializer plug-in XML 01011 11010 00111

  11. RDBMS Monitoring (Nagios DB) EGI sites (GOC-DB) Helpdesk (GGUS) RESTful WS LDAP XSL depends expired startup aggregator RDBMS SOAP HTTP XSLT Repeat this for each data view… XML HTML YAML JSON XML XSLT YAMLXML PathSelector 01011 11010 00111

  12. XSL Monitoring (Nagios DB) Helpdesk (GGUS) EGI sites (GOC-DB) LDAP XSL depends expired startup RDBMS SOAP XSLT aggregator XSLT HTTP 6) Connect data views XML HTML YAML JSON XML XSLT YAMLXML PathSelector 01011 11010 00111

  13. XSL Monitoring (Nagios DB) Helpdesk (GGUS) EGI sites (GOC-DB) LDAP depends startup expired HTTP XSLT RDBMS aggregator HTTP XSLT SOAP 7) Connect Lavoisier instances XML HTML YAML JSON XML XSLT YAMLXML XML files PathSelector 01011 11010 00111

  14. XSL Monitoring (Nagios DB) Helpdesk (GGUS) EGI sites (GOC-DB) LDAP startup expired aggregator RDBMS XSLT 8) Query the data views XML HTML POST YAML JSON GET • Query data views through… • REST • http://host/aggregator.json/root/n[@id='...'] • with GET/POST request • SOAP XML XSLT YAMLXML 01011 11010 00111

  15. Monitoring (Nagios DB) Helpdesk (GGUS) EGI sites (GOC-DB)

  16. Monitoring (Nagios DB) Helpdesk (GGUS) EGI sites (GOC-DB)

  17. Usage in EGEE uses ≈ 50 data views source: Cyril L'Orphelin

  18. Usage in EGI currently deployed in : Czech Republic Belarus Portugal / Spain Greece source: Cyril L'Orphelin

  19. trigger connector How to extend Lavoisier ? serializer cache transformer transformer ?

  20. trigger connector 1) Select plug-in type serializer cache validator transformer deserializer ?

  21. large amount of data random access tree-based event-based fixed schema support non-XML input Object Model easiness efficiency trigger standard connector Data Binding DOM DOM4J SAX-like Stream 2) Select interface type serializer cache validator transformer deserializer 00101110

  22. X X X X X X X X X X X X X X X X X trigger connector X X X X X 3) Implement selected interface Data Binding DOM DOM4J SAX-like Stream serializer cache validator transformer deserializer

  23. trigger connector Chaining plug-ins… Data Binding DOM DOM4J SAX-like Stream serializer X X X X X X X X cache validator transformer X X X X deserializer X X X X X X X X X X Possible links between connectors and other plug-ins…

  24. connector Chaining plug-ins : the usual way <a> <b c="1"/> <b c="2"/> <b c="3"/> </a> transformer <bar> <x> <y>1</y> <y>2</y> </x> <x y="3"/> </bar> transformer 01001110 11010011 10011001 00101101 01001110 11010011 10011001 00101101 <foo> <m n="1"/> <m> <n>2</n> <n>3</n> </m> </foo>

  25. connector Chaining plug-ins : DOM trees <a> <b c="1"/> <b c="2"/> <b c="3"/> </a> transformer transformer 01001110 11010011 10011001 00101101 01001110 11010011 10011001 00101101

  26. used memory max DOM Events XML size connector Chaining plug-ins : XML events <a> <b c="1"/> <b c="2"/> <b c="3"/> </a> transformer transformer 01001110 11010011 10011001 00101101 01001110 11010011 10011001 00101101

  27. Efficiency engine optimizations optimized plug-ins chaining in-memory/on-disk caches plug-ins optimizations event-based Reliability persistent cache of views data validation error management Reusability of development efforts plug-ins of data (thanks to cache) raw data transformed data Maintainability users not impacted by technology changes performance tuning split competencies / roles Conclusion : main benefits

  28. XML POST GET RDBMS cache transformer WS RESTful LDAP transformer trigger trigger connector Conclusion : split competencies • users • business logic • service administrator • characteristics of data and data sources • usage, amount, expiration, latency, dependencies… • configuration capabilities of Lavoisier • validation, filtering, cache and fallback mechanisms… • plug-ins developer • technologies used by the data sources 01011 11010 00111

  29. BACKUP SLIDES

  30. Example : XSDTransformer <xsd:element name="site"><xsd:complexType> <xsd:sequence> <xsd:element name="vo"><xsd:complexType> <xsd:annotation> </xsd:annotation> <xsd:attribute name="name"/> <xsd:attribute name="url"><xsd:annotation> </xsd:annotation></xsd:attribute> </xsd:complexType></xsd:element> <xsd:element name="nbVos" type="xsd:string"><xsd:annotation> </xsd:annotation></xsd:element> </xsd:sequence> <xsd:attribute name="id"/> </xsd:complexType></xsd:element> <site id="CC-IN2P3"> <services> <voms>cclcgvomsli01.in2p3.fr</voms> </services> <organizations> <vo name="EGEODE"><ms>true</ms></vo> <vo name="DTEAM"><ms>false</ms></vo> </organizations> </site> <site id="CC-IN2P3"> <vo name="EGEODE" url="https://cclcgvomsli01.in2p3.fr:8443/voms/egeode"/> <vo name="DTEAM"/> <nbVos>2</nbVos> </site>

  31. Example : XSDTransformer <xsd:element name="site"><xsd:complexType> <xsd:sequence> <xsd:element name="vo"><xsd:complexType> <xsd:annotation><lav:template match="organizations/vo"/></xsd:annotation> <xsd:attribute name="name"/> <xsd:attribute name="url"><xsd:annotation> <lav:template match="ms[text()='true' or text()='TRUE']/text()"> https://<lav:value-of select="ancestor::organizations/preceding-sibling::services/voms/text()"/> :8443/voms/<lav:value-of select="ancestor::vo/@name">return $.toLowerCase();</lav:value-of> </lav:template> </xsd:annotation></xsd:attribute> </xsd:complexType></xsd:element> <xsd:element name="nbVos" type="xsd:string"><xsd:annotation> <lav:add><lav:value-of select="count(preceding-sibling::organizations/vo)"/></lav:add> </xsd:annotation></xsd:element> </xsd:sequence> <xsd:attribute name="id"/> </xsd:complexType></xsd:element> <site id="CC-IN2P3"> <services> <voms>cclcgvomsli01.in2p3.fr</voms> </services> <organizations> <vo name="EGEODE"><ms>true</ms></vo> <vo name="DTEAM"><ms>false</ms></vo> </organizations> </site> <site id="CC-IN2P3"> <vo name="EGEODE" url="https://cclcgvomsli01.in2p3.fr:8443/voms/egeode"/> <vo name="DTEAM"/> <nbVos>2</nbVos> </site>

More Related