1 / 31

IRMIS3 Status

IRMIS3 Status. Gabriele Carcassi 15 Oct 2010. IRMIS3 Status. This talk is about: what is currently downloadable from the website what is ready for production what is actively supported. Project infrastructure. Build and test server Automated tests.

solomon-chapman
Download Presentation

IRMIS3 Status

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. IRMIS3 Status Gabriele Carcassi 15 Oct 2010

  2. IRMIS3 Status • This talk is about: • what is currently downloadable from the website • what is ready for production • what is actively supported

  3. Project infrastructure Build and test serverAutomated tests SourceForge project (mailing list, bug tracking, wiki).Monthly release cycle. Production server at BNL since January 2009 IE or firefox on Windows/Linux/Mac supported “alpha” - AuthZ and AuthN only designed Likely implemented by the end of the year

  4. IRMIS3 Service Architecture

  5. IRMIS3 architecture Browser(Firefox, IE, …) Web server(Glassfish) Javaapplets IRMIS DB (MySQL) Data service AJAX components XML protocol • Data service enforces business rules • (clients can be dumb) Data service allows db schema orbusiness rule changes while in production Scripts and CLI(perl, Python, …) Data service provides a good abstractionlayer on which to integrate

  6. Software stack 3rd party Java apps Jython scripts 3rd party Perl/Pyton scripts Integration with external tools (i.e. physcs) Few database utilities: backup, consistency check, etc… Web applications Client JavaScript bridge Applets and Widgets Java Client API XML protocol (REST style WS) Data Service layer Server Database layer

  7. Getting data <componentType id="90" description="Oscilloscope: 4 Channel @ 500MS/s; 500MHz BW" name="HP54540A"> <manufacturer id="5" name="Agilent (HP)"/> <formFactor id="5" description="Freestanding"/> <functions> <function id="46" description="CCMS"/> <function id="8" description="Instrument"/> </functions> <requires> <interface id="143" description="GPIB_Slave" relType="control"/> <interface id="5" description="Freestanding" relType="housing"/> <interface id="78" description="120VAC" relType="power"/> </requires> <presents> <interface id="234" description="Port" relType="control"/> </presents> <ports> <port name="Chnl 1"> <portType id="13" name="BNC-F" group="RF Connectors"> <pinDesignator id="303" name="1"/> </portType> <pin usage="Chnl 1" pinDesignatorId="303"> <signalType id="1" direction="IN"/> </pin> </port> <port name="Chnl 2">...</port> <port name="Chnl 3">...</port> <port name="Chnl 4">...</port> <port name="RS232">...</port> ... </ports> </componentType> XML returnedREST style (data service) De-normalized data Id references This is an example of what the data service might deliver

  8. Writing data XML describing a transaction sent through a POST <transaction xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='http://xml.bnl.gov/schema/irmis' xsi:schemaLocation='http://xml.bnl.gov/schema/irmis irmis.xsd'> <create> <manufacturer name="Acopian"/> </create> <update> <componentType id="2" name="Site" description="Overall facility. Root of housing hierarchy."> <manufacturer id="1" name="None"/> <formFactor id="1" description="Virtual"/> <properties/> <functions/> <requires/> <presents/> <ports/> </componentType> </update> <delete> <formFactor id="4" description="IndustryPack"/> </delete> <connect> <cable color="blue" label="AA" portAId="123" portBId="321"/> <conductor pinAId="123" pinBId="321"/> <conductor pinAId="124" pinBId="322"/> </connect> </transaction> different independent “commands” no lock “FOR UPDATE” idempotent actions (error or same result) allows remote/disconnected work

  9. Implementation Java SE 6 Java EE 5Glassfish 2.x/3.x Stream out XML as soon as possibleto reduce delay Servlet Read Business logic JPA SAX MySQL JAXB Write JAXB to convert transaction requestto Java objects JPA vs JDBC: simpler queries to maintain6% performance loss

  10. Performance • Dell Precision M4300 laptop, Intel Core Duo T7500, 2.20 GHz - APS dataset • Entire component data set streamed in about 5 seconds (31338 components, 63911 relationships) • Most other queries under 100 ms (manufacturers 88ms, interfaces 95ms) • Roughly half of the time is spent in DB access and half in XML generation • Streaming starts right after DB access • Only tuning done is on the database query (was “good enough”) • Other areas: hardware/os tuning, xml generation improvements, profiling, caching, …

  11. IRMIS3 Client API

  12. Client API • To access the service, XML and HTTP is all you need • Available in all languages • Two libraries provided that allow to work with objects • PHP API from MSU • Java API from BNL • Only going to talk about the Java API

  13. Java API: design principles • The Java API implement the XML protocol and provides objects that represent the data • All reads are cached: no need to keep caches in your code. • All reads are synchronized • you can have reads on multiple threads on the same data and will not trigger multiple reads on the server (extremely useful for GUIs)

  14. Java API design principles • The read access is object oriented • Data is exposed through interfaces • Component, ComponentType, Manufacturer • Some classes function as “finders” for the classes containing the data • Components.getById(123), FormFactors.getByName(“Rack”) • You can follow references from one object to the other • component.getComponentType().getManufacturer()

  15. Java API design principles • The write access is protocol oriented • You create a transaction • You call methods to add elements in the transaction • Manufacturers.createManufacturer(transaction, “ACME”) • You save the transaction, which may fail • The read access is refreshed only after the data is saved

  16. Importing components from Excel (using Jython) from gov.bnl.irmis.api import * from java.net import URI from java.io import File from jxl import * workbook = Workbook.getWorkbook(File(“components.xls")) sheet = workbook.getSheet(0) nRows = sheet.getRows() nColumns = sheet.getColumns() irmisDB = IrmisDB.getInstance(); irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data")) transaction = irmisDB.createTransaction(); for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber) transaction.save(); print "Operation successful"

  17. Importing from Excel (Jython) from gov.bnl.irmis.api import * from java.net import URI from java.io import File from jxl import * workbook = Workbook.getWorkbook(File(“components.xls")) sheet = workbook.getSheet(0) nRows = sheet.getRows() nColumns = sheet.getColumns() irmisDB = IrmisDB.getInstance(); irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data")) transaction = irmisDB.createTransaction(); for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber) transaction.save(); print "Operation successful" Load excel file

  18. Importing from Excel (Jython) from gov.bnl.irmis.api import * from java.net import URI from java.io import File from jxl import * workbook = Workbook.getWorkbook(File(“components.xls")) sheet = workbook.getSheet(0) nRows = sheet.getRows() nColumns = sheet.getColumns() irmisDB = IrmisDB.getInstance(); irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data")) transaction = irmisDB.createTransaction(); for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber) transaction.save(); print "Operation successful" Connect to the service

  19. Importing from Excel (Jython) from gov.bnl.irmis.api import * from java.net import URI from java.io import File from jxl import * workbook = Workbook.getWorkbook(File(“components.xls")) sheet = workbook.getSheet(0) nRows = sheet.getRows() nColumns = sheet.getColumns() irmisDB = IrmisDB.getInstance(); irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data")) transaction = irmisDB.createTransaction(); for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber) transaction.save(); print "Operation successful" Loop over the rows

  20. Importing from Excel (Jython) from gov.bnl.irmis.api import * from java.net import URI from java.io import File from jxl import * workbook = Workbook.getWorkbook(File(“components.xls")) sheet = workbook.getSheet(0) nRows = sheet.getRows() nColumns = sheet.getColumns() irmisDB = IrmisDB.getInstance(); irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data")) transaction = irmisDB.createTransaction(); for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber) transaction.save(); print "Operation successful" Get data outCreate components

  21. Fluent API • Navigating through hierarchies can be tedious • On top of the “beans” API there is a fluent API (internal domain specific language) Search API Builders Change Set API

  22. IRMIS3 User interface

  23. UI design • Web interface: uses applets and Web Start (both greatly improved with Java 6u10) • Uses Swing standard components plus open source (SwingX, Flamingo, JIDE, …) • UI elements can optionally be used in custom pages and integrated with javascript • UI available only for Components and Cabling

  24. UI elements Component hierarchy tree widget (filtering, sorting, search, …) Component pathwidget

  25. Component applications

  26. Cabling applications

  27. Status by area

  28. Components • Support for component and component types is production level • Data model has been tested extensively through previous use at other sites • Performance tests were done using APS datasets • Most of the work now is in UI development • Most applications at their 3rd iteration

  29. Cabling • Support for cabling is still beta • No considerable amount of data has been put into it • No performance tests done (see lack of data) • UI at their second iteration • Need users!

  30. Lattice • There is preliminary support for lattice • Only through the API, no UI • A separate tree structure of “elements”, each of an “element type”. Each relationship has alignment information. • Lattice cannot be modifiedin place: copy on write createsa new version • Right now, provided as-is 2 1 2 1 1 1 1

  31. PV • PV service and API is considered alpha. No UI yet. No crawler scripts provided yet.

More Related