ALMA Integrated Computing Team Coordination & Planning Meeting #2 Santiago, 28-29 January 2014

1. ALMA Integrated Computing TeamCoordination & Planning Meeting #2 Santiago, 28-29 January 2014 ASDM relational database Rafael Hiriart / Jorge Avarias

2. Original Goals • Our main motivation to create this database was to improve DataCapturer and QuickLook. • DataCapture was running out of memory, observations were reaching a limit of ~40 minutes with 32 antennas. • The original DataCapture “incremental writes” architecture – based on incremental writes at the level of the ASDM and XML Store – was dropped because of lack of resources in HLA/ASDM and Archive. It was suggested at this time to explore a relational DB solution. • We implemented a workaround for the Pointing table, the biggest ASDM table. By saving this table incrementally to a file and streaming it at the end of the observation to the binary store, we now support long observations (several hours). • We continued working on DC relational database aiming to improve QuickLook. This subsystem, which currently relies in a complicated collaboration between its own components, DataCapture an Quicklook, can be refactored to be a simple application that queries the database, performs a little processing and plot results.

3. Status First version of the relational database has been delivered (ICT-150) as part of release 10.4. It is working in the OSF. A new version of QuickLook is being delivered with 10.6. This has become quite urgent because of frequent “freezes” of QuickLook running in the OMC. Besides, QuickLook plots don't scale well with the number of antennas. A more interactive application is required, able to support filtering on antennas, scans, baselines, etc.

4. Demo

5. Implementation Based on Hibernate The Hibernate mapping file is generated using Java reflection to know what are the columns and the keys of every table. ASDM Java row classes cannot be used directly as mapping classes for Hibernate. New wrapper classes were created to circunvent problems: Define the no-param constructor. Setters need to be public. Some getters need to be fixed (they throw runtime exceptions). Database is auto-generated by Hibernate, currently deployed in Scheduling database. Custom Hibernate types were implemented to handle simple types like angle, temperature, speed, etc.; and complex types like timeInterval, arrays, IDL enumerations.

6. Problems/Improvements Wrapper classes were manually (and painfully) produced. Mappings are created statically. The system is fragile over changes in the ASDM. The ideal way to create these artefacts would be to generate them as part of the HLA/ASDM build. Relationships between tables (FK) are not supported. Arrays are stored in the database as strings following the ASDM array serialization (<num-dim> <dim1> <dim2> … <value-1> <value-2> ...). Only Java is supported.

7. Our plan We stop here! The mappings, as they are, fullfill our requirements to refactor QuickLook. We don't have plans (or resources) to continue dedicating to this task. Jorge is in transition to 100% Scheduling. Other resources in Control are fully allocated. We are well aware of the interest to use this database in a broader context, but The database could need significant work/modifications to be useful for other applications (e.g., QA). DB is temporary. If a permanent DB is needed, APO needs to be involved. Maintainance, migrations, data duplication, etc., will need to be addressed. These developments are well outside Control group's scope and expertise.

8. Decision from Last Leads Meeting ICT-2048, action for Alisdair Manning: Provide a list of options to ICT management about possible delays to "fund" making the ASDM relational database permanent, and integrated into the code generation system. To be discussed in Alisdair's presentation.