PRIMA DRL Design Jeroen de Jong CDR, 30 Sep 2004

1. PRIMA DRL DesignJeroen de JongCDR, 30 Sep 2004 • Implications of the requirements • Design • Implementation issues

2. Requirement implications • Software needs to be both robust and flexible • Maintainable over many years -> well designed and organized code • Clear distinction between interactive and non-interactive modules • All pipeline algorithms must be available in standard ESO environment

3. Pipeline and DAF • The ESO pipeline will do batch data reduction of nightly data. • The Data Analysis Facility (DAF) will be used for error analysis and calibration of the entire PRIMA dataset. • The pipeline will receive and parse calibration files from the DAF.

4. DAF requirements • For proper scientific verification the DAF will have a database which contains all information about whatever has been done with the data. • The database may only be modified through well defined procedures. • A user interface must provide comprehensive access to all data. • An interactive experimental command lineenvironment will be provided.

5. Modules • Common modules • Mathematical and Astronomical Library • Data Reduction Algorithms • FITS I/O Library • Configuration Files • Common Tools Library (e.g. logfile handling)

6. Modules • Pipeline Modules • CPL Plugins to Data Reduction Algorithms • Pipeline Recipe Executor • Pipeline Configurator (to parse configuration files) • Calibration Matrix Interface

7. Modules • Non-interactive DAF modules: • Database • Analysis Chain Manager:Sets up analysis chains in the database, manages branches to try different algorithms. • Sequencer: Executes algorithms and stores results in database. • Calibration Engine: Generates the Calibration Matrix.

8. Modules • Interactive DAF modules: • Trend Analysis: Set of tools and algorithms for identifying and fitting trends and/or systematic errors. • WWW User Interface: Web application server which provides the standard user interfaces. • Visualisation Tool: Library for drawing standard scientific diagrams and displaying images. • Interactive Console: Provides an interactive command prompt to experiment with the data and

10. Data Reduction Algorithms • Get all relevant data for an observation • Check quality of data • Convert laser metrology to observed wavelength • Compute scalar products of baseline and separation vector as function of time. • Correct scalar products for earth's motion • Compute weighted mean for each star/ref pair • Get calibrated baseline solution • Derive the projected separation • Correct for differential chromatic refraction

11. DAF Database • Keeps relations between data products • XML files? FITS headers? • File for each analysis step • Relations defined with special tags

12. DAF Trend Analysis • Most complex DAF module • Identifies and fits systematic errors in the data • First phase: Interactive plotting and fitting • Second phase: Automatic selection and optimization routines will be introduced. • Probably needs to disentangle several simultaneously occuring trends. • Uses inside DAF same infrastructure as DRA

13. DAF Analysis Chain Manager • Scans and classifies FITS files • Fills the raw data tables in the database • Propagates each classified file throughthe analysis step files • Maintains the consistency of the database after: • (De)selection of data • Changes in the recipe

14. DAF Sequencer • Executes the algorithms through plugin interface • Stores all results in the database • Recursively traverses through analysis chain • Only recomputes steps when data is newer, or the parameters or algorithm have changed.

15. WWW User Interface • Why? Easier than standard GUI: • Many actions just forms and tables. • Only event driven interaction in Applets • Use Web Application Server: • Handles accounts and roles • Keeps state of user in memory • Webpages generated based on user ID and accompanying memory contents

16. Web Server MainServlet VisualizationPlugin AbstractApplication ObsBlockEditor CatalogEditor AnalysisTools CalibrationTool DataSelector ServerContainer Sequencer AnalysisChainManager CalibrationEngine

17. Interactive Console • Interactive scripting and command line environment • All algorithms and utilities from pipeline and DAF are available • Readonly access to database • Scripts can easily be plugged into DAF by extending an API

18. Visualization Tool • Provides all 2D scientific diagrams • Displays images • Must be suitable for Applets • Easy interface from interactive console • Possible candidates: • Scientific Graphics Toolkit (SGT) from NOAA • VO tools???

19. External Libraries • Mathematical and Astronomical: • Is there a sufficient complete ESO numerical library?? • Maybe GNU Scientific Library? • FFTW (Fastest Fourier Transform of the West) • Common Pipeline Library • FITS libraries: CPL FITS I/O with Java wrappers? • GANDALF as base for many DAF modules

20. Programming Languages • ANSI C for Common and Pipeline Modules • All computations and ESO standard infrastructure • Java for most DAF infrastructure modules • Very robust object oriented language (derived from C++) • Standard libraries available for almost every software protocol or task • Jython for the Interactive Consoleand plugin scripts • Fully transparant Java interface (no extra coding required)

21. Language Interfaces Jython Java JNI IDL ANSI C Python FORTRAN

22. Comparison with other language combinations

23. Conclusion • Modules are identified • Investigation of external tools is ongoing • No investigation yet on ESO standards • Reasonable DAF design based on present expertise • Some implementation issues addressed