SSC SI Data Processing Pipeline Plans

1. SSC SI Data Processing Pipeline Plans Tom Stephens USRA Information Systems Development Manager SSSC Meeting – Sept 29, 2009

2. Going Forward • There’s more work to do • Increased scope of existing pipelines • Additional pipelines for non FSI instruments • NASA is increasing the project management requirements for the pipeline development • The goal is for the Science Center to shoulder the vast majority of the burden of this new work • We will deal with the increased software development requirements • We will take on more user support functions in regards to general investigators • Allow the instrument teams to focus on the instrument and their science • However, some increased involvement from the instrument teams will be required.

3. Centralizing the Pipeline Development • Final production pipelines will be implemented at the Science Support Center (SSC) • Goal is to develop a unified system that is easier and more efficient to develop, use, maintain and manage • Benefits • Standardized code base in a controlled set of languages and standards • Initially C/C++, IDL, Python and JAVA • Eases maintenance • Reduces learning curve for new development team members • Unified code base • Maximize reuse of pipeline components • Simplify development • Ease maintenance • Unified interface • Probable that calibration pipelines cannot be fully automated • Reduces friction when operators need to run different pipelines • Improved project management • Maximize efficient use of resources • Ability to focus resources on problematic/challenging areas

4. Roles and Responsibilities – Instrument Teams • Provide algorithms for data reduction • Can be in any “reasonable” format • Teams have software they are using for their own data reduction to do their science, this can be provided “as is” to help build the official pipelines • Science Center staff will translate what is provided into pipeline code to run in the DCS • Includes documentation on software and algorithms • Also includes sample test data • Work with SSC staff to validate implemented pipelines • Work with SSC staff to support guest investigators using instrument for observing • This will proceed along the currently established plans • The goal is for the instrument teams to not need to provide substantive additional support over what they are currently contracted to do

5. Roles and Responsibilities – Science Center • Responsible for final implementation of pipeline code • Coding • Integration into the DCS • Documentation • Testing/Validation • Maintenance • Responsible for ensuring suitable calibration data is taken for pipeline reduction • Operates pipelines and validates output science data

6. SSC-SI Team Interface Points • Each instrument will have a primary and backup Instrument Scientist at the SSC • Understands the instrument • Works with instrument teams to develop pipeline algorithms • Assists in validation of developed pipeline • Help support guest investigators with observation planning and archive data • Helps to define calibration observations • Provide SSC institutional knowledge about instrument • One or more software developers will also work with the instrument teams, through the instrument scientist, to implement and test the data pipeline code. • Implemented software will be provided back to instrument teams for their own use if desired.

7. Initial Impact – FSI teams • Initial pipeline development will proceed as currently planned • Pipelines will be developed and delivered to SSC • SSC will take ownership and be responsible for future maintenance and changes • We will want to get preliminary deliveries of your pipeline software sooner rather than later • We need to work out the mechanism for delivery • Integrate software with the DCS early and often • SSC staff are willing to assist with development if desired or needed

8. Initial Impact – Other Instruments • Increased interaction between SSC staff and SI team • Improve SSC understanding of instrument • Improve SSC understanding of data analysis • Work with SSC staff to develop algorithms/sample code for data processing • Use “whatever” is comfortable to instrument teams • Will take already existing tools/software • SSC will do the work of fitting the algorithms into the DCS pipelining system

9. Longer Term Impacts • Focuses on maintenance aspects of instrument and pipeline development • Should not be major additional effort over current plans • Instrument teams provide input to SSC during operation/maintenance period after initial pipelines developed • Improved algorithms/techniques for data reduction/calibration • Pipeline modification due to changes in instrument response or capabilities • Work with SSC staff to validate implementation and data produced by processing pipelines • SSC continues to be responsible for actual implementation, operation and maintenance

10. Staffing • This is an increase in the scope of the SSC’s pipeline development work • More and expanded pipelines • Increased software development process overhead • Additional staff will be required to accomplish this work • SSC is working on a detailed staffing plan to address the development effort • Details are still being worked with NASA • Final development timeline and impact to SI teams will depend on the final details

11. Timeline • Goal is to have a fully functioning pipeline implemented for each instrument as they start their official commissioning flights after the 2011 down time • However we’d like to get the existing FORCAST pipeline installed in the DCS for early science as a test for the SSC • Not an Early Science Requirement • Help us identify and address issues with integration • Timing of commissioning will drive ramp-up of pipeline development effort and staff over the next two years • Start with re-integration of FORCAST & FLITECAM pipelines • Integrate FIFI LS pipeline • Using “lessons learned”, expand work to implementation of expanded pipelines and develop pipelines for other instruments • For a given instrument we plan to start implementation ~2 years before commissioning flights are scheduled to begin

12. Summary • There is a lot of work to be done to implement the SMOR-II recommended pipelines • The majority of this effort will fall on the SOFIA Science Center • We do not want to place any additional burdens on the instrument teams than are absolutely necessary • We are developing detailed plans of how to implement this work

13. Backup slides

14. High Level Development Process Improvements and updates developed by SI teams and SSC Pipeline/algorithm delivery from SI Team SSC Pipeline Implementation Pipeline Validation Error and Issue Reports Pipelines deployed to production