1 / 14

SSC SI Data Processing Pipeline Plans

SSC SI Data Processing Pipeline Plans. Tom Stephens USRA Information Systems Development Manager SSSC Meeting – Sept 29, 2009. Going Forward. There’s more work to do Increased scope of existing pipelines Additional pipelines for non FSI instruments

haviva-ross
Download Presentation

SSC SI Data Processing Pipeline Plans

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. SSC SI Data Processing Pipeline Plans Tom Stephens USRA Information Systems Development Manager SSSC Meeting – Sept 29, 2009

  2. Going Forward • There’s more work to do • Increased scope of existing pipelines • Additional pipelines for non FSI instruments • NASA is increasing the project management requirements for the pipeline development • The goal is for the Science Center to shoulder the vast majority of the burden of this new work • We will deal with the increased software development requirements • We will take on more user support functions in regards to general investigators • Allow the instrument teams to focus on the instrument and their science • However, some increased involvement from the instrument teams will be required.

  3. Centralizing the Pipeline Development • Final production pipelines will be implemented at the Science Support Center (SSC) • Goal is to develop a unified system that is easier and more efficient to develop, use, maintain and manage • Benefits • Standardized code base in a controlled set of languages and standards • Initially C/C++, IDL, Python and JAVA • Eases maintenance • Reduces learning curve for new development team members • Unified code base • Maximize reuse of pipeline components • Simplify development • Ease maintenance • Unified interface • Probable that calibration pipelines cannot be fully automated • Reduces friction when operators need to run different pipelines • Improved project management • Maximize efficient use of resources • Ability to focus resources on problematic/challenging areas

  4. Roles and Responsibilities – Instrument Teams • Provide algorithms for data reduction • Can be in any “reasonable” format • Teams have software they are using for their own data reduction to do their science, this can be provided “as is” to help build the official pipelines • Science Center staff will translate what is provided into pipeline code to run in the DCS • Includes documentation on software and algorithms • Also includes sample test data • Work with SSC staff to validate implemented pipelines • Work with SSC staff to support guest investigators using instrument for observing • This will proceed along the currently established plans • The goal is for the instrument teams to not need to provide substantive additional support over what they are currently contracted to do

  5. Roles and Responsibilities – Science Center • Responsible for final implementation of pipeline code • Coding • Integration into the DCS • Documentation • Testing/Validation • Maintenance • Responsible for ensuring suitable calibration data is taken for pipeline reduction • Operates pipelines and validates output science data

  6. SSC-SI Team Interface Points • Each instrument will have a primary and backup Instrument Scientist at the SSC • Understands the instrument • Works with instrument teams to develop pipeline algorithms • Assists in validation of developed pipeline • Help support guest investigators with observation planning and archive data • Helps to define calibration observations • Provide SSC institutional knowledge about instrument • One or more software developers will also work with the instrument teams, through the instrument scientist, to implement and test the data pipeline code. • Implemented software will be provided back to instrument teams for their own use if desired.

  7. Initial Impact – FSI teams • Initial pipeline development will proceed as currently planned • Pipelines will be developed and delivered to SSC • SSC will take ownership and be responsible for future maintenance and changes • We will want to get preliminary deliveries of your pipeline software sooner rather than later • We need to work out the mechanism for delivery • Integrate software with the DCS early and often • SSC staff are willing to assist with development if desired or needed

  8. Initial Impact – Other Instruments • Increased interaction between SSC staff and SI team • Improve SSC understanding of instrument • Improve SSC understanding of data analysis • Work with SSC staff to develop algorithms/sample code for data processing • Use “whatever” is comfortable to instrument teams • Will take already existing tools/software • SSC will do the work of fitting the algorithms into the DCS pipelining system

  9. Longer Term Impacts • Focuses on maintenance aspects of instrument and pipeline development • Should not be major additional effort over current plans • Instrument teams provide input to SSC during operation/maintenance period after initial pipelines developed • Improved algorithms/techniques for data reduction/calibration • Pipeline modification due to changes in instrument response or capabilities • Work with SSC staff to validate implementation and data produced by processing pipelines • SSC continues to be responsible for actual implementation, operation and maintenance

  10. Staffing • This is an increase in the scope of the SSC’s pipeline development work • More and expanded pipelines • Increased software development process overhead • Additional staff will be required to accomplish this work • SSC is working on a detailed staffing plan to address the development effort • Details are still being worked with NASA • Final development timeline and impact to SI teams will depend on the final details

  11. Timeline • Goal is to have a fully functioning pipeline implemented for each instrument as they start their official commissioning flights after the 2011 down time • However we’d like to get the existing FORCAST pipeline installed in the DCS for early science as a test for the SSC • Not an Early Science Requirement • Help us identify and address issues with integration • Timing of commissioning will drive ramp-up of pipeline development effort and staff over the next two years • Start with re-integration of FORCAST & FLITECAM pipelines • Integrate FIFI LS pipeline • Using “lessons learned”, expand work to implementation of expanded pipelines and develop pipelines for other instruments • For a given instrument we plan to start implementation ~2 years before commissioning flights are scheduled to begin

  12. Summary • There is a lot of work to be done to implement the SMOR-II recommended pipelines • The majority of this effort will fall on the SOFIA Science Center • We do not want to place any additional burdens on the instrument teams than are absolutely necessary • We are developing detailed plans of how to implement this work

  13. Backup slides

  14. High Level Development Process Improvements and updates developed by SI teams and SSC Pipeline/algorithm delivery from SI Team SSC Pipeline Implementation Pipeline Validation Error and Issue Reports Pipelines deployed to production

More Related