AeroADL — One year later

1. AeroADL — One year later Scott Houchin Project Leader, VIIRS SDR analysis and data access tools scott.houchin@aero.org, 571-307-3914 Engineering & Technology Group, Electronics and Sensors Division 15 November 2012

2. Stage inputs and run ADL with one command! View this slide in slide show mode • Given 3 VIIRS science RDR files:/rdr/RNSCA-RVIRS_npp_d20111128_t0829253_e0830507_b00439_c20111128102607267562_noaa_ops.h5/rdr/RNSCA-RVIRS_npp_d20111128_t0830507_e0832160_b00439_c20111128102729213164_noaa_ops.h5/rdr/RNSCA-RVIRS_npp_d20111128_t0832160_e0833414_b00439_c20111128102901627326_noaa_ops.h5 • Run the ADL VIIRS SDR Controller, starting with RDR data, stopping after calibration, but using an experimental version of the VIIRS-SDR-F-LUT/data/sn-002/VIIRS-SDR-F-LUT_sn002.bin • adl v/control -glrdrtocal -od /data/mytestrun \rdr=/rdr/RNSCA-RVIRS_npp_d20111128_t0830507_…h5 \sdrf=/data/sn-002/VIIRS-SDR-F-LUT_sn002.bin adl command VIIRS SDR Controller algorithm RDR to Calibration GuideList Staging directory Nominal Science RDR file adl v/control -glrdrtocal -od /data/mytestrun rdr=/rdr/RNSCA-RVIRS_npp_d20111128_t0830507_…h5 sdrf=/data/sn-002/VIIRS-SDR-F-LUT_sn002.bin F LUT override

3. AeroADL — quick recap and update • Rebranded Aerospace ADL Front End • Focus on the analyst loop • Moving toward wrapper around unmodified ADL • Simplified installation and configuration • Multiple users share one installation • Updated to ADL 4.0 with patches and Mx6.3 tarball • Partial OMPS support confirmed • NP vs TC not anticipated; other sensors may be better supported Developer loop Gather inputs Run algorithm Examine outputs Modify code Analyst loop

4. DMS organization evolved from files to database • Significant DMS reorganization from initial version of ADL Front End • Driven by need to balance having all versions of a LUT available for selection by DMS rules with need to force ADL to use a specific version of a LUT • DMS organized into directories by collection short name • For each item required by a particular algorithm, provide that subdirectory of the DMS as an input directory • To override the DMS with a specific LUT version, stage LUT in execution-specific input directory and eliminate DMS subdirectory from input directory list • Desire to form community wide best practices to facilitate a common DMS

5. DMS organization example • $AEROADL_DMS/VIIRS-SDR-COEFF-B-LUT/ • 4eb2d0ce-8d0b0-a8c0d009-f4aa7aa7.asc* • 4eb2d0ce-8d0b0-a8c0d009-f4aa7aa7.VIIRS-SDR-COEFF-B-LUT_npp_20020101010000Z_20020101010000Z_ee00000000000000Z_PS-1-D-NPP-5-PE-_devl_dev_all-_all.bin • 4ed4e84c-350bc-dd828c19-16f85671.asc* • 4ed4e84c-350bc-dd828c19-16f85671.VIIRS-SDR-COEFF-B-LUT_npp_20110822010100Z_20111101010000Z_ee00000000000000Z_PS-1-N-CCR-11-117-JPSS-DPA-001-PE-_noaa_all_all-_all.bin • $AEROADL_DMS/VIIRS-SDR-DELTA-C-LUT/ • 4eb2d0ce-c2d39-a8c0d009-784d81c0.asc* • 4eb2d0ce-c2d39-a8c0d009-784d81c0.VIIRS-SDR-DELTA-C-LUT_npp_20020101010000Z_20020101010000Z_ee00000000000000Z_PS-1-D-NPP-4-PE-_devl_dev_all-_all.bin • 4ed4e84c-7098d-dd828c19-8bdc5952.asc* • 4ed4e84c-7098d-dd828c19-8bdc5952.VIIRS-SDR-DELTA-C-LUT_npp_20110822010100Z_20111101010000Z_ee00000000000000Z_PS-1-N-CCR-11-117-JPSS-DPA-001-PE-_noaa_all_all-_all.bin • 4f70c10b-cf265-a8c0d009-16bf6329.asc* • 4f70c10b-cf265-a8c0d009-16bf6329.VIIRS-SDR-DELTA-C-LUT_npp_20120217000000Z_20120220000000Z_ee00000000000000Z_PS-1-N-CCR-12-330-JPSS-DPA-002-PE-_noaa_all_all-_all

6. AeroADL available on GRAVITE Information Portal • GRAVITE ICF shell: /apps/user/aero/aeroadl/latest • Subversion: https://svn.gravite.gov/aeroadl/trunk • Docs in https://svn.gravite.gov/aeroadl/trunk/doc • PolarWander update service • /apps/user/aero/aeroadl/latest/DMS/USNO-PolarWander-UT1-ANC-Int • https://svn.gravite.gov/aeroadl/trunk/DMS/USNO-PolarWander-UT1-ANC-Int • DMS update service • Tied to 7_DELIVERIES_TO_AERB/SDR/VIIRS/Cal CasaNOSA subversion • Can expand beyond VIIRS SDR Cal with simple configuration change • /apps/user/aero/aeroadl/latest/DMS • https://svn.gravite.gov/aeroadl/trunk/DMS

7. Where do we go from here? • AeroADL approaching limit of what can be done without “working around” ADL • Desire stronger ties between community tool development and ADL development • Maximize use of JPSS budget by looking to community to where Raytheon is constrained • We’re all in this together! • ADL team has provided great support, but the answer frequently includes “no one has tried that before”

8. Feedback from the guinea pig

9. Best practices to drive future ADL development • It is clear that ADL is being used in ways not fully anticipated by original requirements from JPSS • Each ADL power-user likely has developed their own best practices for efficient use • Aerospace looking for a “formal” forum where best practices and community needs can be discussed and where we can have greater input into ADL development plans • E.g., best-practice DMS organization natively supported by ADL DMS GUI and CLI tools • Informal forums (ADL wiki and web-forum) and discussions are a good start, but aren’t meeting the need

10. Need improved support for non-software developers • Building ADL is still a daunting process for non-C++ developers • Not always easy to know what to rebuild • Looking for support and generation of dependencies in Makefiles and use of “make -j” instead of parallel build script • Pages and pages of warnings get in the way of finding real errors and confuse non-C++ programmers • Raytheon reports that this is planned for correction at some point by IDPS team • Aerospace encourages JPSS to make this a priority

11. Still need ADL to move closer to analyst workflow • Growing use of ADL at Aerospace has identified needs not currently on the JPSS directed development path for ADL (through ADL 4.2) • Many needs driven by desire to minimize maintenance effort • AeroADL requires patched version of ADL • UW patches for HDF, … • Add hooks into ADL configuration through files in ADL/cfg directory • Significant effort required to update ADL within AeroADL • AeroADL philosophy • Front-end to a read-only, unmodified “ADL” directory where one installation can be easily used by multiple analysts with different setups and workflows • Several feature requests already submitted through Paul Meade If a configuration option is overwritten by an ADL update, we’ve failed!

12. What would it take to make ADL read-only? • Fine-grain overrides to parameters in ADL/cfg directory • Need ability to override individual items in DDSADL_CFG.xml (and others) on a run by run basis, particularly for log and perf directories • No ADL user should be writing into the ADL directory as part of ADL use • DCONFIG is a good first start, but it still requires reintegration of modifications when core files are updated as part of ADL update • Suggest immediate partial solution to change DMS_PATH and similar variables in unpacker.xml and packer.xml to new environment variable • ${ADL_HOME}/log → ${ADL_LOG_DIR} • Default definition of new variable of ADL_LOG_DIR to “$ADL_HOME/log” in Raytheon distributed envSetup files makes change transparent to users

13. What would it take to make ADL read-only? • Ability to augment configuration with a separate set of configuration files, stored outside of the directory specified by DCONFIG • E.g., Installation specific GuideLists • Adding this capability will not satisfy need for fine-grain override of properties within files in DCONFIG • If additions or modifications to default items in ADL/cfg, must not be required to fold those into the override or additional configuration files

14. What would it take to make ADL read-only? • Ability to override HEAP vs. DMS without modifying _CFG.xml files • Splitting the VIIRS SDR controller into parts that can be run independently requires some HEAP outputs to be persisted to disk • E.g., persisting GEO-VIIRS-OBC-IP when required to be able to run just the Cal with multiple LUT options on the same input data • Override on a GuideList basis • Individual GuideLists can specify whether an output is persisted • E.g., running VIIRS SDR through Cal does not require GEO-VIIRS-OBC-IP to be output

15. Can we get some Mac love? • Visually surveying the community makes me think Mac OS X support for ADL might be popular • 75% of Aerospace ADR 4663 team are Mac users at the desktop • How about a formal community survey of interest? • Aerospace has made an effort to compile ADL 3.1 on Mac OS X 10.6.8, without success • COTS compiled and installed correctly • Successful build prevented by differences in AIX/Linux and Mac linker • AIX/Linux linker supports option to ignore duplicate symbols, Mac linker does not • Minimal effort made to revolve this, but primary culprit appears to be definition of string constants in .h files • Primarily in the IDPS code, not the ADL code

16. A first step toward Mac OS X support • Likely very low risk and low effort fix for multiple definitions of strings — define them as static • static const std::string HDF = "HDF"; • In global context, static restricts scope to compilation unit (the .cpp file) • A better solution that would be to move them to a .cpp file and converted the current definitions in the header files to external declarations • As in ProCmnShortNames.cpp • In .h: extern const std::stringHDF; • In .cpp: const std::string HDF = "HDF"; • Zero idea how much effort is required beyond the strings fix

17. We’re heading in the right direction • ADL greatly improves efficiency of analysis and algorithm development tasks • ADR 4663 (Automated RSB calibration) first real test of completely new capability moving from community science to operational implementation • Working directly in “operational space” has forced a lot of discussion and debates on how to best structure inputs and persist intermediate data from granule to granule. • Would be far less efficient if we were just throwing MATLAB code and whitepapers back and forth over the wall between Aerospace and Raytheon • Next step is to increase community involvement, cross community discussion and participation across the entire ADL development process