LADEE Flight Software: Models to Flight

1. LADEE Flight Software: Models to Flight Nathan Benz Peter Berg Howard Cannon Pat Castle Scott Christa Eleanor Crane Doug Forman Greg Limes Mike Logan MasoudMansouri-Samani Craig Pires Fritz Renema Larry Shackelford (GSFC) DaniloViazzo Karen Gundy-Burlet, Ph.D. LADEE FSW Lead NASA-Ames Research Center Moffett Field, CA

2. Mission Overview • Lunar Atmosphere and Dust Environment Explorer (LADEE) is a NASA mission that will orbit the Moon and its main objective is to characterize the atmosphere and lunar dust environment. • Low cost, minimal complexity and rapidly prototyped “common bus” design. • Model-Based Software Development • Specific objectives are: • Determine the global density, composition, and time variability of the lunar atmosphere; • Confirm the Apollo astronaut sightings of dust jumps and diffuse emission • Laser Communications Demonstration: 622 Mbs Record download rate from the Moon! Gene Cernan’s drawings of the lunar sunrise Clementine spacecraft image of moon dust corona

3. Review of LADEE Development Process & Software Architecture Some Lessons Learned Model Based Development NPRs/CMMI Model Based Development “Just In Time” organizations Interface Control Documentation Emergent Behavior Current Status Outline

4. Development Approach Model Based Development Paradigm (prototyped process using a “Hover Test Vehicle”) 5 Incremental Software Builds, 2 Major Releases, 3 final sub-releases 5.1: Defects found by I&T and 3DOF 5.2: Defects found by Mission Operations Testing 5.3: Final RTS set for Golden Load Leverage Heritage Software GOTS: GSFC OSAL, cFE, cFS, ITOS MOTS: Broad Reach Drivers COTS: , VxWorks, Mathworks Matlab/Simulink & associated toolboxes Flight Software Overview • Scope • Onboard Flight Software (Class B) • Support Software and Simulators (Class C) • Integration of FSW with avionics • Guiding Documents • NPR 7150.2 Software Engineering Requirements • CMMI Level 2 • NASA-STD-8739.8 NASA Software Assurance Standard

5. Develop Models of FSW, Vehicle, and Environment Automatically generate High-Level Control Software Integrate with hand-written and heritage software. Iterate while increasing fidelity of tests – Workstation Sim (WSIM), Processor-In-The-Loop (PIL), Hardware-in-the-Loop (HIL) Automated self-documenting tests providing traceability to requirements Model Based Development Iterate Early and Often Requirements Verification Design/Algorithm Development Flight Software Modeling Heritage Models Analysis Vehicle & Environment Modeling Workstation Simulations (eg. Simulink) Unit Tests Automated Reporting Hand Developed Apps Code Generation Heritage Software Integrated Tests Processor-in-the-Loop Hardware-in-the-Loop

6. KEY FSW Internal Simulink Task cFS Task Hand Written Task FSW External FSW Architecture OFSW Battery Charge System Cmd & Mode Processor Actuator Manager State Estimator Safe Mode Controller Attitude Control System Thermal Control System Power Control System Simulink Interface Layer Scheduler Stored Commands Memory Manager Memory Dwell Limit Checker House- keeping Memory Scrub Hardware I/O Health & Safety File Manager CCSDS File Delivery Checksum Data Storage Telemetry Output Command Ingest System Support and O/S Services Telemetry Gnd Cmds Hdwr Cmds Sensor Data • GSFC OSAL, cFE, cFS, ITOS (GOTS) • Broad Reach Drivers (MOTS) • Simulink/Matlab, VxWorks (COTS)

7. Model-Based Development with significant software re-use: - Hype or Help? Advantages: High level control software in the “Native Language” for GN&C developers Allowed development to be highly parallelized Modular design for all applications, Simulink or hand-developed Software developers could prototype autocode/integration process independent of Simulink module development Early Requirements definition led to ability to formalize test infrastructure early in development cycle. Easy to communicate design/algorithms/data flow with stakeholders and other subsystems. Simulink Report generator is an extremely powerful tool for driving verification system. Generated code was generally clean and efficient. Static analysis tool found some defects, which were eliminated through changes in modeling practices. The Basic Question

8. Disadvantages: Have to be very careful when patching Simulink applications. Must upload associated parameter table. Cannot change interfaces/buses in flight. Bus changes during development induced significant rework of test harnesses. Personal Experience from a Simulink Newbee: LADEE Propulsion Model Surprisingly fast to model. Amenable to modular development. Think carefully about flow and propagation of signals. Painful to update models/test harnesses. Absolutely needed advice/example models from Simulink expert because of the complexity, range of modeling choices/parameters and their effects on the autocode. Lesson: Model-Based Development is effective when used in a highly disciplined manner. Sloppy development practices lead to bad outcomes, no matter what the language. The Basic Question, cont.

9. My Perspective: Daunting set of rules/regulations/practices that have arisen from lessons learned and bad outcomes from other projects. They are an effective roadmap/checklist that help guide on how to plan and document our solutions to prevent those problems. Strategy: Comply as simply and effectively as possible. Lesson Learned: Safety and Mission Assurance/Independent Reviewers can be a real ally Providing templates and advice about effective practices. Extra experienced eyes during the planning phase leads to fewer process issues and software defects later. Early understanding by SMA/IR of project practices leads to smoother and more effective reviews. When you get to the inevitable time crunch at a critical milestone, it is essential to have practiced, documented processes. Multiple examples in LADEE of schedule being brought in by well-practiced test program NASA Procedural Regulations & Capability Maturity Model Integration

10. Our “final” 2 build cycles (4 & 5) were major deliveries to the spacecraft. We smugly delivered them “Just In Time” to meet I&T schedule needs. I&T picked up both builds and tested extensively with them I&T and the 3DOF testbed identified ICD related defects MOS “Just In Time” schedule not tied to these releases. From a FSW perspective, this delayed discovery of: More defects in EDICD/Fault management logic. Hidden Requirements on the spacecraft simulator model. Our concept of “Test Like You Fly” was not the way MOS actually flew. More importantly, the schedule was not tied to “Golden Load” deadline Milestones for development and certification of MOS RTSs were scheduled for after the need date for inclusion in the “golden load”. Schedule Catch 22: They needed the “golden load” to certify the RTSs for inclusion in the golden load. This put FSW back on the critical path for the spacecraft launch date! Lessons Learned: “Just in Time” philosophy leaves no room for missed schedule dependencies We missed valuable test time of the end-to-end operation of the spacecraft leading to a delay in identifying defects. Some were simply too late too be incorporated in the “golden load” “Just In Time” Organizations

11. Interface Control Documentation • The primary cause of defect escape into Build 5.x was misunderstood or ambiguous ICDs. • Examples: • Fuel Tanks (A,B) = FSW (1,2), Ox Tanks (A,B) = FSW (2,1) • Star tracker: Interpreted 8Hz operation to mean heads alternately operating at 4 Hz. Instead, they operated synchronously. • Reaction Wheels: “Current Limit Flag” was a warning flag, not an error flag. • Mitigation: • EDICD in computer readable format and read into database, so could quickly reconfigure • Ability to upload parameter tables & software patches • Best prevention for ICD problems was our “Travelling Roadshow” • EDU in a mobile chassis loaded with the FSW • Flown to payload sites and integrated with instruments. • Lessons Learned: • Early integration with payloads/instruments essential to clarifying ICDs. • Significant rework possible when one “saves money” by not purchasing instrument engineering development units

12. Emergent Behavior • Prior to one of the orbital phasing maneuvers, the spacecraft went into "safemode” • Fault Management took action because it detected an excessive spacecraft rate from the state estimation system. •  It was determined that the star tracker caused a jump in the state estimator signal. Two primary factors: • The as-built alignment of the star tracker was slightly different than as-designed. • Fixed by a table upload. Reduced rate errors but did not eliminate further safemode transitions. • The star tracker was exhibiting delays in providing the spacecraft orientation and position when one of the cameras pointed at a "Big Bright Object" (BBO) such as the Sun, Moon or Earth. The behavior continued to worsen the closer we got to the moon. • Reworked state estimator model, reran scenarios, re-verified EST Unit Tests, GN&C L4 Requirements & uploaded patch to the spacecraft. • Lesson Learned: • The defects we had to correct under schedule pressure and duress in I&T and ORTs were excellent repeated practice for polishing our maintenance processes.

13. Current Status • LADEE is currently in orbit around the moon performing science operations. • Successful Laser Communications demonstration: 622Mbs downlink rate. Very useful to be able to download a SDRAM partition in less than 2 minutes. • Over 2000 hours of flight. • Lowest altitude above the moon’s surface so far is 12.1 Km • Expected to operate until mid-April when eclipses will occur. • LADEE Flight Software Status • Team recertified for CMMI level 2 in May 2013 • Some minor defects found that we’ve worked around. • Table uploads performed (ATS, RTS, FM, Thermal updates & defect reduction) • 2 software patches to account for star tracker behavior • 1 reboot (still under investigation) • LADEE FSW: Enabling a successful mission!