Pre-Software Assurance Symposium Facility Initiatives Briefing

1. Pre-Software Assurance Symposium Facility Initiatives Briefing Independent Verification & Validation of Programmable Logic Devices 8 July 2005 NASA IV&V Facility James Cercone, Ph.D., P.E.,WVU-Tech Michael Beims, SAIC July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

2. Pre-Software Assurance Symposium Facility Initiatives Briefing IV&V Facility • Outline • Review IV&V of PLD Research Project Objectives and Framework • Review of detailed technical findings and VHDL defect taxonomy • Provide overview of Work Instruction development July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

3. IV&V PLD Status IV&V Facility NASA-STD-8739.8 Software V&V is concerned with ensuring that software being developed or maintained satisfies functional and other requirements and that each phase of the development process yields the right products. ….. IV&V is performed by an organization that is technically, managerially, and financially independent of the development organization. For NASA, IV&V is performed and/or managed by the NASA IV&V Facility. …“Software includes programs and operational data contained in hardware (e.g. firmware, programmable logic, and programmable gate arrays).” July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

4. IV&V PLD Status IV&V Facility • IEEE STD 1012-1998 • IEEE Standard for Software Verification and Validation, provides supporting information regarding the integration of IV&V into every step of the Software Development Life Cycle. The IEEE standard, like the NASA Standard, also cites firmware and microcode in its definition of software: “This standard applies to software being developed, maintained, and reused …. The term Software also includes firmware, microcode, and documentation.” July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

5. IV&V PLD Status IV&V Facility • IEEE Std 1076™-2002 • Abstract: • VHSIC Hardware Description Language (VHDL) is defined. VHDL is a formal notation intended for use in all phases of the creation of electronic systems. Because it is both machine readable and human readable, it supports the development, verification, synthesis, and testing of hardware designs; the communication of hardware design data; and the maintenance, modification, and procurement of hardware. Its primary audiences are the implementers of tools supporting the language and the advanced users of the language. • Keywords: • computer languages, electronic systems, hardware, hardware design, VHDL July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

6. NESC Project Activities performed by IV&V • Primary Goals: • Develop an IV&V strategy for PLD’s • Provide field proven PLD Work Instruction (WI) to the IV&V practitioner July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

7. Activities Thus Far Better understanding of PLD’s at WVU Tech and SAIC Primarily via literature searches and attendance at workshops Presentation to IV&V CAWG WVU Tech obtained and learned IDE’s (Integrated development Environment) for both Actel and Xilinx PLD’s. WVU Tech has also obtained and learned Active HDL Limited analysis and simulation of NASA project data IV&V has mapped PLD’s into a better framework for IV&V WI development Identifying a taxonomy of defects in VHDL Domain Via Literature Search By comparing VHDL releases for the same chip Evaluating SW code defects that can be “mapped” to PLD VHDL defects Had initial discussions with JWST as candidate for VHDL Pilot Project Activities/Results thus far (9-1-05 through 8-8-05) July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

8. Results Thus Far Scoped PLD IV&V Framework to understand 05 accomplishments and identify potential future year activity Based on increased understanding, and Realization that existing IV&V Code Analysis WI is insufficient for PLD analysis Defect taxonomy, in process of refinement, for presentation at MAPLD Activities/Results thus far (9-1-05 through 8-8-05) July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

9. Development Environments Idealized Software/PLD Development Requirements Design Code Test • SRS is a CM’d • document • Rigorous flowdown • is common • SDD is a CM’d • document • Manual or tool • generated • Different types of • code (C, C++, etc) • Mature tools avail to • aid in development, • verification • Performed on • implemented code • Unit, Subsystem, • System testing follows • req’ts flowdown • Rigorous process Common PLD Development Requirements Design/code and simulate at functional level Design/code and simulate After chip layout Testing after PLD is programmed • Part of subsystem • Hardware artifact • (e.g. EQ spec, • product functional • spec) • Performed on • implemented PLD • Unit, Subsystem, • System testing follows • req’ts flowdown • It is at this stage that Idealized/Common development • processes diverge • Design process • IDE • Target • PLD’s also have timing concerns that are rare in software development, such as • Synthesized versus native components • Race Conditions • Adequately buffering data July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

10. Common PLD Development Requirements Design/code and simulate at functional level Design/code and simulate After chip layout Testing after PLD is programmed Similar to FSW but artifact expectations need to be articulated by IV&V • 1) Ensure syntax is correct • 2) Identify typical errors • 3) Develop/deploy WI: • Programming Standards and • Defect ID • VHDL • Verilog • Schematics Identify any known issues with IDE and ensure potential errors not present in developed product Verify tests performed by developer (using simulation to generate test cases) Verification Tasks Traceability of Requirements Identify key timing areas and independently simulate Re-simulate key timing functions tbd tbd, independent testing? Validation Tasks Current Year (2005) Activities against PLD IV&V Framework • Our current WI activity focuses on verification of VHDL Design • Develop Work Instruction • Flesh out Work Instruction with Pilot Project • Deploy Work Instruction at Facility July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

11. Common PLD Development Requirements Design/code and simulate at functional level Design/code and simulate After chip layout Testing after PLD is programmed Similar to FSW but artifact expectations need to be articulated by IV&V • 1) Ensure syntax is correct • 2) Identify typical errors • 3) Develop/deploy WI: • Programming Standards and • Defect ID • VHDL • Verilog • Schematics Identify any known issues with IDE and ensure potential errors not present in developed product Verify tests performed by developer (using simulation to generate test cases) Verification Tasks Traceability of Requirements Identify key timing areas and independently simulate Re-simulate key timing functions tbd tbd, independent testing? Validation Tasks Ideas for Future Year Projects b b a c d • Future Projects • Develop WI for verification of Verilog or Schematic designs (item “a”) • Provide WI for Requirements Analysis or Test Analysis of PLD’s (item “b”) • This is probably a straightforward extrapolation of current WIs for FSW • Have the breadth of knowledge on development tools and all target PLD’s (item “c”) • Provide insights on timing/validation aspects of PLD implementation (item “d”) July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

12. Complexity is a Challenge for all Design Representations IV&V Facility ? Functional Trace / Performance Test Design Trace / Functional Test July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

13. Pre-Software Assurance Symposium Facility Initiatives Briefing IV&V Facility Review of detailed technical findings and VHDL defect taxonomy July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

14. Entity  Are Signals defined in the port list as out type signals given values?  Are Signals that are defined in the port list as inout type signals used for both reading and writing?  Are Signals defined in the port list as in type signals used in the architecture? Sample FindingsPotential VHDL“Hot Spots” visible in semantics July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

15. Process  Is there a series of sequential statements followed by a branching structure?  Is there a branching structure followed by a series of sequential statements?  Is each process sensitive list made up of the signals from the Entity’s port list? Sample FindingsPotential VHDL“Hot Spots” visible in semantics July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

16. If Structures  Having elsif and no else statement  Having neither an elsif or else statement  Is there unreachable code inside an else statement?  When using a compound if statement, are all possible conditions covered in subsequent elsif and else statements.  How deep is the nesting of if structure?  Testing Signals in the condition that are not part of the process’s sensitive list Sample FindingsPotential VHDL“Hot Spots” visible in semantics July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

17. Signal Assignment Is the same set of Signals assigned values in each of the if-elsif-else sections?  Is the same set Signals assigned values in each of the case structures when and when others => clauses?  Are all Signals in a component’s port list mapped values during a Component’s instantiation? Sample FindingsPotential VHDL“Hot Spots” visible in semantics July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

18. Sample Findings Static Metrics Analysis of Public Code IV&V Facility July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

19. Sample FindingsStatic Metrics Analysis of Public Code IV&V Facility Note ! July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

20. Sample FindingsStatic Metrics Analysis of Public Code IV&V Facility … July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

21. Sample FindingsStatic Metrics Analysis of Public Code IV&V Facility • Observations related to Code Changes • (absolutes for the public code analyzed) • A code change occurred if: • There were 12 or less files • Average number of lines per file was greater than 400 • There were less than 3 “package” statements • There were less than 11 “entity” statements • There were less than 20 “component” statements • There were less than 13 “architecture” statements • There were less than 20 “clk’event” statements • There were any “while” statements • There were any “wait” statements • There were any “after” statements July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

22. Sample FindingsStatic Metrics Analysis of Public Code IV&V Facility • Observations related to Code Changes • (normalized for the public code analyzed) • A code change occurred if: • 5% or more of the total lines were “signal” statements • 5% or less of the total lines were “in” statements • 5% or less of the total lines were “out” statements • 3½% or more of the total lines were “if” statements • ¼% or more of the total lines were “case” statements July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

23. Sample FindingsStatic Metrics Analysis of Public Code IV&V Facility July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

24. Sample FindingsStatic Metrics Analysis of Public Code IV&V Facility July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

25. Sample FindingsStatic Metrics Analysis of Public Code IV&V Facility July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

26. Sample FindingsSample taxonomy of semantic defects visible in VHDL IV&V Facility July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

27. Sample FindingsSample taxonomy of semantic defects visible in VHDL IV&V Facility July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

28. Sample FindingsSample taxonomy of semantic defects visible in VHDL IV&V Facility Note: NASA experts’ recommended practice prevents an ‘out of sync’ by insuring that resets are never very close to a clock edge. This design is seen in NASA flight software as a D-FF with reset. July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

29. Sample FindingsSample taxonomy of semantic defects visible in VHDL IV&V Facility Synthesis vs Simulation difference visible in VHDL Multiplexer with missing sensitivity signal (signal “b”) process(a,sel) if sel = '1' then out <=1; else out <= b; end if; end process www.synplicity.com/literature/pdf/HDLDesignMethods.pdf July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

30. Fault Detection Matrix High Confidence – IV&V success Less Confidence - Mission Risk July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

31. Fault Detection Matrix Examples True/PositivePotential Hot Spot identified/ Design defect exists “Pre-processor directives and usage are often not controlled by coding standards. 1) #ifdef statements can be left active or inactive and permit non-flight code (e.g. test code) to be compiled. Instances of such errors have been found in Mars program code. 2) #define statements can be left in the code from testing leaving test values or conditions active in the flight code. Instances of such errors have been found in Mars program code.” July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

32. Fault Detection Matrix Examples False/Negative No Hot Spot identified/ Defect exists If neither cond1 nor cond2 is true, then X will retain its value ...basically, X is stored in a latch In general, latches are not usually recommended in synchronous designs process (A,B) begin if (cond1) X <= A + B; elseif (cond2) X <= X – B; end if; end process; July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

33. Taxonomy of Common Visible Defects July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

34. VHDL Code Severity Chart July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

35. Pre-Software Assurance Symposium Facility Initiatives Briefing IV&V Facility Overview of Work Instruction Development July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

36. Work Instruction Development IV&V Facility Background Considerations VHDL specific considerations Taxonomy of potential “Hot Spots” Clock and Reset Lines Sensitivity Lists Features not consistently supported between IDE’s Non-implemented features (i.e some attributes) July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

37. VHDL Code Severity Chart July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

38. VHDL Code Severity Chart Examples Example of High Functional Criticality - The FPGA with this defect is the only functional capability for the Satellite to deploy the solar panels. If the FPGA does not perform this function, the satellite will run out of power, causing loss of the mission. July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

39. VHDL Code Severity Chart Examples Example of Low Functional Criticality - The FPGA controls one side of a dual redundant path to the telemetry transponder. If the FPGA fails, then the telemetry is routed via CPU control, causing a momentary delay in telemetry, if all telemetry is buffered through on-board storage. (Note: since this is a design (software) defect, if there were two identical FPGA's controlling this functionality, instead of an FPGA and the CPU, then the redundant FPGA can be expected to fail in the same manner and there is no functional redundancy, making this a High Functional Criticality defect.) July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

40. Example of Vendor Specific Degree of Compliance “major differences between XVHDL and Express is IEEE VHDL-93 compliance. XVHDL is a fully IEEE VHDL-93 compliant tool. Express supports many of the most commonly used VHDL-93 synthesis constructs, but is not yet fully compliant; it remains officially compliant with the IEEE VHDL-87 standard.” http://www.xilinx.com/xlnx/xil_ans_display.jsp?iLanguageID=1&iCountryID=1&getPagePath=5144 (7/21/2005) Compliance Issues July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

41. Example of Vendor Specific Compliance (two examples) Signal Declaration Supported ("register" or "bus" type signals are not supported) Attribute Only supported for some predefined attributes: HIGH, LOW, LEFT, RIGHT, RANGE, REVERSE_RANGE, LENGTH, POS, ASCENDING, EVENT, LAST_VALUE Otherwise, ignored. http://www.xilinx.com (7/21/2005) Compliance Issues July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

42. Draft of VHDL programming standards geared toward defect identification Defects commonly detected by compilers are not included Includes syntax, timing margins, clock boundaries This draft is in process of update, peer review Align defects with known coding defects Test draft product against actual VHDL text Developer places multiple revs of VHDL on website The results will be presented at MAPLD in September, 05 Defect Taxonomy (VHDL Verification at Functional Design), and Pilot Deployment Work Instruction in 05 addressed VHDL verification. Future tasks needed to address: Verilog and Schematic verification plus validation. Note: GLAST LAT used Actel VHDL for design and this served as basis for IR&D project. MRO project used Xilinx Verilog for design. July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

43. Materials needed to start the verification process are: Design Documentation to analyze performance Actual VHDL Code Code Pedigree (Reused modules, designers, level of experience…) Development and Analysis Tools State diagrams. Clock Trees NASA and IEEE Standards Preliminary Considerations for Defect Detection in VHDL Based Designs July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

44. V&V Process and Procedure at the Code Level Static Metric Analysis Code Coverage (particularly for behavioral level designs) Verification of Clock and Reset Tree’s (if provided) Check for compliance to NASA Standards Check for device resource usage (synthesized vs. board components such as MAC’s, SR, and DFF’s) Check of IDE specific restrictions Check VHDL specific “Hot Spots” Preliminary Considerations for Defect Detection in VHDL Based Designs July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

45. Pre-Software Assurance Symposium Facility Initiatives Briefing IV&V Facility • Conclusion • Review IV&V of PLD Research Project Objectives and Framework • Review of detailed technical findings and VHDL defect taxonomy • Provide overview of Work Instruction development July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

46. Background Slides

47. Independent Technical: IV&V prioritizes its own efforts Managerial: Independent reporting route to NASA Headquarters Financial: Budget is allocated by NASA to the IV&V Facility such that IV&V effectiveness is not compromised Verification (Are we building the product right?) The process of determining whether or not the products of a given phase of the software development lifecycle fulfill the requirements established during the previous phase The product is internally complete, consistent and correct will support the next phase Validation (Are we building the right product?) The process of evaluating software throughout its development process to ensure compliance with software requirements. This process ensures: Expected behavior when subjected to anticipated events No unexpected behavior when subjected to unanticipated events System performs to the customer's expectations under all operational conditions What is IV&V ? July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

48. The Project V&V goes end-to-end Needs sufficient depth to help ensure that they have build the product right and built the right product IV&V needs to be the second line of defense Select the most critical functionality and then IV&V to the appropriate depth -- not exceeding the IV&V point of diminishing returns (maintaining reasonability) The cut-off point should be where we have found the critical defects and also gained enough confidence in the software to support mission assurance requirements and launch recommendations Project Teams should compare our criticality rankings to their knowledge of the development as an independent source and explore differences Project Teams should look at activity just below the IV&V line to ensure adequate V&V resources Maximizing Project V&V and IV&V July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

49. Framework to perform IV&V was updated to Take into consideration that development of PLD’s is different than development of software, Address verification and validation tasks explicitly PLD’s are developed in an environment that combines design, code and simulation simultaneously Timing aspects much more critical in PLD’s Major differences needed to be addressed: PLD development is part of subsystem development – comparable artifacts not consistently generated PLD design can occur in many forms Schematic (representation similar to chip/board design) VHDL (representation similar to Ada) Verilog (representation similar to C/C++) Target system matters Syntax different even when same language used Development environment matters From a syntax standpoint From a capability standpoint (e.g. software motivated or hardware motivated) Which revision of IDE (multiple releases for hw motivated IDE’s) PLD Updated Framework July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

50. The above updated IV&V framework has more detail Allows us to clearly understand what we have accomplished, and what lies ahead Strategy is to perform accomplished tasks well For each task performed, Develop Work Instruction (WI), flesh out internally Test on pilot project Deploy updated WI The fast pace of PLD product evolution requires additional considerations Important to have cognizance of market trends Update WI appropriately with trends that will be implemented in spacecraft in the near term Common PLD Development Requirements Design/code and simulate at functional level Design/code and simulate After chip layout Testing after PLD is programmed Similar to FSW but artifact expectations need to be articulated by IV&V • 1) Ensure syntax is correct • 2) Identify typical errors • 3) Develop/deploy WI: • Programming Standards and • Defect ID • VHDL • Verilog • Schematics Identify any known issues with IDE and ensure potential errors not present in developed product Verify tests performed by developer (using simulation to generate test cases) Verification Tasks Traceability of Requirements Identify key timing areas and independently simulate Re-simulate key timing functions tbd tbd, independent testing? Validation Tasks Updated IV&V Framework for PLD Development July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone