What’s the Exit Strategy?

1. What’s the Exit Strategy? Verifying a Design

2. Agenda – Design Verification • Concepts and implications • The specification connection • Verification before test • Test thoroughness • Tiered verification • Concluding the process • Summary

3. Verification Concepts • Verification – Confirmation, through the provision of objective evidence, that the specified requirements have been fulfilled (Q9000-2000) • Confirmation – the act of ratifying or corroborating (AH)

4. Verification Implications • The verification process is not intended to be a “craps shoot” • Verification is primarily intended to ratify correctness • Verification is NOT primarily intended to reveal incorrectness • The mindset is important! • Doubt that a design will work expresses a lack of confidence in the designer and the design process • Waiting until “verification” to guarantee correctness is an excuse for being sloppy • We should expect our designs to work! Verification simply puts the seal of confirmation on the expectation

5. Verification Implications (cont.) • Verification addresses two processes • Design • Primarily a one-time, iterative process in which the developed concept is proven sound • Fundamental correctness can be proven analytically • Inspection confirms logical correctness in all circumstances • Peer reviews are a manual form of inspection • Functional simulation is an automated form of inspection • Inspections must be tailored to design • Analysis verifies correctness in the presence of variability • Reliability (WC, Derating, FMEA, etc.) • Timing over environments and age

6. Verification Implications (cont.) • Verification addresses two processes (cont.) • Instantiation • Particular instance must be sound • Particular correctness must be demonstrated experimentally • Inspection confirms that instructions are followed • Correct components installed • Correct configuration selected • Correct processes performed • Test and demonstration confirms that operation matches expectations • Functionally and parametrically • Predicted by nominal analytical predictions

7. Verification Implications (cont.) • Verification “after the fact” is too late • Analysis and test are NOT equivalent • Design analysis and inspection must proceed in lockstep with design processes • Implementation tests and inspections should simply confirm what is known • Design efforts fail because they occur in a vacuum • Creativity takes primacy over correctness • Functionality comes first with operability being “shoehorned” after the fact • The “monster” simulation syndrome • Conclusion: Verify as you proceed

8. Verification Implications (cont.) • Correctness is a matter of personal integrity for the designer • Verification is not simply externally imposed task • Verification is something that the designer must want to do (a part of his/her ethos) • Verification is confirmation that the designer is “worth his/her salt”

9. The Specification Connection • “Requirements” are confirmed during verification • The designer does not have a free hand with respect to how a design is confirmed • Specification provides the constraint set under which verification is prosecuted • Therefore, verification must be defined prior to start of design • Not simply in a general manner • Specifically • Since specification is a customer and vendor document • Both customer and vendor must be involved in the verification process • “Trust me” is not a reasonable phrase when it comes to verification

10. The Specification Connection (cont.) • Specification contains a complete [ideally] set of requirements for the design • Some requirements are very specific • “All discrete outputs shall have a source impedance of 2 kOhms. • An adequate test is fairly obvious • Some requirements are not very specific • “The unit shall provide 512 kbytes of SRAM” • Note the implied requirement – The SRAM shall function correctly • What is an adequate functional test for this requirement? (walking 1’s, 0’s addressing, etc?) • Some requirements lend themselves to quantitative analysis – “The unit shall provide a .1° C accuracy at end of life” • Some requirements lend themselves to simple inspection – “The unit shall provide 4 analog output channels” • When do we decide adequacy of the method of verification and the individual verification cases? – Before we start designing

11. The Specification Connection (cont.) • Therefore, verification planning must begin with specification creation • Each requirement must be assigned a method or methods • Each requirement must be assigned a test or analysis case • Must answer question – What is an adequate verification? • Must establish answer to question – When are we done? • Each requirement must be verified at one or more levels [FPGA, board, box, sub-system, system] • Customer must concur with decision • Note that modern verification databases make this relatively straightforward

12. The Specification Connection (cont.) • Advantages to this type of verification planning • More verifiable / verified designs • Up-front focus on programmatically difficult verification can be instituted • Earlier analysis • Simplification of overly complex circuit designs • Guidance for lower-level verification efforts can be established • Sub-module vs. module level • Module vs. unit level • Integral self-test provisions can be included in design • Earlier simulation vector definition (FPGAs / logic)

13. The Specification Connection (cont.) • Advantages (cont.) • More robust test sets • Early knowledge regarding end-item function communicated • Capability of test set matched to need • Precision (timing, voltage, current, etc.) • Speed • Test level (component, unit, system) • Test flow design considerations included in test set design • Knowing verification requirements early makes the entire development process more efficient

14. The Specification Connection (cont.) • Defining requirements and test cases at the same time as the specification clarifies and simplifies the FPGA verification process • Allows early start to simulation vector design • Creates early knowledge of additional functional models (SRAM, peripherals, etc.) needed • Clarifies decision regarding what can be functionally simulated and what must be simulated with back-annotation • Defines levels at which simulations must be run

15. Verification Before Test • It is a relatively bad thing to discover a design flaw during test (better than nothing) • Late in the development cycle • Correction is more complicated / expensive • Personal stress is higher • Yet … a surprising lack of verification occurs early • Types of pre-test verification • Inspection - conformity evaluation by observation and judgment accompanied, as appropriate by measurement or testing (ISO) • Personal self-check • Peer Review • Basic functional verification (Signal audits, functional simulation, etc.)

16. Verification Before Test (cont.) • Types of pre-test verification (cont.) • Analysis – Conformity evaluation of the predictions produced by realistic models of the system • Worst case analysis (voltage, current, frequency, time) using models that incorporate real world degradation of function • Derating analysis using models that reduce stress • Other • Note that pre-test verification is fundamentally conceptual • Not tool dependent • Not design dependent • Can be accomplished many ways

17. Verification Before Test (cont.) • Two approaches to pre-test verification • Verify full design at once (one big peer review, code walkthrough, analysis, simulation) • Benefits • Complete design reduces need to develop assumptions • When proved correct, does not need to be repeated • When design is solid, less work • Reduce final documentation for verification • Drawbacks • Allows small design flaws to propagate for a long time • Complexity reduces visibility (verification more prone to mistakes and oversights) • When design is flawed, it increases work • Leads to corrections that are global (hard to test effectively and predict side effects) • May take longer to execute than an aggregate of smaller verification activities

18. Verification Before Test (cont.) • Two approaches to pre-test verification (cont.) • Verify incremental designs • Benefits • Supports development of “known good” sub-designs • Meshes with a partitioned design approach • Facilitates visibility (fewer mistakes, clearer goals) • Allows small flaws to be fixed quickly (minimizes downstream impact) • Drawbacks • Is more work in the ideal case (no/few flaws) • Increases documentation if formality is vital • Either approach can be made to work, but one or the other must be chosen and implemented

19. Test Thoroughness • Principles for test thoroughness • Every “shall” in the specification that meets the following criteria must be tested • Items that may be affected by the instantiation process (subject to fabrication error) • Items for which the only extant verification is model based • Items for which the testing does not pose unacceptable risk (radiation, overstress, etc.) • Every instance of a “shall” must be tested • 18 interfaces require 18 specific tests • Requirements that have an exclusive character • Must be tested for correct operation • Should be tested at some level for abnormal operation • Example: If you write a 5Ah to something to set it on, you should verify that writing other values does not set it on • Requirements must be tested over a representative range of conditions

20. Test Thoroughness (cont.) • A thorough test is complicated and time consuming • Requires some level of automation to be comprehensive • Requires some level of compromise to be practical • Requires team agreement to be acceptable

21. Test Thoroughness (cont.) • Ensuring thorough, yet practical, tests • Define which tests require manual interaction and which can be automated • Output impedance or analog fine precision may need to be manual • Signal level or analog coarse precision may be automatable • Define which tests must be performed over environmental conditions and which can be performed at room temperature only • Make decision based on character of measurement • Do not decide based purely on practicality • Define level of complexity for all measurement sets • Example: 24 analog inputs (3 eight channel muxes) • Test full range of values for 24, or limited range for 7 of 8 mux inputs and full range for remaining • Define need for repetition of measurement at higher level of integration (tiered verification) • Example: voltage level / signal quality / clock jitter at board level • Repeat at box level?

22. Test Thoroughness (cont.) • Test thoroughness issues affect everything • Customer relationship • Test set design (board, box, system) • Schedule / cost • An adequate test program is not trivial • Cannot wait until system level • Must incorporate lowest level of design • Must be a constant background activity during design process • All programs must balance perfect and good enough • A good test program • Will ensure that the specification is met • Will verify everything that can be affected by fabrication • Will build on the analysis and inspection effort • Will maximize automation without sacrificing test accuracy

23. Tiered Verification • Tiered verification is the process of matching the correct type of verification to the appropriate level of integration • Tiered verification incorporates all types of verification (before and during testing) • Tiered verification applied to the test regime attempts to have thoroughness with practicality • Test a requirement at the lowest conceivable level • Determine which tests must be repeated at higher levels by • Establishing the purpose of a test at a particular level • Determining whether the next level has the potential to change previous measured quantities • Determining what test set capabilities are • Tiered verification cannot be retrofitted into the process • Must be planned up front • Must be implemented throughout the development process

24. Tiered Verification - Example • Need: A set of 16 high-level pulsed discretes to control latching relays used to change the spacecraft power configuration • Basic Requirements • Quantity: 16 • Level: +30V +/- 2 V • Load: 2 kOhm, 4 mH • Duration of pulse: 50 ms • Rise / Fall time max: 1 ms • Command capability: Unique bit pattern, only one at a time, arm first • S/W requirements: Various (beyond scope of example)

25. Tiered Verification – Example (cont.) • Tier 1 – Basic Verification (functional simulation) • Verify each channel fires for a specific bit pattern / address • Verify cannot be fired without an arm command • Verify that other complementary patterns don’t accidentally fire the pulse • Analyze drive capability of FPGA and input / output requirement / capability of translation chip • Evaluate glitch hardness of logic selected

26. Tiered Verification – Example (cont.) • Tier 2 – Board level • Test to confirm signal level between FPGA and driver chips is compatible • Test rise, fall, duration, level with dummy load • Repeat test for all 16 outputs • Replace dummy load with relay, verify that the relay switches for all 16 outputs • Repeat over temperature if necessary

27. Tiered Verification – Example (cont.) • Tier 3 – Box level • Attach GSE (verifies with relays) • Functionally pulse all outputs • Verify relay switches • Verify gross drive duration • Tier 4 – Box and operational software level • Verify functionality commanded through software • GSE verifies correct relay switch • Tier 5 – System • Verify gross functionality as needed for system operation

28. Tiered Verification - Notes • Most complex and detailed functionality is verified at lower levels • Performance • Error cases • Predictive of future operation • Higher levels focus on functionality / operation • Lower level verification requires more manual touch • Higher level verification is more automated • STE and test procedure developed as appropriate • Purpose is maximum test completeness • As well as “bang for the buck”

29. Concluding the Process • (“… through the provision of objective evidence…”) • The importance of traceability • Something is only verified (formally) when it is documented • The tie between verification item and specification must be made (verification matrix or database) • The importance of planning • Establish the links between requirement, method, level, test case early • Use the established structure to develop verification items • The importance of buy in • All responsible must complete verification and report results • Systems engineers must track and close • People must keep up with the process • Only personal commitment from all involved will ensure that the process is successful

30. Summary • Verification is to prove success, not avoid failure • Verification planning is an integral part of the project lifecycle • Must be initiated with specification (including customer buy-in) • Must be included in design effort • Must be used to develop documentation and appropriate test hardware • Verification is more than final test • Incorporates analysis and inspection throughout design process • Begins at lowest possible level • Develops proof of compliance at all levels of operation • A thorough test is complicated • Requires extensive work • Must trade perfect for practical (a tiered test approach can help) • Verification processes must be tracked and documented • The whole team must buy in