HW/SW- Codesign

1. Verification and Debugging. HW versus SW Ondrej Cevan HW/SW- Codesign

2. Outline • what is testing • embedding testing in design process of HW & SW • testing tools in HW, SW • Simulator, Emulator, Monitor

3. Verification and Debugging What is Testing? • “to place a product or piece of equipment under everyday and/or extreme conditions and examine it for its durability, etc.” (wiktionary.org) • with testing we want to achieve: • the object meets its specification • estimation for reliability • save cost (maintenance) • improve quality • debugging: the activity of locating, analyzing and removing a bug (an error state) from the system.

4. Verification and Debugging Design Flow and Testing • V-model in SW design • assigns testing activities to development activities • (figure from Yilin's slide)

5. Verification and Debugging Design Flow and Testing • model in HW-design • extension of every test process in V-model • Simulations are checks for the designer to make sure correct behavior is being implemented • we need to simulate the HW-clock

6. Verification and Debugging HW: System Level Algorithmic Level Register Transfer Level Logic Level Circuit Level Chip Abstraction Levels • SW: • System Specification Language • High-level Language • Assembler • Computer Instructions

7. Verification and Debugging HW-model • Behavior Simulation • verifies the correctness of the HDL description • not every code is synthesis-able • Functional Simulation • verification after synthesis, design with internal components • Pre Layout Simulation • after technology mapping, design with components of the target technology • Post Layout Simulation • to verify the results of place and route process • timing and functionality of the design checked

8. Verification and Debugging Verification Methods (HW/SW) • Review (of documents & code) • Formal Verification (model checking, theorem proving) • Dynamic Verification: • functional testing , structural testing, performance, reliability, security testing • simulations, black-box, white-box, code coverage, path coverage, throughput, resource usage, response time, stress/load testing, fault injection, simulation of attack ...

9. Verification and Debugging Verification Methods/Tools HW • Hardware Implemented Fault Injection (HIFI) • Hardware in the Loop • Remote debugging tools (in-circuit emulator (ICE) = on-circuit debugger (OCD)= background debug module (BDM)) • Logic Analyzer • Oscilloscope (manual I/O testing) • Simulation-based fault injection(inject faults in a simulation model- on electrical level, gate level, register level, VHDL level, system level) • FPGA Prototyping

10. Verification and Debugging SW/HW-Simulator • simulator resembles real-world environment • Hardware in the Loop, Software in the Loop, Model in the Loop • Testbench (generates stimuli- input vector, emulates test cases) • test case created from the specification document

11. Verification and Debugging SW/HW-Debugger • an analyze tool for developers to test and debug other programs • program or fraction of code is processed line by line • one can stop the program, read the values of variables and set new values to the variables • certain test environments, test situations can be enforced • also good for failure reproduction and fault analysis • examples: GNU debugger (gdb), dbx, vhdldbx, Eclipse

12. Verification and Debugging SW/HW- Emulator • ability of a program or device to imitate other program or device • in theoretical sense, Church-Turing thesis implies that any operating environment can be emulated within any other (in practice it can be quite difficult to achieve it) • enables to test code for specific HW on a common PC • we can emulate HW faults in the design phase (e.g. with fault injection) • also SW faults can be emulated (Ballista tool) • other ex.: wine, Commodore 64 emulator, ...

13. Verification and Debugging HW-Emulator • Simulation: executes RTL code serially, good debug environment, user interface, easy, flexible, low cost, BUT: not fast enough for large designs • Prototype: executes fully in parallel, fast, BUT needs logic analyzer to analyze limited number of signals, little to no debug capability • HW-Emulator: rich debug environment, parallel execution, provides many features that can be found in logic simulators, a specific case of HW Emulator: In-Circuit Emulator

14. Verification and Debugging Monitor • software tool or hardware unit • acts concurrent with the system or component • observes /monitors, verifies, analyses and records the execution of the system • e.g. measures the response time depending on the amount of user transactions (performance testing)

15. Verification and Debugging When do we have tested enough? • exhaustive testing is infeasible in practice • we define coverage criteria that have to be met by a testing • functional coverage (black-box view) • structural coverage (white-box view) • code coverage: path coverage, state coverage, statement coverage, decision coverage, MCDC ... • data flow coverage • in industry safety standards are used that describe the testing procedure and also the system development (RTCA/DO-178b, IEC 61508), certification is also used in some domains

16. Verification and Debugging Effort and Time • many factors influence the time spent on testing • know-how, experience, test-object (is it new to the test team?), test strategy, requirements on the object: is it safety critical?, how intensive must the object be tested?, what quality of the object should be achieved? ... • in theory in SW the test effort should be approximately 50% of the total development effort(rule of thumb according to the book Basiswissen Softwaretest) • in HW it should be approximately 70-80% (according to the slides from VL Hardware Modeling) • in practice: much much less 10-20% (according to the slides VL Hardware Modeling)

17. Verification and Debugging Summary • HW: • verification needs HW an SW tools • more effort is needed: no possibility for patches, the whole system/component must be replaced in case of failure • SW: • SW tools for verification are sufficient • no need for extra testing hardware (oscilloscope..)

18. Verification and Debugging List of used sources • slides: VO Testing of Embedded Systems (Raimund Kirner, Astrid Ademaj) • slides: VL Hardware Modeling (Martin Delvai) • slides: VO Embedded System Engineering (Wilfried Elmenreich) • book: Basiswissen Softwaretest, Andreas Spillner and Tilo Linz • wikipedia.org • wiktionary.org