ELN5622 Embedded Systems Class 10 Spring, 2003

1. ELN5622Embedded SystemsClass 10Spring, 2003 Aaron Itskovichitskova@algonquincollege.com

2. Outlook • Testing, & Verification, Reliability • Test Plan Creation & Execution. • Relaiability & Correctness • Design for Test & Debugging • Testing (Yield, Field Return, Golden System, Coverage), JTAG

3. Why test? • To reduce risk to both user and company • To reduce development and maintenance cost • To improve performance • To find bugs in software and hardware

4. To Find the Bugs • Halting Theorem proves it’s impossible to prove that an arbitrary program is correct • Given the right test you can prove that a program is incorrect • Testing is not about proving “correctness” of a program but about finding bugs • The only way to “know” how many bugs left is to test it with carefully designed test plan • Known bug is already “half bug”

5. To reduce the risk and costs • Minimize risk to yourself, your company and your customers • Earlier you detect the problem cheaper the fix

6. Costs of bugs • In 1990 HP sampled the cost of errors in software development during the year. The answer $400 million, shocked HP into a completely new effort to eliminate mistakes in writing software. The $400 waste , half of ot spent in the labs on rework and half in the field to fix the mistakes that escaped from lab amounted to 1/3 of the company total R&D budget

7. How to make bug fixing cheaper? • If we can’t ensure correctness of the released system how to make bug fix cheaper? • Design system to be field upgradeable • Re-configurable hardware (FPGA) • Separate application software from boot • Use Flash or EEPROM as application storage

8. When to test? • As early as possible • Statistically about 70% of the bugs found during the integration phase of the project were generated by code that had never been exercised before

9. What tests? • Every time the program is modified, it should be retested to assure that the changes didn’t break some unrelated behavior - REGRESSION TESTING • Individual developers test at the module level by writing stub code to substitute for the rest of the system hardware and software – UNIT TESTINGH

10. Test case design • Functional testing (black box) • Can and should be written in parallel with the requirements document • Coverage testing (white box) • Coverage test implies that your code is stable • Both kind of testing are necessary to test rigorously your embedded design

11. A bit of history • The first known computer bug came about in 1946 when a primitive computer used by Navy to calculate the trajectories of artillery shells shut down when a moth got stuck in one of it’s computing elements, a mechanical relay. Hence, the name bug for computer error.

12. When to stop testing • When the boss says • When the new iteration of the test cycle founds fewer than X new bugs • When a certain coverage threshold has been met without uncovering any new bugs • In case your system is mission critical look into DO-178B specification.

13. Choosing Test Cases • Functional tests • Stress tests: test that intentionally overload input input channels, memory buffers… • Boundary value tests: Inputs that represent “boundaries”within particular rangeand input values that should case the output to transition across a similar boundary in the output range • Exception test: Test that should trigger a failure mode or exception mode • Error guessing: Test based on the prior experience with testing similar products • Random tests: Usually the least productive form of testing • Performance tests: Test that performance expectation from the requirements are met

14. Choosing Test Cases • Coverage tests • Statement coverage: Test cases selected because they execute every statement in the program at least once. • Decision or branch coverage: Test cases chosen because they cause every branch (both true and false path) to be executed at least once. • Condition coverage: Test cases chosen to force each condition (term) in decision to take on all possible logic values.

15. Practical alternatives • Gray box testing-what it is? • White box tests – expensive to maintain need to be reengineered every time code is changed • Gray box – exploit knowledge of implementation without being intimately tied to the coding details

16. Some distinguishers of the embedded system • Embedded system must run reliably without crashing for long periods of time. • Embedded software must often compensate for problems with the embedded hardware • Real world events are usually asynchronous and non deterministic, making simulations tests difficult and unreliable • Did you read software “license agreement”?

17. Measuring test coverage • Code instrumentation methods - aka Software logging • Printf: intrusive - slows down system • Low intrusion Printf • Usage of logic analyzer to measure coverage • Decision coverage (DC): measures results of decision points in the code • Modified decision coverage (MDC): One step farther than DC evaluates the terms that make up decision point. • Hardware instrumentation methods (logic analyzer, trace,

18. How to test performance

19. Manufacturing tests • Build in self test (BIST) • Test bed • Golden system concept • JTAG boundary scan • Yield • Field return • Fault correlation and root cause analyzes