Automated Testing Environment

1. Automated Testing Environment Concepts required for testing embedded systems adopted in this course (quizzes, assignments and laboratories) Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

2. To be tackled today • Why test, and what kinds of tests are there? • Difference between • TLD – Test Last Development • TDD – Test Driven Development (Test First Development) • How do you add tests? • More details on the testing process • Other kinds of available tests – time and access • Design of custom TESTs and ASSERTS Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

3. Why test? • Unit test • I AM CURRENTLY BUILDING A NEW FUNCTION – does it work the way I expect? • Testing my and my partner’s understanding of the Design • WE ARE DESIGNING A NEW PRODUCT-- If we can’t explain (to ourselves via a test) the product ideas – how do we know when product is working? • Regression testing • MY PARTNER ADDED A NEW FEATURE to a product – how do I know it did not “break something” I wrote? • System testing • PROOF TO THE CUSTOMER that the product (Lab. or assignment) works the way that was requested. Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

4. What are the problems with this sort of test? Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

5. VisualDSP development environment(IDDE) Lab. station (ICT318 / 320) BF533 Evaluation Board Each time we want to send a message (via printf( ) or cout) we must “stop” the processor from doing what we want it to do, and then send messages over the interface – Slow, plus the testing can impact on peripheral performance. E.g. no sound while printing in Assignment 2 and Lab. 1 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

6. More issues with this sort of test! • Tests are mixed up “with production code” • Must remove tests before release to customer • Unreliable – may result in test messages at wrong time • Difficult to “add” and “remove tests” • Can’t be automated – therefore unlikely to have tests run often during development – leads to unreliable product Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

7. E-TDD Tool • E-TDD – embedded test driven development • Build-the-tests-first ideas for product development has been made popular by the “Agile” approach • Requires an automated testing approach • In this class, we are going to use the E-TDD tool for its automated behaviour • Will use in both “test last” development (what most people currently do) and in “test first” development • I (customer) will provide tests so that you can check that you have developed the “right” stuff at the “high” level • You will build additional tests to check your own code at the “low level” Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

8. Test Environment • Can download “LabTestEnvironment.zip” file from the web • LabX directory iswhere you keepyour code • LabXTests is whereyou keep the testsfor LabX • E-TDDIncludes directory contains “everything” needed to make test environment work Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

9. Build a new VDSP project “Assignment1Test” and place in “AssignmentTests” directory • Now add certain standard files to the project – same set of files for all assignments and laboratory • All found in E-TDDIncludes directory Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

10. Now add your Assignment1 subroutines files to the Assignment1Test project • Don’t move or copy the files into the test directory – just “add” them to the project • Note the use of “Assignment1.h” which has the prototypes for all “C++” and “assembly code” functions used (from Assignment dir.) This files are “added” NOT copiedfrom the Assignment1 directory Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

11. Now we need to develop and add “Testmain.cpp” Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

12. Now we need to develop and add Assignment1Tests.cpp” 1 2 3 4 5 6 7 8 9 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

13. Tests have a standard format 1 3 4 8 9 TESTGROUP NAME TEST NAME TEST TYPE Add further tests at this location TESTGROUP NAME, TEST TYPE Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

14. Tests then need to be customized 1 2 3 4 5 6 7 8 9 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

15. Test Components • The ACTUAL Test (This is a TEST MACRO) CHECK(result1 == result2) • Would provided better test messages if we “self-documented the code” CHECK(result_CPP == result_ASM); • Test CONTROL statements (ALL MACROS) • TEST_CONNECT( test_name) • LINK_TEST( test_name, test_type) • TEST_LEVEL( which_test_level) Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

16. Stages to “automating” the tests • Activate the E-TDD Gui (E-TDDIncludes directory) • This causes the GUI to appear, and adds new “menus” to VisualDSP Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

17. Stages to “automating” the tests 2) Select TDD Connection | Refresh Connection List This automatically builds the Force Connect Test file Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

18. Now “build and run” the tests Quick test failure visualization Detailed test failure information Clicking on the “detailed test info”automatically takes you to the “test” that failed Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

19. Test information modes arecontrolled via “ActivateTests.cpp” • Show Failures only • Show Failures and Successes Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

20. Add a separate test to show that result2(from the ASM code) is equal to 7 Add further tests at this location Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

21. Write the tests for “self-documenting” messages(What does “result1 == result2” mean if 200 tests present?) HAVE WANT Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

22. Write the “CPP” code to get to workbut “Write the ASM for speed” FAIL SUCCESS SUCCESS Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

23. Did the ASM code “destroy” or even “use” the CPParray[ ] defined in CPPassignment1.cpp? Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

24. Available ASSERTS LONG INT tests – 32 bit – typical for “C++” • CHECK(condition) – Success if passes • CHECK_EQUAL(expected, actual) • EXPECTED_FALSE(condition) – Success if fails • ARRAYS_EQUAL(expected, actual, length) • CHECK_VALUES(expectedTemp, condition, actualTemp) DOUBLES_EQUAL(expected, actual, threshold) Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

25. ASSERTS for embedded systems • Short int tests – 16 bit – typical for embedded systems • SHORTS_EQUAL(expected, actual) • SHORTS_CHECK(expected, condition, actual) • USHORTS_EQUAL(expected, actual) • USHORTS_CHECK(expected, condition, actual) • Timing of routine – Blackfin Specific • MAXTIME_ASSERT(max_time, function) • Useful utility • MEASURE_EXECUTION_TIME(timetaken, function) • Examples available in the tests associated with Lab. 1 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

26. You can add your own ASSERTSASSERT FORMAT #define CHECK_VALUES(expectedTemp, condition, actualTemp)\ if (!((expectedTemp) condition (actualTemp))) \ {\ FAILURE1(#expectedTemp##" !"#condition##" "#actualTemp);\ }\ else {\ SUCCESS1(#expectedTemp##”""#condition##" "#actualTemp);\ } Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

27. Practice – Assert Design • Design an assert that will determine whether the lower 8 bits of a 32-bit variable is identical to the lower 8 bits of a second 32-bitvariable. • The two variables should remain unchanged by the assert (since they may be reused later) • The upper 24 bits of each variable MUST be ignored in the assert • Does it matter if the variables were defined in the .data section of an assembly code routine or in an C++ code routine? Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada

28. Handled today – some easy quiz and midterm questions • Why test, and what kinds of tests are there? • Difference between • TLD – Test Last Development • TDD – Test Driven Development (Test First Development) • How do you add tests for Assignment 1? • More details on the testing process • Other kinds of available tests – time and access • Design of custom TESTs and ASSERTS Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada