10.0 DEBUGGING TECHNIQUES

1. 10.0 DEBUGGING TECHNIQUES • Introduction • Rule of Thumb: Write good, bug-free code from start if you could • Testing/Debugging embedded software is more difficult than application software • Post-shipment application problems are more tolerable than embedded (real-time or life-critical) software

2. 10.0 DEBUGGING TECHNIQUES • 10.1 Testing on Host Machine • Some reasons why you can’t test (much, if any) on target machine: • Test early (target may not ready or completely stable) • Exercise all code, including exceptions (real situations may be difficult to exercise) • Develop reusable, repeatable test (difficult to do in target environment, and likelihood of hitting the same bug is low) • Store test results (target may not even have disk drive to store results)

3. 10.0 DEBUGGING TECHNIQUES • 10.1 Testing on Host Machine – 1 • Basic Techniques • Fig 10.1 – • Target system on the left: (hardware-indep code, hardware-dep code, hw) • Test system (on host) on the right: (hardware-indep code – same, scaffold – rest) • Scaffold provides (in software) all functionalities and calls to hardware as in the hardware-dep and hardware components of the target system – more like a simulator for them!

5. 10.0 DEBUGGING TECHNIQUES • 10.1 Testing on Host Machine – 2 • Basic Techniques • Fig 10.2 – • Radio.c -- hardware independent code • Radiohw.c – hardware dependent code (only interface to hw: inp() and outp() supporting vTurnOnTransmitter() and vTurnOffTransmitter() functions • Inp() and outp() must have real hardware code to read/write byte data correctly - makes testing harder!! • Fig 10.3 – • Replace radiohw.c with scaffold, eliminating the need for inp() and outp() – both are simulated in software – a program stub!!

12. 10.0 DEBUGGING TECHNIQUES • 10.1 Testing on Host Machine – 3 • Calling Interrupt Routines – • Embedded systems are interrupt-driven, so to test based on interrupts • 1) Divide interrupt routines into two components • A) a component that deals with the hardware • B) a component of the routine which deals with the rest of the system • 2) To test, structure the routine such that the hardware-dependent component (A) calls the hardware-independent part (B). • 3) Write component B in C-language, so that the test scaffold can call it • E.g., Fig 10.4 – • Hw component (A) is vHandleRxHardware(), which reads characters from the hw • Sw component (B) is vHandleByte, called by A to buffer characters, among others • The test scaffold, vTestMain(), then calls vHandleByte(), to test if the system works [where vTestMain() pretends to be the hardware sending the chars to vHandleByte()]

17. 10.0 DEBUGGING TECHNIQUES • 10.1 Testing on Host Machine – 4 • Calling the Timer Interrupt Routine • Design the test scaffold routine to directly call the timer interrupt routine, rather than other part of the host environment, to avoid interruptions in the scaffold’s timing of events • This way, the scaffold has control over sequences of events in the test which must occur within intervals of timer interrupts • Script Files and Output Files • To let the scaffold test the system in some sequence or repeated times, write a script file (of commands and parameters) to control the test • Parse the script file, test system based on commands/parameters, and direct output – intermixture of the input-script and output lines – into an output file • The commands in the script cause the scaffold to call routines in the B (sw-indp) component -- See Fig 10.5 and Fig 10.6 – for the cordless bar-code scanner

20. 10.0 DEBUGGING TECHNIQUES • 10.1 Testing on Host Machine – 5 • More Advanced Techniques • Making the scaffold automatically control sequence of events – e.g., calling the printer interrupt many times but in a controlled order to avoid swamping • Making the scaffold automatically queue up requests-to-send output lines, by automatically controlling the button interrupt routine, which will cause successive pressing of a button to let the next output line be received from the hardware (the printer interrupt routine). In this way, the hardware-independent software is controlled by the scaffold, where the button interrupts serve as a switch • The scaffold may contain multiple instances of the software-independent code, and the scaffold serves as a controller of the communication between the instances – where each instance is called by the scaffold when the hardware interrupt occurs (e.g., the scanner or the cash register). In this way, the scaffold simulates the hardware (scanner or register) and provides communication services to the software-independent code instances it calls. – See Fig 10.7

22. 10.0 DEBUGGING TECHNIQUES • 10.1 Testing on Host Machine – 6 • Objections, Limitations, and Shortcomings • 1) Hard to test parts which are truly hardware dependent, until the target system is operational. Yet, good to test most sw-independent parts on host (see Fig 10.8) • 2) Time and effort in writing scaffold – even if huge, it is worthwhile • 3) Having the scaffold run on the host and its RTOS – scaffold can run as low priority task within the RTOS and have nicely integrated testing environment • 4) The hard to justify limitations – can’t tell in scaffold until the actual test • Writing to the wrong hardware address – software/hardware interactions • Realistic interrupt latency due to differences in processor speeds (host v. target) • Real interrupts that cause shared-data problems, where real enable/disable is the key • Differences in network addressing, size of data types, data packing schemes – portability issues

24. 10.0 DEBUGGING TECHNIQUES • 10.2 Instruction Set Simulators • Using software to simulate: • The target microprocessor instruction set • The target memory (types - RAM) • The target microprocessor architecture (interconnections and components) • Simulator – must understand the linker/locator Map format, parse and interpret it • Simulator – takes the Map as input, reads the instructions from simulated ROM, reads/writes from/to simulated registers • Provide a user interface to simulator for I/O, debugging (using, e.g., a macro language)

25. 10.0 DEBUGGING TECHNIQUES • 10.2 Instruction Set Simulators – 1 • Capabilities of Simulators: • Collect statistics on # instructions executed, bus cycles for estimating actual times • Easier to test assembly code (for startup software and interrupt routines) in simulator • Easier to test for portability since simulator takes same Map as the target • Other parts, e.g., timers and built-in peripherals, can be tested in the corresponding simulated versions in the simulated microprocessor architecture • What simulators can’t help: • Simulating and testing ASICs, sensors, actuators, specialized radios (perhaps, in future systems!!) • Lacking I/O interfaces in simulator to support testing techniques discussed (unless additional provision is made for I/O to support the scaffold; and scripts to format and reformat files between the simulator, simulated memory, and the scaffold)

26. 10.0 DEBUGGING TECHNIQUES • 10.3 The assert Macro • The assert is used (with a boolean-expression parameter) to check assumptions • If the expression is TRUE nothing happens, if FALSE, a message is printed and the program crashes • Assert works well in finding bugs early, when testing in the host environment • On failure, assert causes a return to the host operating systems (can’t do on target, and can’t print such message on target – may not have the display unit) • Assert macro that runs on the target are useful for spotting problems: • 1) disabling interrupts and spin in infinite loop – effectively stopping the system • 2) turn on some pattern of LEDs or blinking device • 3) write special code memory for logic analyzer to read • 4) write location of the instruction that cause problem to specific memory for logic analyzer to read (the Map can help isolate which source code is the culprit!) • 5) execute an illegal op or other to stop the system – e.g., using in-circuit emulators

28. 10.0 DEBUGGING TECHNIQUES • 10.4 Using Laboratory Tools – Hardware-focused • Lab tools help reveal hard-to-find, very infrequently occurring bugs • Types useful to software engineers: • Voltmeters (measure voltage diff); Ohmmeters (measure resistance/connectedness) • Oscilloscopes (scopes) test events that repeat periodically – monitoring one or two signals (graph of time v. voltage), triggering mechanism to indicate start of monitoring, adjust vertical to know ground-signal, used as voltmeter (flat graph at some vertical relative to ground signal), test if a device/part is working – is graph flat? Is the digital signal coming through – expecting a quick rising/falling edge (from 0 – VCC or VCC – 0) – if not, scope will show slow rising/falling – indicating loading, bus fight, or other hardware problem • (See Fig 10.10, Fig 10.11, Fig 10.12, Fig 10.13, Fig 10.14)

34. 10.0 DEBUGGING TECHNIQUES • 10.4 Using Laboratory Tools – Hardware-focused - 1 • Logic Analyzer • Like storage scopes that (first) capture many signals and displays them simultaneously • It knows only of VCC and ground voltage levels (displays are like timing diagrams) – Real scopes display exact voltage (like analog) • Can be used to trigger on-symptom and track back in stored signal to isolate problem • Many signals can be triggered at their low and/or high points and for how long in that state • Used in Timing or State Mode

35. 10.0 DEBUGGING TECHNIQUES • 10.4 Using Laboratory Tools – Hardware-focused – 2 • Logic Analyzers in Timing Mode • Find out if an event occurred – did cordless scanner turn on the radio? • Measure how long it took software to respond to an interrupt (e.g., between a button interrupt signal and activation signal of a responding device – to turn off an bell) • Is the software putting out the right pattern of signals to control a hardware device – looking back in the captured signal for elapsed time • (See Fig 10.15 on response time) • (See Fig 10.16 on elapsed time) • (See Fig 10.17 – a typical Logic Analyzer with on-screen button, mouse, keyboard, network adaptor, disk storage for storing configurations/settings, ribbon cables)

39. 10.0 DEBUGGING TECHNIQUES • 10.4 Using Laboratory Tools – Hardware-focused – 3 • Logic Analyzers in State Mode • Captures signals only on clock-event occurring from the attached hardware • Typical use: instructions read/fetched by microprocessor, data read from or written to ROM, RAM, or I/O devices • To do so, connect LA to address and data signals and RE/ signal on the ROM (or RAM) • If triggering is on rising edge of RE/ pin, address and data signals will be captured • Output of LA, called trace, is stored for later analysis – see Fig 10.18 • LA can be triggered on unusual event occurrences, then capture signals therefrom – especially for debugging purposes (e.g., writing data to wrong address, tracking a rarely occurring bug, filtering signals for select devices or events) • LA can’t capture all signals, e.g., on fetch from caches, registers, un-accessed memory

41. 10.0 DEBUGGING TECHNIQUES • 10.4 Using Laboratory Tools – Hardware-focused – 4 • In-Circuit Emulators (ICE) • Replaces target microprocessor in target circuitry (with some engineering) • Has all the capabilities of a software debugger • Maintains trace, similar to that of an LA’s • Has overlay memory to emulate ROM and RAM for a specified range of address within the ICE (rather than the system’s main ROM or RAM) – facilitates debugging • ICE v. LA • LA’s have better trace and filtering mechanism, and easier to detail and find problems • LA’s run in timing mode • LA’s work with any microprocessor – ICE is microprocessor-specific • LA’s support many but select signals to attach, ICE requires connecting ALL signals • ICE is more invasive

42. 10.0 DEBUGGING TECHNIQUES • 10.4 Using Laboratory Tools – Hardware-focused – 5 • Hardware Peculiarities that Make Debugging Difficult • Inter-pin distances/intervals for attaching probes – getting smaller • Providing sockets for debugging hardware – simply increases product size • ASIC’s encase signals that are hard to probe and track using LA’s or ICE’s • Use of RISC architectural designs makes it difficult to track when read/write happen in on-board (microprocessor) caches – different from the external RAM or ROM • Increasingly necessary to know the lab tool chain as it influences the design of product

43. 10. DEBUGGING TECHNIQUES • 10.4 Using Laboratory Tools – Hardware-focused – 6 • Software-Only Monitors • Monitors allow running an embedded system in the target environment, while providing debugging interfaces on both the host and target environments • A small portion of the Monitor resides in the target ROM (debugging kernel or monitor): • The codes receives programs from serial port, network, copies into target’s RAM, and run it with full debugging capabilities to test/debug the programs • Another portion of monitor resides on host – provides debugging capability and communicates with the debugging kernel over serial port or network, without hardware modifications • Compiled, linked (may be located into Map) code is downloaded from the host (by the portion on the host) to the target RAM or flash (received by the kernel) • Other designs: ROM Emulator interface and JPAG comm. port on the target processor • (See Fig 10.19)