Toward Target Level Testing and Debugging for Embedded Software Koehnemann, Lindquist

1. Toward Target Level Testing and Debugging for Embedded SoftwareKoehnemann, Lindquist Ritu Varma Roshanak Roshandel Manu Prasanna

2. Introduction • Identifies problems with the test case execution for embedded systems and propose solutions • Testing is done to reveal errors whereas debugging is done to locate and correct the cause of error. • Embedded systems are usually constrained by • Concurrent design • Real time constraints • Distributed hardware architecture • Device control dependencies • Execution visibility and control severely restricted • Why testing in Embedded Systems necessary?

3. Software Testing • Testing for embedded systems has four stages: • Module level Testing • Integration Testing • System Testing • Hardware/Software Integration Testing • Testing Methods which address the problem identified by the embedded systems • Concurrency • Real-time constraints • Embedded Environment are of interest to this paper.

4. Testing Concurrent Systems • Concurrency increases testing difficulties: • Large set of execution sequences • Subsequent Execution with the same input  different results • Valid result as well as how the program arrived at that result • Effect of the environment on testing and its results • Acceptable output more important than correct output- Control over program execution • Research: • Static analysis: must examine a large set of state  not scalable • Dynamic testing: passive watch of execution and replaying the execution in case of failure • Runtime control of concurrent program

5. Non-intrusive Testing • Control and testing techniques must not interfere with behavior of test program, timing issues • Why not instrumentation? • Insert probes to watch execution, link users object code with the rest of debugging system, Code interferes with execution behavior of the program • Intrusive, voids execution of system • Non-intrusive debugger for Ada • Separate processor executes testing system and communicates with the target processor through specialized hardware • Good start but don’t deal with the high level activities (process scheduling ,fault handling) only low level ( monitoring bus activity) • Other Real time embedded tools: • ROM monitors, Emulators, Bus Monitors

6. Impact of Underlying System • Problem: dealing with abstraction • Concurrency, IPC at high level and implementation at complier level (on run-time system) • In Development phase abstraction is fine but in Testing phase? • Complication • Without sufficient control over program , there is no way to ensure that a test case is testing the code. • Implementation difference in the host and the target environment. • Compilers provide runtime system operation to bridge the gap between language construct and hardware.

7. Current State of Embedded Testing • Concurrency causes Problems for testing and debugging • H/S integration testing reveal problems such as: • Incorrect handling of interrupts, Distributed Communication problems, Incorrect order of concurrent events, etc. • Goal: reveal errors early as possible Problem: Target level testing tools don’t exist • Two environments- • Host : supports s/w development and tools which give user control over program execution (high level interface) • Target: no support for software development and tools as minimum cost and space are the goals of the target. Target level testing occurs late in the development cycle and a small part is allocated for H/S integration testing. But,Target is the only location where errors can be revealed.

8. Current Solutions • Hardware solutions • Attempts at gaining execution visibility and program control • Bus monitors, ROM monitors and in-circuit emulator • Minimal effectiveness for software development - low level machine data gathered only • Mapping b/w low level events and entities in the program • Software solutions • Attempts to reduce the cost and time spent testing on target • Determining factors in software testing: • Level of Criticality of software module • Test platform availability • Test Classification • Criteria's that assign program modules to test platforms • Type of software, Hardware requirements, Test classification, Platform availability, Certification Requirements • Current solution attempts to reduce the amount of target testing

9. Are these Techniques Effective? • Expensive • Low level of functionality • Available too late • Poor test selection criteria • What about new architectures?

10. Expensive • Custom validation facilities • Little or no reuse • Hardware is late and erroneous Software must be malleable

11. Testing Problems • Host: high-level interface, language • Target: machine instruction and physical address • Translation: Tedious and error prone • Migration to target • Hardware’s errors have to be mitigated by software • Late in the development process – costs

12. More Problems • Test cases and scenarios depend on tools and platforms instead of theoretical test criteria • Guidelines not being followed • Hardware architecture changes rapidly • Solution: • Adding functionality to the underlying system to support debugging • Hardware and run-time system Specialized Middleware for development and testing of embedded systems?

13. Model Debugging SystemAdd functionality to target • Ada Semantic Interface Spec toolkit • Two communicating processor: • Running the test program • Running the debugger • Target processor can: • Execute and debug code only at a break point • Run the debugger as a separate processor • Provide a separate execution unit to execute the debugger • Interface between internal and external debugger

14. Changes in HW Architecture • Hardware partitioning of memory • Computational facilities for debugger • Hardware break points • Dedicated bus

15. Memory Partitioning • Software is infinitely malleable • Any change requires several tests and retests • Late changes is expensive • Hardware should bound software • SW is usually partitioned based on critical level instead of design factors  Hardware partitioning – protected address space for processes – restriction on access

16. Computational Facilities for Debugger • External debugger vs. Internal debugger • Internal debugger: located on target, executes from the target without interfering with the application • Runs as a regular process on the processor • Low-level or periodic high level process • Architecture provides separate facility for debugger execution • Target processor provides computational facility • Computation Facility utilize architecture to dump some representation of instruction being executed • Info must be parsed to create history. Implement state machines for messages sent by HW. • Program the (stateful) functional unit within the processor. At boot time, the internal debugger code gets loaded and reside there. Messages are passed between internal and external debuggers.

17. Hardware Breakpoints • Software and conditional breakpoints are not feasible for real time systems • Breakpoint registers – data and instruction types • State of a processor – all internal registers, pipeline and cache information must be saved when a break is encountered • Conditional breakpoints – how are they dealt with?

18. Architecture Support for Abstractions • Utilities must be built into architecture to provide basis for emulation, testing and program visibility • A new level of abstraction is needed: • Basic type (int, float) • Process, memory management, semaphore, etc • Hardware should be more aware of programming elements (software) • Awareness of the programming environment is important

19. Dedicated Bus • An interface between the processor and the external world • Protocol used across the connection • data rate, volume, direction, etc. • Master-slave relationship • Internal and external debuggers • Bus and debugger • Determination of active element within the processor • Processor is active – sends messages to debugger • Use special debugger portion of RTS – both debugger and RTS are active • Perhaps a combination of the two

20. Runtime System Additions • Goal: to minimize data and computational requirements of the internal debugger along with communication with the external debugger • Standardize services that provide visibility into RTS abstractions • Processes: concurrency is a common abstraction • Need to support modifying task state, communication and synchronization, and process scheduling • Interrupt management, Time management, Memory management, Exception Handling

21. Summary • Embedded system testing techniques are ill-equipped • Architectural and RTS changes have to be made • Evaluate the changes and check feasibility • Definition of Embedded Systems?? • What is the border between HW/SW? • Software techniques more suitable for less resource constraint system • Need for classification of embedded system?? • Need for specialized middleware for both design and testing of embedded systems • Need for software development techniques although not discussed in the paper