An Effective Method to Control Interrupt Handler for Data Race Detection

1. An Effective Methodto Control Interrupt Handlerfor Data Race Detection Makoto Higashi †, Tetsuo Yamamoto‡, Yasuhiro Hayase †, Takashi Ishio † and Katsuro Inoue † † Osaka University ‡ Ritsumeikan University

2. Outline • Motivation • Embedded software and data race conditions • Approach • Control of an interrupt handler • Case study • uClinux • Summary and future work AST2010

3. Reliability of embedded software • There are many safety-critical embedded software • Pacemaker: risk of losing human life • Mobile phone: risk of intercepting personal data • Embedded system consists of external devices and control software • Embedded software becomes aware of inputs from external devices through interruptsor I/O memory AST2010

4. Interrupt-driven software • We focus on interrupt-driven software in embedded software, where processing is initiated when external devices signal the CPU • Interrupts add fine-grained concurrency to the software main(void) handler(void) Main routine Interrupt handler interrupt if (op == 1) … op = 0 interrupt return return

5. Particular fault to interrupt-driven software • Main routine shares memory with interrupt handler • There is a failure at the particular timing • Interrupt occurs at unexpected timing • Interrupt handler changes shared memory It is important to detect data race conditions AST2010

6. Example of data race condition Divide 100 by x in case of x != 0 Divide 100 by x in a state of x == 0 variable x Main routine divide(void) Interrupt handler access access interrupt_handler(void) interrupt access yes interrupt x != 0 x = 0 interrupt ret = 100 / x no interrupt return return Detection of data race conditions through testing AST2010

7. Testing Process • Testing process of non interrupt-driven software • Input values to module • Check return value from the module • Testing process of interrupt-driven software • Combination Interrupt handler with other module • Consider a wide variety of Interrupt timing AST2010

8. Key ideas 1/2 • Condition of data race condition • Main routine accesses a variable twice • 1st access is reading or writing • 2nd access is reading • Main routine assumes that the value of the variable is unchanged • Interrupt occurs between 1st and 2nd access • Interrupt handler changes the value of the variable x = 3 x != 0 interrupt Assumes x != 0 interruput no ret = 100 / x ret = 100 / x AST2010 Assumes a == 3 return

9. Key ideas 2/2 • Control of execution path of interrupt handler • Embedded software uses memory-mapped I/O for communicating with external devices • Load instructions read the value of memory • Substitute user specified value for the value of memory AST2010

10. Our work • Objective: Testing of interrupt-driven software • Approach: Detection of faults related to interrupts (data race conditions) to cause interrupts automatically • Result: Detection a fault not to cause interrupts manually AST2010

11. Mechanism to cause interrupts Machine language CPUemulator LDR ADDR ADD MOV ・ ・ ・ ・ ・ ・ Interrupt handler: ・ ・ ・ ・ ・ ・ RET interpreter interrupt instruction Check read instruction or write instruction Configuration file Mechanism to cause interrupts A user specifies the kind of interrupt AST2010

12. Prevention of infinite loop Mechasin to cause interrupts Access memory Compare current program counter with the saved counter Saved counter Current program counter LDR ADDR ADD MOV ・ ・ ・ ・ ・ ・ Interrupt handler: ・ ・ ・ ・ ・ ・ RET Saved counter Result of comparison If the saved counter is different from the current program counter, casuse an interrupt and save program counter Current program counter interrupt Prevent causing interrupts at the same location Interrupt handler AST2010

13. Mechanism to substitute values Read instrunction CPUemulator memory LDR ADDR ADD MOV ・ ・ ・ ・ ・ ・ Memory access Memory access Memory access 0100011 1010101 0111100 0101111 ….. ….. Mechanism to substitute values check if the memory address is user specified address If yes, Substitute the value Return user specified value Return value Return value AST2010

14. Configuration file 1/2 • Interrupt • The kind of interrupt • Support only one kind of interrupt in single file • Memory address • An address which is mapped to external device • If you know memory address of global variable, the address can be specified AST2010

15. Configuration file2/2 • Function name • Substitute new value within only specified function • Because it is very slow to substitute values within all functions • New value • Constant • Global variables • Current value AST2010

16. Case study • We have applied our method to software which contains data race condition • Aim • Investigate the process to detect the data race condition • Target software • uClinux AST2010

17. Data race condition on uClinux • When sending characters of queue, the code accesses out of queue • After checking the count of queue, the routine sends characters • Just after the checking, interrupts occurs Main routine Interrupt handler ・ ・ The count of queue is 1 if (xmit_cnt <= 0 || ……) return; ・ ・ ・ xmit_cnt--; if (xmit_cnt <= 0 || ……) return; ・ ・ ・ xmit_cnt--; interrupt The count of queue is 0 Access out of queue AST2010

18. Procedure to detect the data race condition • Assign 5 to the count of queue • Because interrupts occurs 4 times before the count of queue is checked • Call main routine Static void rs_flush_chars(struct tty_struct *tty){ struct m68k_serial *info = ……; m68328_uart *uart = ……; ・ ・ ・ if (xmit_cnt <= 0 || ……) return; ・ interrupt interrupt interrupt interrupt AST2010

19. Testing process • Doubt the possibility to cause a data race condition within a certain module A • Assume an interrupt handler B to cause the data race condition in cooperation with module A • Specify the kind of the interrupt handler to configuration file • Test module A AST2010

20. Cost • CPU cycles • Our method: 72,417,488 • Normal execution: 4,836,078 About 15 times • The total number of cycles took in the interrupt handler: 69,952,632 AST2010

21. Data race condition that our mechanisms cannot detect 1: unsigned int len = 0; 2: void str_cpy(char *buf, char *str); 3: { 4: len = strlen(str); 5: if((0 < len) && (len <= strlen(str))) 6: memcpy(buf,str,len+1); 7: } 8: 9: void interrupt_handler(void){ 10: len++; 11: } No data race condition interrupt interrupt data race condition AST2010

22. Summary and future work • We have implemented 2 mechanisms to a CPU emulator to test for data race conditions in interrupt-driven software • Causes interrupts automatically • Substitute values of memory • Future work • More appropriate timing of interrupts • Multiple kinds of interrupts AST2010