An Undergraduate Course on Software Bug Detection Tools and Techniques

1. An Undergraduate Course on Software Bug Detection Tools and Techniques Eric Larson Seattle University March 3, 2006

2. Introduction • Course was taught at Seattle University in Winter 2005 quarter. • 9 senior undergraduate students • senior elective • Key contributions: • First course on software bug detection course geared toward undergraduate students. • Assignment infrastructure where students can create their own software bug detection tools.

3. Talk Outline • Goals and Background • Course Content • Programming Assignments • Results • Future Work

4. Goals of the Course • Learn and analyze algorithms that can be used to find bugs in software. • Understand why software bug detection is hard. • Gain experience developing software bug detection tools. • Become better programmers by thinking about software bug detection.

5. Non Goals of the Course • Testing • Only briefly mentioned in the course (types of testing, coverage criteria) • A course on testing would make a nice complement to this course. • Material in this course would be relevant in a testing course. • Debugging • Once a bug is found, what is the source of the bug? • Occasionally, we would talk about enhancements to a tool that would make debugging easier. • Students mentioned they would like to have learned more.

6. Key Challenges • Geared toward undergraduates. • Toned down the theory but did not eliminate it. • More practical, give students experience writing software tools. • No textbook. • Used relatively “easy-to-read” papers. • Few prerequisites. • Only prerequisite was a course in algorithms. • No pre-reqs in compilers, software engineering, automata theory, or software testing. • Focused primarily on C programs.

7. Format of the Course • Traditional lecture format • Some in-class activities and discussions • Assignments • Three programming assignments (in pairs) • Daily homework exercise (individual) • Grading • Class participation 5 % • Homework 20 % • Exams (midterm and final) 30 % • Programming assignments 45 %

8. Course Content Course was broken down into four units: • Program Analysis and Terminology • similar to back-end compiler analysis • special attention on interprocedural and pointer analysis • Dynamic Bug Detection • start of unit focused on testing • adding instrumentation to programs to: • manage additional state about variables or memory used in the program • check additional state to detect bugs

9. Course Content • Static Bug Detection • symbolic path simulation • constraint analysis • model checking • Other Topics • concurrent programs • safe languages • preventing security attacks (ex: StackGuard)

10. Assignment Infrastructure • Assignments were completed using a source to source converter called SUDSE. • Contains static analyses for static bug detection. • Converts a subset of C to instrumented C. • No enums, floating point values, or unions. • Since students were familiar with C++, the cin and cout I/O statements were added. • Internal representation is an AST of simplified C statements for easier analysis. • Side effects and short circuited operations are removed. • Complex expressions broken down into two or more simpler expressions. • Also suitable for other types of assignments. • code coverage tool • compilers • profiler

11. // Original code int bar(int x) { int a[5]; int i; for (i = 0; i < 5; i++) { a[i] = i * i; } return a[x]; } // Simplified code int bar(int x) { int a[5]; int i; int T1, T2; i = 0; T1 = i < 5; while (T1) { a[i] = i * i; i = i + 1; T1 = i < 5; } T2 = a[x]; return T2; } Example: Simplification

12. Assignments • Program analysis • control-flow graph and data flow analysis • uses of uninitialized variables. • Dynamic bug detection • more on this later… • Static bug detection • null deference checker. • open-ended (students can use any technique they wanted)

13. Dynamic Array Checker • In this assignment, students had to create a tool that detects array out-of-bounds errors. • In part 1, students had to add instrumentation calls to interesting statements: • Array declarations • Array references / pointer dereferences • Pointer assignments • Arrays/pointers go out of scope • In part 2, students had to write the instrumentation routines to manage the array state and check for errors.

14. Example: Instrumented Code #include "ptr_table.h" int bar(int x) { int a[5]; create_array_entry((void *) a, 5); int i; int T1, T2; i = 0; T1 = i < 5; while (T1) { check_array_bounds((void *) a, i, __FILE__, __LINE__); a[i] = i * i; i = i + 1; T1 = i < 5; } check_array_bounds((void *) a, x, __FILE__, __LINE__); T2 = a[x]; delete_array_entry((void *) a); return T2; }

15. Instrumentation Functions void create_array_entry(void *addr, int size); • Adds an entry to the array table. • Entry is indexed by addr and stores the size of the array. void check_array_bounds(void *addr, int index, const char *file, int line); • Search for the entry in the table using addr. • Check to make sure that the index is not negative and less than the size of the array. • If the check fails, print out an error that contains the file name and line number.

16. Results • Course was a success! • Based on student feedback and personal observations. • Assignments represented a mix of several key areas in CS: theory, OO-development, algorithms • The second assignment was the most successful. • First assignment was not interesting or challenging to the students who had background in compilers. • Not enough time for the open-ended third assignment. • Mixed reactions on favorite course topics. • Program analysis was either favorite or least favorite. • Some students wanted to learn more about debugging.

17. Future Modifications • Allow more time for the third assignment. • Reduce the first assignment to a warm-up exercise. • Add section on debugging techniques. • Use existing bug detection tools (valgrind). • Explore software engineering practices. • Where do software bug detection tools fit in different software engineering processes? • Coding standards.

18. Questions and Answers