Backstop: A Tool for Debugging Runtime Errors

1. Backstop: A Tool for Debugging Runtime Errors Chris Murphy, Eunhee Kim, Gail Kaiser, Adam Cannon Columbia University

2. Observation • Novice/CS1 Java programmers have difficulty dealing with uncaught exceptions and the resulting stack traces java.lang.NumberFormatException: For input string: "4.5" at java.lang.NumberFormatException.forInputString (NumberFormatException.java:48) at java.lang.Integer.parseInt(Integer.java:456) at java.lang.Integer.parseInt(Integer.java:497) at Test.main(Test.java:124)

3. Problems • Students are often confused by runtime errors in Java • Stack traces were not developed with the introductory-level CS student in mind! • Few, if any, tools exist to help novice programmers interpret and debug runtime errors

4. Our Solution: Backstop • A command-line tool to produce detailed and helpful error messages when an uncaught runtime error (exception) occurs • Also provides debugging support by displaying the lines of code that are executed as a program runs, as well as the values of any variables on those lines

5. Hypothesis • “Backstop allows students to understand and debug runtime errors more easily than just the Java stacktrace” • Backstop is part of Columbia University’s ongoing efforts to develop tools to help CS1 students work with Java

6. Overview • Related Work • Handling of Runtime Errors • Usability Study • Debugging Support • Conclusions and Future Work

7. Related Work • Programming environments with easier-to-understand compiler errors • BlueJ, ProfessorJ, DrJava, … • Identify most common compiler errors • Gauntlet, Expresso, HiC, … • Identifying logical errors • InSTEP, DEBUG, … • Debugging • CMeRun

8. Handling of Runtime Errors • Designed to catch any uncaught exceptions in the student’s program • Once the exception (or any other Java runtime error) is caught, interpret its meaning and provide a user-friendly message that also seeks to provide some enlightenment as to what caused the error

9. Handling of Runtime Errors • BackstopRuntime “wraps” the student’s (compiled) Java program • Uses reflection to invoke the “main” method • Catches any uncaught exception and dispatches it to the appropriate handler • StackTraceElements and source code are used to produce a friendlier, more helpful error message that is appropriate for the type of exception

10. ********************************************************* ************************ ERROR ************************** ********************************************************* An error occurred at line 124 of Test.java in the method "main". I'm sorry, the program cannot continue. The line of code that caused the error is int myNum = Integer.parseInt(arg); It seems that the code wants to convert a String to an integer. However, the String "4.5" appears to be a floating point value, not an integer. Don’t forget, an integer can’t contain a decimal point. You may want to use a different datatype to store the value, like float. This is called a NumberFormatException. The full error is as follows: java.lang.NumberFormatException: For input string: "4.5" at java.lang.NumberFormatException.forInputString (NumberFormatException.java:48) at java.lang.Integer.parseInt(Integer.java:456) at java.lang.Integer.parseInt(Integer.java:497) at Test.main(Test.java:124) Refer to “the code”, not “you” Reinforce what is already known Make a suggestion Also use Java terms

11. Usability Study • Research Question: “Is the information provided by Backstop helpful for understanding runtime errors?” • 17 students (8 male, 9 female) who had just completed a CS1 summer course • Students were asked to find/fix a runtime error in code they did not write • After completion, were asked subjective questions about the experience

12. Usability Study • Subject was shown the code for a program that reads a String and displays the number of occurrences of each letter • Subject was then told how the algorithm works • It was then suggested that the code would produce an error for some inputs • Subject was suggested to try a sentence like “hello world” • Backstop error appeared

13. Problem Code // loop through and increment the values for each char for (int i = 0; i < s.length(); i++) { // get the character at index i char c = s.charAt(i); // increment the value in the array of occurrences // figure out its index by subtracting 'a' from it occur[(int)(c - 'a')]++; }

14. **************************************************** ********************** ERROR *********************** **************************************************** An error occurred at line 30 of StringCount.java in the method "main". I'm sorry, the program cannot continue. The line of code that caused the error is occur[(int)(c - 'a')]++; It seems that the code tried to use an illegal value as an index in an array. The code was trying to access an element at index -65 of the array called "occur". The expression "(int)(c - 'a')" had the value -65 when the error occurred. Remember, you cannot have a negative index. Be sure that the index is always positive. This error is called an ArrayIndexOutOfBoundsException. The full error is as follows: java.lang.ArrayIndexOutOfBoundsException: -65 at StringCount.main(StringCount.java:30)

15. Questions • Was it more useful than the stacktrace? • All said “yes” • Did the output mislead you? • Two said “yes”; one admitted he didn’t read it • Describe the tone (vs. stack trace) • Six said it was “friendly” • Four said “too long” • Two said “too pedantic for a general audience” • Three did not notice it

16. Observations • Most common complaint was that the output is too long • One possibility is giving the option to see more details or user-adjusted verbosity levels • Average time to complete the task with Backstop was around 5.8 minutes • Understanding the cause took 1-2 minutes

17. General Debugging Support • Display each line of code as it gets executed • File name • Line number • Source code • Also display the values of any variables

18. Original source code sum = x + y; y = x; Line number printed before code is executed, in case of exception Modified source code System.out.print("\nFib.java:27: sum = x + y;"); System.out.print(" (x was " + x);System.out.print(" , y was " + y );System.out.print(")"); sum = x + y; System.out.print(" (sum is now " + sum + ")"); System.out.print("\nFib.java:29: y = x;");System.out.print(" (x was " + x);System.out.print(")");y = x; System.out.print(" (y is now " + y + ")"); Everything on one line to retain original line number Verb tense indicates change of variable values Output at runtime Fib.java:27: sum = x + y; (x was 2 , y was 1) (sum is now 3) Fib.java:29: y = x; (x was 2) (y is now 2)

19. StringCount.java:27: char c = s.charAt(i); (i was 0) (c is now h) StringCount.java:30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now 1) StringCount.java:27: char c = s.charAt(i); (i was 1) (c is now e) StringCount.java:30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now 1) StringCount.java:27: char c = s.charAt(i); (i was 2) (c is now l) StringCount.java:30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now 1) StringCount.java:27: char c = s.charAt(i); (i was 3) (c is now l) StringCount.java:30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now 2) StringCount.java:27: char c = s.charAt(i); (i was 4) (c is now o) StringCount.java:30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now 1) StringCount.java:27: char c = s.charAt(i); (i was 5) (c is now !) StringCount.java:30: occur[(int)(c - 'a')]++; ************************************************************************* ********************************* ERROR ********************************* ************************************************************************* An error occurred at line 30 of StringCount.java in the method "main". I'm sorry, the program cannot continue. The line of code that caused the error is occur[(int)(c - 'a')]++; It seems that the code tried to use an illegal value as an index to an array. The code was trying to access an element at index -64 of the array called "occur". The variable "(int)(c - 'a')" had the value -64 when the error occured. Remember, you cannot have a negative index. Be sure that the index is always positive. This error is called an ArrayIndexOutOfBoundsException. The full error is as follows: java.lang.ArrayIndexOutOfBoundsException: -64 at StringCount.main(StringCount.java:30) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

20. Questions • Was it helpful? • All but one said “yes” • Did the output mislead you? • Five said “yes”, but four said “only at first” • Describe the appearance • Five said more spacing would make it easier to read • Four said more spacing would make it harder to read • Three said the spacing would make no difference

21. Evaluation • Backstop helped students identify and fix errors in code they did not write • Backstop automates/mimics the way that many CS1 students debug a Java program on the command line • Still need a study involving the use of both parts of Backstop to debug a runtime error

22. Conclusion and Future Work • Our contribution: Backstop • Makes the error messages produced by uncaught Java exceptions easier to understand (compared to the Java stacktrace) • Provides friendlier and more useful information about how to fix the error • Automates simple debugging approach • Future work could include integration with an IDE, and further studies to measure Backstop’s effectiveness

23. Backstop: A Tool for Debugging Runtime Errors Chris Murphy, Eunhee Kim, Gail Kaiser, Adam Cannon Columbia University http://www.cs.columbia.edu/~cmurphy/backstop cmurphy@cs.columbia.edu