Finding and Preventing Run-Time Error Handling Mistakes

1. Finding and Preventing Run-Time Error Handling Mistakes (subliminal OOPSLA advertisement) Wes Weimer George Necula UC Berkeley

2. Finding and Preventing Run-Time Error Handling Mistakes (subliminal OOPSLA advertisement) Wes Weimer George Necula UC Berkeley

3. The Context • Reliability in programs • Think Java or C++ • But not protocols or specifications • Run-time errors • OS resource exhaustion • Network connectivity problems • Database access errors • Disk problems • Error-handling mistakes • Forgetting to release a lock when a disk error occurs

4. Glossy Summary • It is difficult to write error-handling code • We have an analysis for finding mistakes • Dataflow analysis and fault model • We found over 800 mistakes in 4 MLOC • We characterize these mistakes • What goes wrong in off-the-shelf code? • Existing PL features are insufficient • We propose a new feature • “compensation stacks” track your obligations • And back it up with case studies

5. Error-Handling is Significant • IBM Study: up to two-thirds of a program can be devoted to error handling [Cri87] • Our Survey: for Java, between 3% and 46% • Transitively reachable from catch, finally • Older, bigger programs have more handling • Mistakes in which “applications don't properly handle error conditions that occur during normal operation” are reported as one of the top ten causes of Java web app security risks

6. Unfortunately, it often does not work • Most common exception handlers: • Do Nothing • Print stack trace, abort program • Higher level invariants should be restored, interface requirements should be respected • Aside from “handling” the error, the code should clean up after itself • Why does this not happen? • It is difficult for programmers to consider all execution paths

7. Error Paths #1 – Original Code • ptII3.0.2/ptolemy/actor/gui/JNLPUtilities.java • Special Thanks: Christopher Hylands Brooks (verified) // Copy sourceURL to destinationFile without doing any byte conversion. private static void _binaryCopyURLToFile(URL sourceURL, File destinationFile) throws IOException { BufferedInputStream input = new BufferedInputStream(sourceURL.openStream()); BufferedOutputStream output = new BufferedOutputStream( new FileOutputStream(destinationFile)); // The resource pointed to might be a pdf file, which // is binary, so we are careful to read it byte by // byte and not do any conversions of the bytes. int c; while (( c = input.read()) != -1) { output.write(c); } input.close(); output.close(); } // line 294

8. Error Paths #1 – Original Code • ptII3.0.2/ptolemy/actor/gui/JNLPUtilities.java • Special Thanks: Christopher Hylands Brooks (verified) // Copy sourceURL to destinationFile without doing any byte conversion. private static void _binaryCopyURLToFile(URL sourceURL, File destinationFile) throws IOException { BufferedInputStream input = new BufferedInputStream(sourceURL.openStream()); BufferedOutputStream output = new BufferedOutputStream( new FileOutputStream(destinationFile)); // The resource pointed to might be a pdf file, which // is binary, so we are careful to read it byte by // byte and not do any conversions of the bytes. int c; while (( c = input.read()) != -1) { output.write(c); } input.close(); output.close(); } // line 294

9. Error Paths #2 – Try-Finally // Copy sourceURL to destinationFile without doing any byte conversion. private static void _binaryCopyURLToFile(URL sourceURL, File destinationFile) throws IOException { BufferedInputStream input; BufferedOutputStream output; try { input = new BufferedInputStream(sourceURL.openStream()); output = new BufferedOutputStream( new FileOutputStream(destinationFile)); // The resource pointed to might be a pdf file, which // is binary, so we are careful to read it byte by // byte and not do any conversions of the bytes. int c; while (( c = input.read()) != -1) { output.write(c); } } finally { input.close(); output.close(); } }

10. Error Paths #3 – Try-Finally // Copy sourceURL to destinationFile without doing any byte conversion. private static void _binaryCopyURLToFile(URL sourceURL, File destinationFile) throws IOException { BufferedInputStream input; BufferedOutputStream output; try { input = new BufferedInputStream(sourceURL.openStream()); output = new BufferedOutputStream( new FileOutputStream(destinationFile)); // The resource pointed to might be a pdf file, which // is binary, so we are careful to read it byte by // byte and not do any conversions of the bytes. int c; while (( c = input.read()) != -1) { output.write(c); } } finally { input.close(); output.close(); } } XXX

11. Error Paths #3 – Try-Finally // Copy sourceURL to destinationFile without doing any byte conversion. private static void _binaryCopyURLToFile(URL sourceURL, File destinationFile) throws IOException { BufferedInputStream input; BufferedOutputStream output; try { input = new BufferedInputStream(sourceURL.openStream()); output = new BufferedOutputStream( new FileOutputStream(destinationFile)); // The resource pointed to might be a pdf file, which // is binary, so we are careful to read it byte by // byte and not do any conversions of the bytes. int c; while (( c = input.read()) != -1) { output.write(c); } } finally { input.close(); output.close(); } } XXX

12. Error Paths #4 – Nested Try-Finally // Copy sourceURL to destinationFile without doing any byte conversion. private static void _binaryCopyURLToFile(URL sourceURL, File destinationFile) throws IOException { BufferedInputStream input; BufferedOutputStream output; input = new BufferedInputStream(sourceURL.openStream()); try { output = new BufferedOutputStream( new FileOutputStream(destinationFile)); try { // The resource pointed to might be a pdf file, which // is binary, so we are careful to read it byte by // byte and not do any conversions of the bytes. int c; while (( c = input.read()) != -1) { output.write(c); } finally { output.close(); } } finally { input.close(); } }

13. Where are we? • Summary and Context • Error Handling is Hard: Many Paths • Finding Error-Handling Mistakes • Characterizing Mistakes • Old & New PL Features • Case Studies • Grand Finale

14. Defining Error-Handling Mistakes • Want to show that programs make mistakes when run-time errors occur • What does it mean to make a mistake? • Safety Policy: • If a socket is opened, it must be closed before the program terminates • If a resource is acquired, it must be released • We consider four generic resources: • Sockets, files, streams, database locks • From a survey of catch, finally, finalize • Program should release them along all paths, even those with run-time errors

15. Fault Model • “exceptions” and “run-time errors” are highly correlated in Java [CDCF’03] • Ex: “Server crashes during call” -> UnmarshalException • Methods come with signatures listing exceptions they can throw • Fault Model: • A called method can terminate normally or raise any of its declared exceptions • A called method for which we have no signature can terminate normally or raise any exception for which there is an enclosing catch block or that is declared to escape the caller

16. Analysis Summary • Build Control-Flow Graph from source • Special attention to exceptional control flow • Methods can raise declared exceptions • Symbolically execute each method • Track outstanding resources along paths • Abstract away data values • Safety Policy determines when a resource has been acquired and when it has been released • At joins, merge paths with equal resources • If there is a path to the end of a method that has an outstanding resource, note it

17. Analysis Example: Buggy Program try { Socket s = new Socket(); s.send(“GET index.html”); s.close(); } finally { } // close should be in finally

18. Analysis Example start new socket send close end

19. Analysis Example start { } new socket send close end

20. Analysis Example start { } new socket { socket } send close end

21. Analysis Example start { } { } new socket { socket } send close end

22. Analysis Example start { } { } new socket { socket } send { socket } close end

23. Analysis Example start { } { } new socket { socket } send { socket } { socket } close end

24. Analysis Example start { } { } new socket { socket } send { socket } { socket } close { } end

25. Analysis Example start { } { } new socket { socket } send { socket } { socket } close { } { } end

26. Analysis Example: Path with Mistake Works for arbitrary type state FSMs as well. These examples show simple two-state policies. start { } { } new socket { socket } send { socket } { socket } close { } { } end

27. Report Filtering • Sources of False Positives: • if (sock != null) sock.close(); • sock_field_we_close_later = sock; • return sock; • Filter Error Reports • Whenever an error path contains one of the above forms, we remove a resource of the appropriate type • Removes all false positives in practice • Introduces false negatives • (3 in a random sample of ~50 eliminated reports, suggests 30 false negatives total)

28. Analysis Results

29. Where are we? • Summary and Context • Error Handling is Hard: Many Paths • Finding Error-Handling Mistakes • Characterizing Mistakes • Old & New PL Features • Case Studies • Current Work: Spec Mining • Grand Finale

30. Characterizing Mistakes • Common: Protect Some Areas, But Not All • staf’s STAXMonitor class: ObjectInputStream ois = null; try { ois = new ObjectInputStream(/* ... */); // ... } catch (StreamCorruptedException ex) { if (ois != null) { ois.close(); } showErrorDialog(/* ... */); return false; } Object obj = ois.readObject(); // no try ois.close(); // no finally

31. Characterizing Mistakes (#2) • Model: aX() must be followed by cX() • try-finally not nested (osage) • try { a1(); a2(); } finally { c2(); c1(); } • forget a resource (quartz) • try { a1(); a2(); } finally { c1(); } • unprotected loops (ohioedge) • for (…) { a1(); work(); c1(); } • various: flags and early release try { a1(); f=0; if (…) { f=1; c1(); } // close early work(); a2(); } finally { c2(); if (!f) c1(); } // or close late

32. Characterizing Mistakes (#3) • Complicated “flag-work” is common • This example adapted from Brown’s undo int f = 0; // flag tracks progress try { a1(); f = 1; work(); a2(); f = 2; work(); a3(); f = 3; work(); } finally { switch (f) { // note fall-through! case 3: try { c3(); } catch (Exception e) {} case 2: try { c2(); } catch (Exception e) {} case 1: try { c1(); } catch (Exception e) {} } }

33. Characterizing Mistakes (#4) • This approach does not scale well • Work control flow is duplicated in cleanup code int f = 0; // flag tracks progress try { a1(); f = 1; work(); if (…) { did_a2 = true; a2(); f = 2; } work(); a3(); f = 3; work(); } finally { switch (f) { // note fall-through! case 3: try { c3(); } catch (Exception e) {} case 2: if (did_a2) { try { c2(); } catch … } case 1: try { c1(); } catch (Exception e) {} } }

34. Destructors / Finalizers • Great for stack-allocated objects • Error-handling contains arbitrary code: • Example adapted from undo, which has 17 unique cleanup actions, one 34 lines long • Called by garbage collector • Too late! Programmers often use flags specifically to free resources early • No ordering guarantees • Socket sock = new Socket(); • Stream strm = new Stream(sock); • If sock and strm are collected in the same sweep, sock.finalize may be called before strm.finalize

35. We Propose: “compensation stacks” • Store cleanup code in run-time stacks • First-class objects, can pass them around • When “action” succeeds, push “cleanup” • “action” and “cleanup” can be arbitrary code • Pop all cleanup code and run it (LIFO) … • when the stack goes out of scope • or at end-of-scope if there is an exception • and/or early (when programmer specifies) • and when the stack is finalized • Trace of events (loosely): • a1 a2 a3 a4 a5 • a1 a2 … aX cX … c2 c1

36. Compensation Stacks • Generalized destructors • No made-up objects needed for local cleanup • Can have multiple stacks • e.g., one for each request in a webserver • Annotate important interfaces to take compensation stacks • Cannot make a new socket without putting “this.close()” on a stack of things-to-do • Everything on a stack will be run on all paths • Implicit “current scope” stack can optionally be assumed by default

37. Where are we? We’re almost done! • Summary and Context • Error Handling is Hard: Many Paths • Finding Error-Handling Mistakes • Characterizing Mistakes • Old & New PL Features • Case Studies • Grand Finale

38. Implementation, Case Studies • Extend Java with such compensation stacks • Annotate key interfaces (socket, stream, DB) • Annotate existing programs to use compensation stacks • Both for library resources (easy) • And for “unique” cleanup actions • Add no new error handling • Ensure that existing handlers are run on all paths • Run programs on third-party workloads • Mark Brody, George Candea, Tom Martell, …

39. Case Study #1: Brown’s undo • IMAP/SMTP Proxy (operator time travel) • Built for reliability • 35,412 lines of Java, 128 annotation sites • Contains many unique cleanup actions • 8- to 34-line code fragments • As well as many standard “close” actions • And up to 5 simultaneous resources in sequence • Results • 225 lines shorter (~1%) • No performance changes • Stack overhead dwarfed by I/O overhead

40. Case Study #2: Sun’s petstore • “Amazon.com lite” e-commerce, inventory • Raises 150 exceptions over 3,900 requests • Avg Response Time: 52.06ms (std dev 100ms) • 34,608 lines of Java, 123 annotation sites • Two hours of work • Standard “close” cleanup actions • 3 simultaneous resources: database handles • Results • 168 lines shorter (~0.5%) • 0 such exceptions over 3,900 requests • Avg Response Time: 43.44ms (std dev 77ms)

41. Conclusion • It is difficult to write error-handling code • We have an analysis for finding mistakes • Dataflow analysis and fault model • We found over 800 mistakes in 4 MLOC • We characterize these mistakes • Programmers forget some paths • Existing PL features are an awkward fit • We propose a new feature • “compensation stacks” track your obligations • And back it up with case studies

42. Any Questions?

43. False Negatives • hibernate2 - if, return, if, if, FN#1 • jatlite - if, if, if, if, field • quartz - FN#2, if, if, if, FN#3 • ejbca - if, if, if, if, if • hsqldb - if, if, field, if, field • jboss - field, if, field, if, return • mckoi-sql - fld, fld, fld, ret, ret • osage - ret, if, if, ret, if • portal - ret, ret, ret, ret, ret • staf - field

44. False Negative #1 (ResultSet) public Iterator iterate(Object[] values, Type[] types, RowSelection selection, Map namedParams, SessionImplementor session) throws HibernateException, SQLException { PreparedStatement st = prepareQueryStatement( getSQLString(), values, types, selection, false, session ); try { bindNamedParameters(st, namedParams, session); setMaxRows(st, selection); ResultSet rs = st.executeQuery(); advance(rs, selection, session); return new IteratorImpl( rs, session, getReturnTypes(), getScalarColumnNames() ); } catch (SQLException sqle) { JDBCExceptionReporter.logExceptions(sqle); closePreparedStatement(st, selection, session); throw sqle; } }

45. False Negative #2 (ObjectInputStream) protected Object getObjectFromBlob(ResultSet rs, String colName) throws ClassNotFoundException, IOException, SQLException { Object obj = null; Blob blobLocator = rs.getBlob(colName); InputStream binaryInput = null; try { if (null != blobLocator && blobLocator.length() > 0) { binaryInput = blobLocator.getBinaryStream(); } } catch (Exception ignore) {} if (null != binaryInput) { ObjectInputStream in = new ObjectInputStream(binaryInput); obj = in.readObject(); in.close(); } return obj; }

46. False Negative #3 (ObjectInputStream) protected Object getObjectFromBlob(ResultSet rs, String colName) throws ClassNotFoundException, IOException, SQLException { Object obj = null; InputStream binaryInput = rs.getBinaryStream(colName); if (binaryInput != null) { ObjectInputStream in = new ObjectInputStream(binaryInput); obj = in.readObject(); in.close(); } return obj; }

47. What Are Those Benchmarks?

48. Show A Nested Try-Finally // cayenne-1.0b4/src/…/dba/oracle/OraclePkGenerator.java, line 254 protected List getExistingSequences(DataNode node) throws SQLException { // check existing sequences Connection con = node.getDataSource().getConnection(); try { Statement sel = con.createStatement(); try { ResultSet rs = sel.executeQuery("SELECT LOWER(SEQUENCE_NAME) FROM ALL_SEQUENCES"); try { List sequenceList = new ArrayList(); while (rs.next()) sequenceList.add(rs.getString(1)); return sequenceList; } finally { rs.close(); } } finally { sel.close(); } } finally { con.close(); } } Most nested try-finally cases are not this easy to follow.

49. Show Us Another Bug // com/ohioedge/j2ee/api/org/tool/ejb/LetterTemplateEJB.java, line 154 private StringBuffer getColumnStringBufferFromReader() { StringBuffer buf = null; java.io.Reader reader = null; PreparedStatement prepStmt = null; ResultSet rs = null; Connection cn = null; try { cn = ConnectionFactory.getConnection("jdbc/LetterTemplateDB"); StringBuffer qry = new StringBuffer(); qry.append(" SELECT \"letterTemplate\" FROM LetterTemplate "); qry.append(" WHERE \"letterTemplateID\" = "+getLetterTemplateID()); prepStmt = cn.prepareStatement(qry.toString()); rs = prepStmt.executeQuery(); while (rs.next()) reader = rs.getCharacterStream(1); buf = readIntoStringBuffer(reader); rs.close(); prepStmt.close(); } catch (Exception e) { log.error(this.getClass().getName()+".getColumnStringBufferFromReader():"); e.printStackTrace(); }finally { try { cn.close(); } catch (Exception e1) { log.error(this.getClass().getName()+".getColumnStringBufferFromReader():"); e1.printStackTrace(); } } return buf; } Note error handler that prints a stack trace and does nothing else ...

50. Show A False Positive // org/axiondb/tools/Console.java, line 89 try { _conn = DriverManager.getConnection(buf.toString()); _stmt = _conn.createStatement(); } catch (SQLException e) { cleanUp(); throw e; } // … public void cleanUp() { try { _rset.close(); } catch (Exception e) {} try { _stmt.close(); } catch (Exception e) {} try { _conn.close(); } catch (Exception e) {} }