Simplify Testing using Fault Injection

1. Simplify Testing using Fault Injection Rob Grzywinski rgrzywinski@yahoo.com

2. Motivation • Writing and testing an IO-intensive application • Robust, asynchronous event (message) collector / router • NIO (early 1.4 days) • How to best simulate and test various failure modes? • NIO buggy as all get-out • Networks are inherently flaky and problematic • Didn’t want to have to obfuscate or drastically change code in order to test it

3. Agenda • Quick Introduction to Fault Injection • Why Use Fault Injection • Fault Injection Examples • Beyond Fault Injection • Quick Introduction to AOP • Fault Injection in Java • AspectJ • Javassist • Wrap Up

4. Fault Injection 101 • A fault is an abnormal condition or defect which may lead to a failure[10] • Fault injection involves the deliberate insertion of faults or errors into a computer system in order to determine its response [9] • The goal isn’t to recreate the conditions that produced the fault

5. Fault Injection 101 (cont) There are two primary steps to Fault Injection: • To identify the set of faults that can occur within an application, module, class, method, etc. • e.g. if the application does not use the network then there’s no point in injecting network faults • In practice this isn’t as easy as it sounds e.g. 3rd party libs • To exercise those faults to evaluate how the application responds • Does the application detect the fault, is it isolated and does the application recover from it? [8]

6. Example byte[]readFile() throws IOException { ... final InputStream is = new FileInputStream(…); ... while((offset < bytes.length) && (numRead = is.read(bytes, offset, (bytes.length - offset))) >= 0) offset += numRead; ... is.close(); return bytes; } (from http://javaalmanac.com/egs/java.io/File2ByteArray.html) What could go wrong with this code? • new FileInputStream() can throw FileNotFoundException • InputStream.read() can throw IOException and IndexOutOfBoundsException and can return -1 for end of file • is.close() can throw IOException How do we test how the application responds to one of these situations? Specifically, how do we inject these faults so that we can test how the application responds?

7. Possible Techniques • Force the situations at the OS level • Quite hairy to reproduce reliably when they’re needed • Refactor the snot out of it • Replace the call to InputStream.read() to some local instrumented method • Create our own instrumented InputStream subclass possibly using mock objects • Inject the subclass via IoC (requires some framework such as PicoContainer or Spring) • Just comment out the code and replace with throw new IOException() • Egad! We need a way to inject a fault without changing the code!

8. Why Use Fault Injection • Fault occurred in a production application and it is determined that that is a valid fault (i.e. it is expected to occur) • Inject the fault in a testing environment without having to reproduce the conditions that actually caused the fault • Any fault that is difficult or inconvenient to reproduce in testing: • Out of memory • Disk full • Database crash • 3rd party library / Java bug • Deadlock • Resource contention • Dumb users • Invalid / stale sessions • Corner cases 

9. Why Use Fault Injection (cont) • You publish an API and you want to be tolerant of user input or response • It is inconvenient or undesirable to refactor code to facilitate testing • e.g. 3rd party library • You believe that testing code coverage is a good metric to measure for code quality [11] • Injecting faults forces exception paths to be followed More to come …

10. Reminder You don’t have to recreate the conditions that caused the fault … … you only have to recreate the fault e.g. if an OhNoException crashed your application because some little old lady drove her 1966 Buick Skylark into your datacenter -- you don’t need the Skylark or the little old lady to fix the application

11. Quick Aside Traditionally, Fault Injection is centers around faults derived from hardware: • Disk faults such as file missing and corrupt file • Network faults such as host not found, host inaccessible, and high latency • Memory faults such as corrupt memory • Operating system faults such as unexpected hardware interrupts Traditionally used in mission-critical fault tolerant environments

12. Quick Aside (cont) Fault Injection comes in two favors: • Hardware-based • Typically requires specialized hardware • Software-based • Traditionally attempts to mimic hardware-based fault injection and typically involves direct interaction with the operating system • More recently expanded to include any fault that can occur in software such as: • Mutated bytecode • Exceptions • Invalid or mutated input / output values • Deadlock / resource contention • Application Security • etc.

13. Example Creating a directory and temporary file final File directory = new File(…); final boolean success = directory.mkdirs(); if(success) { final File file = File.createTempFile(…, …, directory); ... } Faults: • Making the parent directory(s) fails (returns false) • Temp file creation failure (throws IOException) • Permissions • Parent directory(s) deleted • Injection of a specific temp file

14. Example (cont) Great uses of Fault Injection that I can’t easily show in examples: • Deeply nested exceptions • Effects of unchecked exceptions • Consequences of using catch(Exception e) i.e.NullPointerExceptions that go unnoticed • Complex / real-world cases

15. Intelligent Fault Injection Fault injection doesn’t have to be all on or all off Logic can be coded around injected faults e.g.InputStream.read() • Throw IOException after n bytes are read • Return -1 (EOF) one byte before the actual EOF • Sporadically mutate the read bytes

16. Beyond Fault Injection The techniques that I’m going to demonstrate aren’t specific to injecting faults (i.e. conditions that may lead toa failure) • “What if” analysis • Test difficult corner cases e.g.some optimization routine runs when certain conditions are met • Initialize application state to some desired value • etc. The techniques are similar to mock objects or IoC but much more granular

17. Summary Intercept operations and inject code to: • Perform a custom function • Return a custom value • Modify a input parameter • Throw an exception • Set the state of an object to a specific value

18. Checkpoint Done: • Quick Introduction to Fault Injection • Why Use Fault Injection • Fault Injection Examples • Beyond Fault Injection Next: • Quick Introduction to AOP • Fault Injection in Java • AspectJ • Javassist • Wrap Up

19. Checkpoint (cont) Questions?

20. Aspect Oriented Programming 101 Join Point a point in the flow of a running program [14] It is the “where” in AOP • Not all AO environments support all join points e.g. Spring AOP only supports method-based join points

21. Aspect Oriented Programming 101 (cont) Join point examples • reading or writing a field • calling or executing a method or constructor • catching or throwing an exception A “join point shadow” is the location of a join point in the source code or bytecode of the program [5] We will use “join point” and “join point shadow” interchangeably

22. Aspect Oriented Programming 101 (cont) Pointcut a set of join points [6] Effectively a query where the join points are the data that is being queried – the “which”

23. Aspect Oriented Programming 101 (cont) Advice code that executes at each join point picked out by a pointcut[7] Advice is the code that you’re injecting -- the “what”

24. Aspect Oriented Programming 101 (cont) Aspect a logical collection of advice An aspect is to AO what an object is to OO

25. Aspect Oriented Programming 101 (cont) Introduction (aka Mixin or inter-type declaration) advice that provides additional functionality to a class e.g. methods or member variables can be added via an Introduction to the “interface” of a class

26. Aspect Oriented Programming 101 (cont) Weaving the process by which specific join points are identified by pointcuts and advice is injected into bytecode

27. How AOP Works • Compile-time a specialized compiler is used to weave the source code and advice together to produce bytecode • Post-compile (aka binary weaving) weaves advice into existing class files modifying them as necessary • Load-time a classloader performs binary weaving as the class files are loaded into the JVM or a Java agent is used via JVMTI (JVM Tool Interface) • Run-time classes that are already loaded by the JVM are woven with aspects via JPDA (Java Platform Debugger Architecture) to replace method bodies. Dynamic proxies can also be used.

28. Load-Time Weaving We’re going to focus on load-time weaving (LTW) • Advantange (over compile-time or post-compile): does not modify classes on disk • Disadvantage: increases application startup time

29. Java AOP • AspectJ (http://www.eclipse.org/aspectj/): Excellent all-around solution. • dynaop (https://dynaop.dev.java.net/) Bob Lee’s stripped down AOP: Requires proxy creation in code (i.e. not 100% transparent). • CGLIB (http://cglib.sourceforge.net/): a roll-your-own solution using proxies. Requires proxy creation in code (i.e. not 100% transparent). For load-time weaving a custom class loader would be needed. • BCEL (http://jakarta.apache.org/bcel/): roll-your-own solution requiring knowledge of Java byte code to write advice. For load-time weaving a custom class loader would be needed. • Javassist (http://www.csg.is.titech.ac.jp/~chiba/javassist/): A large step above CGLIB / BCEL in that it includes a stripped down compiler and modifies the advised class on the fly. Load-time weaving is supported. • Spring AOP (http://www.springframework.org/docs/reference/aop.html): Limited to method-based joinpoints. If using the @AspectJ annotations then it effectively becomes AspectJ otherwise standard Spring-style XML configuration. JDK dynamic proxies or CGLIB for implementation. If you’re already using Spring and only need to advise coarse-grained objects (i.e. the objects that you’re IoC’ing) and method-based jointpoints are suitable then this is a good solution. • JMangler (http://roots.iai.uni-bonn.de/research/jmangler/): A step above BCEL / CGLIB but a step below Javassist. Requires knowledge of Java byte code (it uses the BCEL API for advice). Load-time weaving is supported. • JBoss-AOP (http://labs.jboss.com/portal/jbossaop/): Effectively on par with AspectJ. If you’re using JBoss then this would be the best solution. Even has support in their IDE.

30. Questions?

31. AspectJ • AspectJ http://www.eclipse.org/aspectj/ • Documentation http://www.eclipse.org/aspectj/doc/released/ • AspectJ FAQ http://www.eclipse.org/aspectj/doc/next/faq.html • AspectJ Development Environment Guide http://www.eclipse.org/aspectj/doc/next/devguide/index.html • AspectJ Programming Guide http://www.eclipse.org/aspectj/doc/released/progguide/index.html • AspectJ Quick Reference http://www.eclipse.org/aspectj/doc/released/quick5.pdf

32. Load-Time Weaving AspectJ Load-time Weaving can be accomplished via [13]: • -javaagent:pathto/aspectjweaver.jar • aj.bat • Custom classloader (WeavingURLClassLoader) Both require a META-INF/aop.xml to be located in the classpath

33. AspectJ Examples There are three primary classes in the example: • Main • Creates and calls the other classes • ReadFile • Accepts a File on construction • displayExistence() displays if the file exists • byte[] getBytesFromFile() returns the contents of the file as a byte array • TemporaryFile • readWriteTemporaryFile() • Creates a temp file • Writes data into it • Reads contents line-by-line • Reads contents byte-by-byte • Deletes the temp file

34. AspectJ Pros • Quite mature • Integration with IDEs (e.g. AJDT) • Responsive forums • Does 95% of what you need

35. AspectJ Cons • Expect to burn a lot of time figuring out the difference between call and execute, within and withincode, etc. • Can add quite a bit of time to incremental compiles (Also see bugs #146071 and #145438) • It’s very easy to confuse the IDE • No access to local variables • You have to be careful about other errors in your application. A compile error in some completely unrelated file may cause errors in aspects. • Brittle with respect to refactoring especially in the case of load-time weaving. If using AJDT with compile-time weaving then refactorings apply across aspects.

36. Javassist • Javassist http://www.csg.is.titech.ac.jp/~chiba/javassist/ • Javadoc http://www.csg.is.titech.ac.jp/~chiba/javassist/html/index.html • Tutorial http://www.csg.is.titech.ac.jp/~chiba/javassist/tutorial/tutorial.html

37. Javassist Example

38. Javassist Pros • Provides a relatively simple interface to do fault injection at the level of the bytecode • Provides a limited compiler to take away the need to work directly in bytecode (e.g. BCEL)

39. Javassist Cons • Limited compiler • There’s much that you need to roll in order to use it • Confusing code fragment language

40. Future Work • Simplify fault injection + JUnit e.g. run a set of test cases with different faults • Library of faults for java.* and javax.* • Profiler for java.* and javax.* faults • Encourage fault libraries and profilers for common 3rd party libs

41. Questions?

42. References [1] Harbulot, B. & Gurd, J. (2005). A join point for loops in AspectJ. http://www.cs.iastate.edu/~leavens/FOAL/papers-2005/harbulot-gurd.pdf [2] http://www.eclipse.org/aspectj/doc/released/faq.html#q:joinpoints [3] http://www.aspectprogrammer.org/blogs/adrian/2004/06/go_from_stateme.html [4] Ossher, H. & Tarr, P. Operation-Level Composition: A Case in (Join) Point http://www.ccs.neu.edu/research/demeter/SOP/Tarr.pdf [5] Hilsdale, E. & Hugunin, J. Advice Weaving in AspectJ http://hugunin.net/papers/aosd-2004-cameraReady.pdf [6] http://en.wikipedia.org/wiki/Pointcut [7] http://www.eclipse.org/aspectj/doc/released/faq.html#q:advice [8] Hsueh, M, Tasi, T, & Iyer, R. Fault InjectionTechniques and Tools. IEEE CS Press, 1997, pp. 75-82 [9] Clark, J. & Pradhan, D. Fault Injection. IEEE CS Press, 1995, pp. 47-56 [10] http://en.wikipedia.org/wiki/Fault_%28technology%29 [11] http://blogs.msdn.com/bwill/archive/2003/09/23/51169.aspx [12] Espak, M. Improving Efficiency by Weaving at Run-time http://www.old.netobjectdays.org/pdf/03/papers/ws-yrw/415.pdf [13] http://www.eclipse.org/aspectj/doc/released/devguide/ltw-configuration.html [14] http://en.wikipedia.org/wiki/Join_point

43. Thank You Rob Grzywinski http://www.realityinteractive.com/rgrzywinski/ rgrzywinski@yahoo.com