Recovering Configuration Information for Configurable Systems via Symbolic Execution

1. Recovering ConfigurationInformation for Configurable Systems via SymbolicExecution Justin Gorham, Xusheng Xiao, Tao Xie North Carolina State University jwgorham,xxiao,txie@ncsu.edu

2. Modern Software Systems • Designed to be highly customizable • Extensibility • Flexibility • Portability

3. Configurable Systems • A software system with mechanisms in place for implementing preplanned variations in system behavior • Characterized by configuration options • Each configuration option has a configuration domain

4. Formal Definition • Configuration option – A label that represents a dimension of a system that can be set • Configuration domain – The set of possible values that can be selected for a configuration option • ‘configuration’: - a mapping of configuration options to a subset of their respective domains to create an instantiation of a configurable system

5. Example: Microsoft Word option

6. Example: Microsoft Word option domain

7. Configuration Binding Time • Early Feature Binding • Software Product Lines • Boeing: Bold Stroke Avionics Software Family • Nokia: Mobile Phones • Toshiba: Software Product Line for Electric Power Generation Plant Monitoring and Control • Dynamically Reconfigurable Systems • NASA’s Deep Space 1 Remote Agent software • User Configurable Systems

8. Runtime User Configurable System • Configuration options are assigned at runtime • Obtain values typically through configuration files and command lines • Options typically bind at startup time or at an early stage of runtime

9. Example: (win)grep • Command Line Options • /h|/H - printing a usage help explanatory text; • /c - print a count of matching lines for each input file; • /i - ignore case in pattern; • /l - print just files (scanning will stop on first match); • /n - prefix each line of output with line number; • /r - recursive search in subdirectories; • /E:reg_exp - the Regular Expression used as search pattern. • /F:files - the list of input files. The files can be separated by commas as in

10. Example: JPF • Uses a dictionary object to parse a hierarchical set of property files • Site Properties • Project Properties • Application Properties • Primarily uses a <key> = <value> style of configuration

11. JPF: Site Properties • site.properties file is machine specific and not part of any JPF project • Each project is listed as a <name>=<directory> pair

12. JPF: Site Properties • site.properties file is machine specific and not part of any JPF project • Each project is listed as a <name>=<directory> pair jpf-core = ${user.home}/projects/jpf/jpf-core jpf-shell = ${user.home}/projects/jpf/jpf-shell jpf-awt = ${user.home}/projects/jpf/jpf-awt ... extensions=${jpf-core},${jpf-shell}

13. JPF: Project Properties • Each JPF project contains a jpf.properties file in its root directory • defines the paths that need to be set for the component to work properly

14. JPF: Project Properties • <project-name>.native_classpath • <project-name>.classpath • <project-name>.test_classpath • <project-name>.sourcepath

15. JPF: Project Properties • <project-name>.native_classpath • <project-name>.classpath • <project-name>.test_classpath • <project-name>.sourcepath jpf-aprop = ${config_path} #--- path specifications jpf-aprop.native_classpath = build/jpf-aprop.jar;lib/antlr-runtime-3.1.3.jar jpf-aprop.classpath = build/examples jpf-aprop.test_classpath = build/tests jpf-aprop.sourcepath = src/examples #--- other project specific settings listener.autoload=${listener.autoload},javax.annotation.Nonnull,... listener.javax.annotation.Nonnull=gov.nasa.jpf.aprop.listener.NonnullChecker ...

16. JPF: Project Properties • <project-name>.native_classpath • <project-name>.classpath • <project-name>.test_classpath • <project-name>.sourcepath jpf-aprop = ${config_path} #--- path specifications jpf-aprop.native_classpath = build/jpf-aprop.jar;lib/antlr-runtime-3.1.3.jar jpf-aprop.classpath = build/examples jpf-aprop.test_classpath = build/tests jpf-aprop.sourcepath = src/examples #--- other project specific settings listener.autoload=${listener.autoload},javax.annotation.Nonnull,... listener.javax.annotation.Nonnull=gov.nasa.jpf.aprop.listener.NonnullChecker ...

17. JPF: Application Properties • Used to tell JPF what main class it should start to execute • Stored in *.jpf properties files that are part of the test projects • target setting defines the main class of SUT • Also define target_args along with other JPF properties that define how the application is to be checked

18. JPF: Application Properties #--- dependencies on other JPF projects @using = jpf-awt @using = jpf-shell #--- what JPF should run target = RobotManager #--- other stuff that defines how to run JPF listener+=,.listener.OverlappingMethodAnalyzer shell=.shell.basicshell.BasicShell awt.script=${config_path}/RobotManager-thread.es cg.enumerate_random=true ... • Used to tell JPF what main class it should start to execute • Stored in *.jpf properties files that are part of the test projects • target setting defines the main class of SUT • Also define target_args along with other JPF properties that define how the application is to be checked

19. Scenario • Developer/Testers • QA to validate configurations • Software Configuration Space Explosion • End Users

20. Software Configuration Space Explosion • Example • 20 configuration options • The domain for each option is either ‘true’ or ‘false’ • 220 = 1,048,576 possible configurations • Add another option, configuration space doubles • Infeasible to test a large configuration space

21. Motivation • Extract configuration information for a runtime user configurable system • How to extract available configuration options • How to extract constraints among options and their values • domain of a configuration option • relationships among options and/or their values • How to extract runtime behavior of configurations • Fault due to interaction

22. Extracting Configuration Information • Manually search through source code • Expensive, error prone… • Static analysis • Cannot capture runtime behavior of configurtions • Combinatorial Testing • Black Box, Scalability Issues • Symbolic Execution

23. Symbolic Execution Choose next path • Code to generate inputs for: Solve Execute&Monitor void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Negated condition a==null F T a.Length>0 T F a[0]==123… Done: There is no path left. F T Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890 Data null {} {0} {123…} Thanks to Dr. Tao Xie for reusing his slide

24. SE with Configurable Systems • Using Symbolic Evaluation to Understand Behavior in Configurable Software Systems • Reisner, E; Song, C; Ma, K; Foster, J; Porter A Figures on this page from “Using Symbolic Evaluation to Understand Behavior in Configurable Software Systems “

25. SE with Configurable Systems Figures on this page from “Using Symbolic Evaluation to Understand Behavior in Configurable Software Systems “

26. SE with Configurable Systems Figures on this page from “Using Symbolic Evaluation to Understand Behavior in Configurable Software Systems “

27. SE with Configurable Systems • Required to know configuration options in advance • Program inputs may interact with configuration options

28. Parameterized Unit Test

29. Parameterized Configuration Mechanism

30. Proposed Solution Inputs Configuration Mechanism Unit Tests Code Under Test Option Value Output: Test Suite with concrete inputs

31. Proposed Solution Parameterized Configuration Mechanism PUTs Code Under Test Output: Test Suite with concrete inputs and assumptions on configuration options

32. Our Approach

33. Implementation

34. Challenges • Object Creation and Data Transformation • Static Configuration Options

35. Future Directions

36. END • Questions