s uggestions by Dr Andrew Brooks, Heller Hall 334 d.umn/~abrooks/ abrooks@d.umn

1. MS thesis/project ideasfor MS students in the Department of Computer Scienceat the University of Minnesota Duluth suggestions by Dr Andrew Brooks, Heller Hall 334http://www.d.umn.edu/~abrooks/ abrooks@d.umn.edu Dr Andrew Brooks - MS thesis/project ideas

2. Some presentation material drawn directly from article(s). background research article Kanewala, U.; Bieman, J.M., "Techniques for testing scientific programs without an oracle," Software Engineering for Computational Science and Engineering (SE-CSE), 2013 5th International Workshop on , pp.48-57, 2013doi: 10.1109/SECSE.2013.6615099 http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6615099 Dr Andrew Brooks - MS thesis/project ideas

3. Some presentation material drawn directly from article(s). A gentle introduction to oracles: A Taxonomy for Test Oracles, Douglas Hoffman, Software Quality Week (QW’98), 1998, 8pp. http://www.softwarequalitymethods.com/Papers/OracleTax.pdf A gentle introduction to assertions: Chapter 6 Assertion facility, inJDK 1.4 Tutorial, Greg Travis, Manning Publications Company, 2002. http://www.oracle.com/technetwork/articles/javase/javapch06.pdf Dr Andrew Brooks - MS thesis/project ideas

4. the difficulty of testing software How do we calculate the expected results?(and know that they are correct) • The oracle is the mechanism used to generate expected results. Dr Andrew Brooks - MS thesis/project ideas

5. Oracle methods used by Hoffman • A human calculates the expected results. • A separate program implementing the same algorithm is used to calculate the expected results. • A simulation is used to calculate the expected results. • A hardware simulator is used to calculate the expected results. • An earlier version of the software is used to calculate the expected results. • The same version of the software on a different hardware platform is used to calculate the expected results. • A check is made on the consistency of generated values and end points. • A sample of values can be checked against independently generated expected results from existing commercial or open-source software. Dr Andrew Brooks - MS thesis/project ideas

6. the difficulty of testing scientific software “The existence of an oracle is often assumed in software testing.” “But in many situations, especially for scientific programs, oracles do not exist or they are too hard to implement.” Kanewala and Bieman, 2013 Scientists often write programs to discover answers that are previously unknown. Dr Andrew Brooks - MS thesis/project ideas

7. Metamorphic Testing(a way around the oracle problem) • A metamorphic relation specifies how a particular change to the input of the program would change the output. failure Dr Andrew Brooks - MS thesis/project ideas

8. examples of metamorphic relations Dr Andrew Brooks - MS thesis/project ideas

9. Is it possible to replicate anexperiment? • A report should contain enough detail so that another researcher can repeat the experiment and obtain similar results with similar conclusions. Dr Andrew Brooks - MS thesis/project ideas

10. MS thesis/project idea 1. • First begin by replicating in part or in full one of the previous experiments on metamorphic testing. • References to several emprical studies are given by Kanewala and Bieman. Dr Andrew Brooks - MS thesis/project ideas

11. MS thesis/project idea 1. • Then tackle one of the problems described by Kanewala and Bieman. • automatic detection of metamorphic relations • some recent work has been done by Kanewala and Bieman • identifying the most effective metamorphic relations • identifying the most effective combinations of metamorphic relations • understanding why some program mutations could not be killed using metamorphic testing deliberately introduced mistakes Dr Andrew Brooks - MS thesis/project ideas

12. Assertion Testing(a way around the oracle problem) • Assertions are embedded into source code and evaluated when a test case is executed. An assertion is a predicate placed in a program to indicate that the developer thinks that the predicate is always true at that place. Dr Andrew Brooks - MS thesis/project ideas

13. simple Java example package asserting; publicclass First { publicstaticvoid main(String[] args) { int j = 10; assert j>10; } } What happens when you run this program? console Exception in thread "main" java.lang.AssertionError at asserting.First.main(First.java:7) Dr Andrew Brooks - MS thesis/project ideas

14. Assertion basics(Greg Travis chapter) • Preconditions should be true at the start of a method. • Postconditionsshould be true at the end of a method. • Invariants should always be true. • A speed should not be faster than light speed... Dr Andrew Brooks - MS thesis/project ideas

15. MS thesis/project idea 2. • First begin by replicating in part or in full one of the previous experiments on assertion testing. • References to several empirical studies are given by Kanewala and Bieman. Dr Andrew Brooks - MS thesis/project ideas

16. MS thesis/project idea 2. • Then tackle one of the problems described by Kanewala and Bieman. • automatic invariant detection can yield spurious invariants • can ways be found to reduce or eliminate these? • the effectiveness of assertion checking seems all or nothing depending on the method • can we understand why? • many more experiments are needed • working with specifications to identify assertions • rewriting program comments as assertions Dr Andrew Brooks - MS thesis/project ideas