140 likes | 278 Views
CS527 Topics in Software Engineering (Software Testing and Analysis). Darko Marinov September 13, 2011. Schedule. First few lectures to help you select projects Shared memory: CHESS, IMUnit , CAPP Message passing: Setac Comment analysis: iComment
E N D
CS527Topics in Software Engineering(Software Testing and Analysis) Darko Marinov September 13, 2011
Schedule • First few lectures to help you select projects • Shared memory: CHESS, IMUnit, CAPP • Message passing: Setac • Comment analysis: iComment • Today: Regression testing, note: survey paper • Sep 15: Model-based testing • Sep 20: Mock-based testing? • Sep 22: ? • Sep 27-29: Project proposals due • From Oct 4: Your presentations
Projects • By far the most important part of the course • Meet or email me and/or Shin Hwei • Our discussion can help you select project/paper • You need to explore some topic in depth • “I don't know anything, but I do know that everything is interesting if you go into it deeply enough” Richard Feymanquote • Mostly for I2CS students: Can use the technology from your work or learn some new • Need to be able to show us some results • Preferably work in groups of 2-3 students
Paper Today • Regression Testing Minimization, Selection and Prioritization: A Surveyby Shin Yoo and Mark Harman (STVR 2010) • Survey paper • Presents a broad summary of an area • Regression testing • Test Suite Minimization • Test Case Selection • Test Case Prioritization • Follows: brief overview, projects, questions
… tn t2 t3 tn-1 t1 Minimization • Select a subset of T to cover all requirements r1 x x x r2 x x r3 x x … rk x x
… tn t2 t3 tn-1 t1 in-1 in i2 i3 … tn-1 t1 t3 δ code 1 code 2 selection prioritization Selection and Prioritization i1
Future Directions Listed • Orchestrating regression testing techniques with test data generation • Multi-objective regression testing • Problem of test oracle and its cost • Consideration of other domains • Non-functional testing • Tool support • Little text • Big issue in practice • Often ignored in research
Questions for Discussion (1) • Any actual industrial pieces of software out there that can [do this]? [KN] • What is used at Microsoft/Google/Apple? [SO] • Should we consider [sequential] criteria when testing multithread programs? [YL] • Why did the text based approach work so well with simple diffs? [SB] • In which situations should we use change-aware techniques? [SO] • How [to] ensure that a discarded test case … would not detect faults? [AD]
Questions for Discussion (2) • … a correlation between use of automated refactorings and test case selection? [CM] • … worth it compared to just buying more hardware and running tests in parallel? [DeG] • …the place is of survey papers in CS? [DwG] • … could we pick only one of the approaches listed in this survey? [XW] • Which regression test selection approach offers the best scalability/safety ratio? [AK] • … in a multithreaded regression test, what info would be useful to keep (POR)? [FS]
Old Questions for Discussion, 1 • Correlation between fault detection and requirement coverage? [AL] • How is selection affected by refactoring? [MG] • Many techniques are proposed, are there some tools? [MG] • What's the conclusion? Is minimization technique useful? [HY] • What is minimum hitting set problem in terms of suite minimization? [JN] • Regression testing for porting systems to different OS? (Porting testing?) [DM]
Old Questions for Discussion, 2 • What to do for multithreaded code? [QL] • Can minimization and selection produce same results? [ST] • Are redundant test cases a real problem? • Doesn’t the need for minimization mean that we generated counter-productive tests? • Should we minimize if that can miss faults? • How much redundancy to have in testing? • What are best practices for regression testing of GUI's?
Old Sample Project Ideas (1) • [AA], [AS], [CR], [JC], [MK] • Writing a test suite for faults vs. to meet requirements [AL] • Extend a tool X to generate tests only for modified parts of the code [MG] • Compare existing approaches for test selection [MG] • Perform a large study on all apps from some open-source code base [HY] • Extend or evaluate RETEST [DM]
Old Sample Project Ideas (2) • Develop system based on regression testing minimization techniques to instead prepare sets of non-regression, related test suites that have overlaps in functionality [JN] • Integrate test selection/prioritization with version control [QL] • Combine several approach (e.g., integer programming and data-flow analysis) [ST] • A project on test case minimization would be challenging… (types of experiments in CS/SE)
Old Sample Project Ideas (3) • Study number of redundant or obsolete test cases in open source (how about disabled?) • Implement and compare some approchaes • Extend the CDG-based technique • Generate test cases that target changes • Integrate some of these strategies with capture-replay testing tools • Apply these techniques on web applications • Apply in other new domains • Test power consumption of embedded systems