1 / 12

CS527 Topics in Software Engineering (Software Testing and Analysis)

CS527 Topics in Software Engineering (Software Testing and Analysis). Darko Marinov September 14, 2010. Schedule. First few lectures to help you select projects Testing: ReAssert, UDITA, Pex, Randoop Model checking: JPF Sep 14: CHESS (multithreaded testing)

luna
Download Presentation

CS527 Topics in Software Engineering (Software Testing and Analysis)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CS527Topics in Software Engineering(Software Testing and Analysis) Darko Marinov September 14, 2010

  2. Schedule • First few lectures to help you select projects • Testing: ReAssert, UDITA, Pex, Randoop • Model checking: JPF • Sep 14: CHESS (multithreaded testing) • Sep 16: Regression testing, note: survey paper • Sep 21: MC (static analysis) • Sep 23: REST (GUI testing) • Sep 28: Analysis of code comments? • Testing distributed or real-time systems? • From Oct 7: Your presentations

  3. To-Dos • Email me (and Sandro) for project proposals • Project is the most important part of the course • Our discussion can help you select a project • Our discussion will help you select a paper • By Sep 23https://agora.cs.illinois.edu/display/cs527fa10/People+in+the+Course • Sign up for a slot to present a paper(even if you don’t know which paper yet) • List programming language(s) that you use • If you don’t sign up, you’ll get a random slot • If you sign up for early slots, you get bonus

  4. Paper Today • Iterative Context Bounding for Systematic Testing of Multithreaded Programsby Madan Musuvathi and Shaz Qadeer(PLDI 2007) • CHESS web site • http://research.microsoft.com/en-us/projects/chess/default.aspx • A free version available for academic use only • Slides from a tutorial by Sebastian Burckhardt, Madan Musuvathi, and Shaz Qadeer • Not only about this specific paper

  5. Paper Overview • Problem • Testing multithreaded code is hard • Trying all thread interleavings doesn’t scale • Solution • Bound the number of preemptive context switches • Tool for Windows: CHESS (stateless, reexecutes) • Evaluation also for ZING (stateful) • Evaluation • 6 programs of various sizes • Found 14 previously unknown bugs

  6. Questions for Discussion (1) • Missing by 12:30: [CR], [AS] • Can POR improve CHESS & how much? [JC] • How often CHESS goes to infinite loop in state space graph? [MG] • Does this paper assume correct use of synchronization variables? [AA] • What is the difference between observable and non-observable race? [AA] • Better to limit rather than test threading? [HY] • How to automatically close open systems? [MK]

  7. Questions for Discussion (2) • How to know the size of state space? [AL] • Any advantages inserting (removing??) a scheduling point between two syncs? [DM] • Why not bound nonpreempting switches? [JN] • Can iterative context bounding be applied to message passing programs? [ST] • What’s used in industry, stress testing? [QL] • Soundness vs. completeness • False positives vs. false negatives • Precision vs. recall • Combine CHESS and Pex (or X and Y)?

  8. Questions for Discussion (3) • Why not report all accessed to shared data not protected by a synchronization variable? • How are data races identified? • How are other bugs identified? Assertions? • What other heuristics can one use? POR? • If you analyze models (ZING), how do you know there is bug in actual code? • How to test a complex multithreaded algorithm in mostly sequential code? Extract it out?

  9. Questions for Discussion (4) • How does it reduce time to polynomial? • What about dynamically changing number of threads in programs? Or dynamically created? • Can CHESS handle locks, mutexes, semaphores, etc.? • How to identify preemption or nonpreemption in Java code? • What are limits of this approach? Is it LOC, 10k, 100k, 1M? Is it # of threads, preemptions? • Can it be combined with DFS?

  10. Old Questions for Discussion (1) • Which part of JPF/CHESS can be incorrect? • Can a compiler be model checked? • How difficult is it to set environments? • How to “close a system”? • Comparison of testing, model checking, and static analysis • When is model checking applicable? • What kind of bugs it can find? • Why work with language X not Y?

  11. Old Questions for Discussion (2) • How well does it scale? • What about memory requirements? • What about time requirements? • How can I apply this on my code? • What’s recent work in this area? • A lot of work in many group, including mine • How to use the tool in practice? • How does CHESS/JPF compare states? • How good are heuristics?

  12. Sample Project Ideas • Apply CHESS on some software • Did you read the optional reading/report on JPF? • Compare with more tools and techniques • Consider other heuristics for exploration • Message-passing programs? Evolving code? • Automate creation of environments • Extend JPF implementation of CHESS • Contributed by Igor Andjelkovic, Steven Lauterburg, and Mirko Stojmenovic • Reimplement in another model checker • Improve debugging of multithreaded code

More Related