Poirot – A Concurrency Sleuth

Poirot – A Concurrency Sleuth Shaz Qadeer Research in Software Engineering Microsoft Research

Concurrent programming is difficult

IO_REQUEST_PACKET *irp; irp->Cancel = FALSE; irp->CancelRoutine = NULL; Normal Cancellation … if (irp->Cancel) { IoCompleteIrp(irp); } else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp); } … … irp->Cancel = TRUE; fn = IoSetCancelRoutine(Irp, NULL); if (fn) { fn(irp); } … void CancelRoutine(IRP *irp) { IoCompleteIrp(irp); }

IO_REQUEST_PACKET *irp; irp->Cancel = FALSE; irp->CancelRoutine = NULL; Normal Cancellation … if (irp->Cancel) { IoCompleteIrp(irp); } else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp); } … … irp->Cancel = TRUE; fn = IoSetCancelRoutine(Irp, NULL); if (fn) { fn(irp); } … void CancelRoutine(IRP *irp) { IoCompleteIrp(irp); } Fatal error!

Concurrent programming is difficult • Multiple loci of control resulting in non-local control flow • Code difficult to understand and review

What about verification? • Assertion-based modular reasoning becomes complicated due to non-local interactions • Floyd-Hoare morphs into Owicki-Gries • Even with simple (finite) abstractions, the presence of concurrency makes the analysis computationally very expensive

What about testing? Thread 1 Thread n x = 1; … … … … … x= k; x = 1; … … … … … x= k; • Scheduling nondeterminism • Uncontrollable • Unobservable • Exponential … Number of executions = O( nnk) Exponential in both n and k

Concurrency is important • More than ever before • Increasing importance of communicating systems • networked devices • cyber-physical systems • Distributed programs running on the cloud • EC2, Azure, AppEngine, … • Parallel programs running on multicores and GPUs • TBB, TPL, CUDA, AMP, …

Concurrency testing with CHESS • Deterministic scheduling • make scheduling choices observable and controllable • Search prioritization • combating the combinatorial explosion of possible schedules

Deterministic scheduling Program Tester Provides a Test Scenario While(not done){ TestScenario() } CHESS TestScenario(){ … } CHESS runs the scenario in a loop • Each run is a different interleaving • Each run is repeatable Win32 API Kernel: Threads, Scheduler, Synchronization Objects

Search prioritization (I) • Given p ≥ 0, generate all schedules with up to p preemptions • Pseudo-polynomial number of schedules • polynomial in preemption bound and schedule points • exponential in number of threads • Many bugs with fewer than 2 preemptions • Simple error traces for easier debugging

Search prioritization (II) • Given p ≥ 0 and deterministic schedulers S0, …, Sp-1, schedule according to S0, …, Sp-1 in sequence moving from one to next nondeterministically • e.g., round-robin non-preemptive scheduling with p different round-robin orders • Polynomial number of schedules • Testers can innovate by designing domain-specific deterministic schedulers

CHESS is available • Used internally by Microsoft product groups and externally by Microsoft customers • Binary and source code available at: • http://chesstool.codeplex.com

Limitations of CHESS • Exposing and gaining control of scheduling choices is difficult • most implementation effort and user frustration due to this problem • Testing components that interact extensively with the environment is difficult • Input coverage is not addressed

Static program exploration with Poirot • Symbolic instead of concrete execution • C: Source code for software component • E: Model for environment and scheduler • Explore behaviors of C+E • for all symbolic inputs • for all scheduling choices

Disk Demo: Asynchronous File I/O Request queue In-memory cache tail head cache cacheSize DiskReader(…) { } DiskReader(…) { } AsyncRead(…) { }

Poirot architecture C  Boogie Concurrent C Program Trace Viewer .NET Boogie Corral Concurrent .NET Program Concurrent Boogie Program Coverage Report

Searching with Corral Abstraction Sequentialization Stratified Search Error Trace Concurrent Boogie Program Sequential Boogie Program Concurrent Boogie Program Refinement Coverage Report

Abstraction • Set of global variables G • Set of tracked variables T • Drop writes to variables in G-T • Replace reads to variables in G-T with nondeterministic values

Refinement • Path p • feasible if only variables in T are tracked • infeasible if all variables in G are tracked • Expand tracked set T to U such that p infeasible while tracking only variables in U • Naïve algorithm: linear scan of G-T • New divide-and-conquer algorithm • best case log(|G-T|) • worst case 2*|G-T|

Sequentialization (I) • Given a concurrent program P, construct a sequential program Q such that Q  P • Drop each occurrence of async-call • Convert each occurrence of async-call to call

Sequentialization(II) • Given a concurrent program P, construct a family of programs Qi such that • Q0 Q1  Q2  …  P • iQi= P • Even better if interesting behaviors of P manifest in Qi for low values of i

Context-bounding • Under-approximation parameterized by K ≥ 0 • executions in which each thread gets at most K contexts to execute • As K  , we get all behaviors • Can we create sequentializations for context-bounding?

Sequentializing context switches s2 (s2, l2) l2 Shared Memory T1 T2 Local Memory Local Memory Execution: T1 T2 T1 T2 T1 T2 T1 (s1, l1) T1

Guess and verify • Make copies of global variables • Source-to-source translation • linear in program size and K • Generalizes to dynamically-created threads T1 (s2, l2) (s3, l2) (s1, l1) T1 T2 Guess the effect of T2 Verify the guess

Stratified search Convert loops to recursive calls Call tree given recursion bound r assert no bug main T L assert no bug Summaries(L) VC(p) VC(T) …

Poirot status • Medium-sized C programs • up to 20K low-level systems code • reports precise traces at scale • Small .NET programs • bytecode to Boogie translator in progress • Try: http://www.rise4fun.com/Poirot • Downloadavailable

Why bounded search? Data: Boolean, Integers, Arrays Control: Sequencing, Choice, Iteration, Call, Async-Call Sequencing Choice Iteration Call Async-call Sequencing Choice Iteration Call Async-call + bound Sequencing Choice NP-complete Decidable PSPACE-hard Undecidable Advances in SAT/SMT-solvers have made this problem tractable HAVOC verifier deployed for security analysis in Windows/IE Rationale: It is better to fail at the simpler problem!

Poirot collaborators Akash Lal, MSR Bangalore Shuvendu Lahiri, MSR Redmond

Questions

Poirot – A Concurrency Sleuth