1 / 50

http:// www.cs.utah.edu / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF

Practical Formal Verification of MPI and Thread Programs Sarvani Vakkalanka Anh Vo* Michael DeLisi Sriram Aananthakrishnan Alan Humphrey Christopher Derrick Yu Yang Ganesh Gopalakrishnan* Robert M. Kirby* * = presenters School of Computing, University of Utah, Salt Lake City, UT 84112, USA.

floria
Download Presentation

http:// www.cs.utah.edu / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Practical Formal Verification ofMPI and Thread Programs SarvaniVakkalankaAnh Vo*Michael DeLisiSriramAananthakrishnanAlan HumphreyChristopher DerrickYu YangGanesh Gopalakrishnan*Robert M. Kirby** = presentersSchool of Computing, University of Utah,Salt Lake City, UT 84112, USA http:// www.cs.utah.edu / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408, SRC tasks TJ 1847.001 and TJ 1993, and Microsoft

  2. Additional Acknowledgements for this tutorial • Other students involved:SalmanPervez, Robert Palmer, Guodong Li, GeofSawaya, Subodh Sharma, GrzegorzSzubzda, Jason Williams, Simone Atzeni, Wei-Fan Chiang • External Collaborators:ANL / UIUC : Rajeev Thakur, Bill Gropp, Rusty Lusk IBM : Beth Tibbits LLNL : Bronis de Supinski, Martin Schulz, Dan Quinlan Microsoft : Robert Palmer, Dennis Crain, ShahrokhMortazavi

  3. Overview of Formal Verification, especially Dynamic Verification Overview of MPI Demo of our tool ISP Architecture of ISP Presentation of Any_src_can_deadlock (from Umpire test suite) Our algorithm POE (Partial Order avoiding Elusive interleavings) Presentation of POE-Illustration Present details of POE-Illustration: ISP’s Eclipse framework and GUI Boot into LiveDVD and practice on POE-Illustration 9:00 to 10:30

  4. Coffee Break IMPORTANT : Please give feedback before it is too late Too fast ? Too slow ? Just right !! ? Assuming a lot ? Other suggestions ? We will TRY to take into account these valuable suggestions! 10:30 to 11:00

  5. Illustration of Resource Dependent Deadlocks, and Detection Illustration of Resource Leak, and Detection Iprobe behavior, and illustration using GUI Assertion Violation in Red/Blue Problem Audience Participation in Above Exercises ISP’s Theory : MPI Happens-before Also called “matches before, completes before” in the tool 11:00 to 12:00

  6. Example of Matrix Multiplication: Four Variations Analysis of these variations using ISP, with Audience Participation 12:00 to 12:30

  7. Assisted Problem Solving by Audience 14:00 to 15:00

  8. Overview of Dynamic Verification of Shared Memory Thread Programs 15:00 to 15:30

  9. Dynamic Verification of Thread Programs using Inspect Concluding Remarks 16:00 to 17:30

  10. Overview of Formal Verification methodsfor Validating Concurrent Systems About 30 minutes – by Ganesh

  11. Problem: Engineering Reliable Concurrent Systems

  12. For many important reasons, we advocate Dynamic Formal Verification methods • Designers require a push-button debugger-like interface • But one that offers coverage guarantees and deeper insights

  13. For many important reasons, we advocate Dynamic Formal Verification methods • Testing methods suffer from bug omissions Bug Omissions X

  14. For many important reasons, we advocate Dynamic Formal Verification methods • Testing methods suffer from bug omissions • Static analysis methods generate many false alarms Bug Omissions False Alarms X X

  15. For many important reasons, we advocate Dynamic Formal Verification methods • Testing methods suffer from bug omissions • Static analysis methods generate many false alarms • Model based verification requires tedious model building Bug Omissions False Alarms Tedious Modeling X X X

  16. For many important reasons, we advocate Dynamic Formal Verification methods • Testing methods suffer from bug omissions • Static analysis methods generate many false alarms • Model based verification requires tedious model building • Dynamic verification methods are ideal for designers! • No omissions • No false alarms • No need for modeling Bug Omissions False Alarms Tedious Modeling X X X √

  17. Growing Importance of Dynamic Verification Code written using mature libraries (MPI, OpenMP, PThreads, …) API calls made from real programming languages (C, Fortran, C++) Dynamic Verification Methods are going to be very important for real engineers ! (static analysis and model based verification can play important supportive roles) Runtime semantics determined by realistic compilers and runtimes

  18. A Brief Survey of Dynamic Verification tools • Verisoft Project • Used for telephone switch software verification in Bell Labs • Available

  19. A Brief Survey of Dynamic Verification tools • Verisoft Project • Used for telephone switch software verification in Bell Labs • Available • The Java Pathfinder Project • Developed at NASA for Java Control Software • On SourceForge

  20. A Brief Survey of Dynamic Verification tools • Verisoft Project • Used for telephone switch software verification in Bell Labs • Available • The Java Pathfinder Project • Developed at NASA for Java Control Software • On SourceForge • The CHESS Project • Microsoft Research ; available for academic institutions • In use within Microsoft product groups, and used by academics

  21. A Brief Survey of Dynamic Verification tools • Verisoft Project • Used for telephone switch software verification in Bell Labs • Available • The Java Pathfinder Project • Developed at NASA for Java Control Software • On SourceForge • The CHESS Project • Microsoft Research ; available for academic institutions • In use within Microsoft product groups, and used by academics • Inspect : Our fairly unique Pthread / C verifier • Discussed in this tutorial

  22. A Brief Survey of Dynamic Verification tools • Verisoft Project • Used for telephone switch software verification in Bell Labs • Available • The Java Pathfinder Project • Developed at NASA for Java Control Software • On SourceForge • The CHESS Project • Microsoft Research ; available for academic institutions • In use within Microsoft product groups, and used by academics • Inspect : Our fairly unique Pthread / C verifier • Discussed in this tutorial • ISP : Our very unique MPI / C program verifier • Main focus of THIS TUTORIAL !!

  23. Example : How ISP Effects Dynamic Verification • Through PMPI • Runs the code under a verification scheduler • ‘Hijacks’ native scheduler • By interposing a profiler • Exerts its own Interleaving Generation Control • Selective replay, Dynamic Instruction Rewriting • TRIES HARD to generate only RELEVANT interleavings • Only replays around “non-determinism” • Does ‘stateless’ (replay) verification • Restarts from MPI_Init for each new interleaving

  24. Example : How ISP Effects Dynamic Verification • Somehow Instruments the Source / Binary • Through PMPI • Runs the code under a verification scheduler • ‘Hijacks’ native scheduler • By interposing a profiler • Exerts its own Interleaving Generation Control • Selective replay, Dynamic Instruction Rewriting • TRIES HARD to generate only RELEVANT interleavings • Only replays around “non-determinism” • Does ‘stateless’ (replay) verification • Restarts from MPI_Init for each new interleaving

  25. Example : How ISP Effects Dynamic Verification • Somehow Instruments the Source / Binary • Through PMPI at source level • Runs the code under a verification scheduler • ‘Hijacks’ native scheduler • By interposing a profiler • Exerts its own Interleaving Generation Control • Selective replay, Dynamic Instruction Rewriting • TRIES HARD to generate only RELEVANT interleavings • Only replays around “non-determinism” • Does ‘stateless’ (replay) verification • Restarts from MPI_Init for each new interleaving

  26. Example : How ISP Effects Dynamic Verification • Somehow Instruments the Source / Binary • Through PMPI • Runs the code under a verification scheduler • ‘Hijacks’ native scheduler • By interposing a profiler • Exerts its own Interleaving Generation Control • Selective replay, Dynamic Instruction Rewriting • TRIES HARD to generate only RELEVANT interleavings • Only replays around “non-determinism” • Does ‘stateless’ (replay) verification • Restarts from MPI_Init for each new interleaving

  27. Example : How ISP Effects Dynamic Verification • Somehow Instruments the Source / Binary • Through PMPI • Runs the code under a verification scheduler • ‘Hijacks’ MPI Function Calls • By interposing a profiler • Exerts its own Interleaving Generation Control • Selective replay, Dynamic Instruction Rewriting • TRIES HARD to generate only RELEVANT interleavings • Only replays around “non-determinism” • Does ‘stateless’ (replay) verification • Restarts from MPI_Init for each new interleaving

  28. Example : How ISP Effects Dynamic Verification • Somehow Instruments the Source / Binary • Through PMPI • Runs the code under a verification scheduler • ‘Hijacks’ MPI Function Calls • By interposing a profiler • Exerts its own Interleaving Generation Control • Selective replay, Dynamic Instruction Rewriting • TRIES HARD to generate only RELEVANT interleavings • Only replays around “non-determinism” • Does ‘stateless’ (replay) verification • Restarts from MPI_Init for each new interleaving

  29. Example : How ISP Effects Dynamic Verification • Somehow Instruments the Source / Binary • Through PMPI • Runs the code under a verification scheduler • ‘Hijacks’ MPI Function Calls • By interposing a profiler • Exerts its own Interleaving Generation Control • Selective replay, Dynamic Instruction Rewriting • TRIES HARD to generate only RELEVANT interleavings • Only replays around “non-determinism” • Does ‘stateless’ (replay) verification • Restarts from MPI_Init for each new interleaving

  30. Example : How ISP Effects Dynamic Verification • Somehow Instruments the Source / Binary • Through PMPI • Runs the code under a verification scheduler • ‘Hijacks’ MPI Function Calls • By interposing a profiler • Exerts its own Interleaving Generation Control • Selective replay, Dynamic Instruction Rewriting • TRIES HARD to generate only RELEVANT interleavings • Only replays around “non-determinism” • Does ‘stateless’ (replay) verification • Restarts from MPI_Init for each new interleaving

  31. Example : How ISP Effects Dynamic Verification • Somehow Instruments the Source / Binary • Through PMPI • Runs the code under a verification scheduler • ‘Hijacks’ MPI Function Calls • By interposing a profiler • Exerts its own Interleaving Generation Control • Selective replay, Dynamic Instruction Rewriting • TRIES HARD to generate only RELEVANT interleavings • Only replays around “non-determinism” • Does ‘stateless’ (replay) verification • Restarts from MPI_Init for each new interleaving

  32. Example : How ISP Effects Dynamic Verification • Somehow Instruments the Source / Binary • Through PMPI • Runs the code under a verification scheduler • ‘Hijacks’ MPI Function Calls • By interposing a profiler • Exerts its own Interleaving Generation Control • Selective replay, Dynamic Instruction Rewriting • TRIES HARD to generate only RELEVANT interleavings • Only replays around “non-determinism” • Does ‘stateless’ (replay) verification • Restarts from MPI_Init for each new interleaving

  33. Example : How ISP Effects Dynamic Verification • Somehow Instruments the Source / Binary • Through PMPI • Runs the code under a verification scheduler • ‘Hijacks’ MPI Function Calls • By interposing a profiler • Exerts its own Interleaving Generation Control • Selective replay, Dynamic Instruction Rewriting • TRIES HARD to generate only RELEVANT interleavings • Only replays around “non-determinism” • Does ‘stateless’ (replay) verification • Restarts from MPI_Init for each new interleaving

  34. Example : How ISP Effects Dynamic Verification • Somehow Instruments the Source / Binary • Through PMPI • Runs the code under a verification scheduler • ‘Hijacks’ MPI Function Calls • By interposing a profiler • Exerts its own Interleaving Generation Control • Selective replay, Dynamic Instruction Rewriting • TRIES HARD to generate only RELEVANT interleavings • Only replays around “non-determinism” • Does ‘stateless’ (replay) verification • Restarts from MPI_Init for each new interleaving

  35. Sketch of Stateless / Replay Verification Start system In Initial State L0 L0 U0 U0 Red, Green, and Blue moves Belong to different processes Dotted arrow shows some Dependency (e.g., runtime non-determinism) L1 L2 U1 U2 L1 L2 U1 U2

  36. Exponential number of TOTAL Interleavings – most are EQUIVALENT – generate only RELEVANT ones !! P0 P1 P2 P3 P4 TOTAL > 10 Billion Interleavings !!

  37. Exponential number of TOTAL Interleavings – most are EQUIVALENT – generate only RELEVANT ones !! P0 P1 P2 P3 P4 TOTAL > 10 Billion Interleavings !! B1 B2 A These are the only dependent actions E.g. One ANY-SOURCE (wildcard) receive And two of its MATCHING SENDS Point-to-point actions can be issued in ANY order A B1 B2

  38. Exponential number of TOTAL Interleavings – most are EQUIVALENT – generate only RELEVANT ones !! P0 P1 P2 P3 P4 TOTAL > 10 Billion Interleavings !! B1 B2 A Only TWO RELEVANT Interleavings ! These are the only dependent actions E.g. One ANY-SOURCE (wildcard) receive And two of its MATCHING SENDS Point-to-point actions can be issued in ANY order A B1 B2

  39. Workflow of ISP Scheduler that generates ALL RELEVANT schedules (one per partial order) MPI Program Executable Proc1 Proc2 …… Procn Run Interposition Layer MPI Runtime

  40. Hijack Calls, Generate Relevant Interleavings Scheduler P0 P1 P2 Isend(1) sendNext Barrier Isend(1, req) Barrier Irecv(*, req) Barrier Isend(1, req) Barrier Wait(req) Wait(req) Recv(2) Wait(req) MPI Runtime

  41. Hijack Calls, Generate Relevant Interleavings Scheduler P0 P1 P2 Isend(1) Barrier sendNext Isend(1, req) Irecv(*, req) Irecv(*) Barrier Barrier Barrier Barrier Isend(1, req) Wait(req) Recv(2) Wait(req) Wait(req) MPI Runtime

  42. Hijack Calls, Generate Relevant Interleavings Scheduler P0 P1 P2 Isend(1) Barrier Barrier Isend(1, req) Irecv(*, req) Barrier Barrier Irecv(*) Barrier Barrier Barrier Isend(1, req) Barrier Wait(req) Recv(2) Wait(req) Wait(req) Barrier MPI Runtime

  43. Hijack Calls, Generate Relevant Interleavings Scheduler P0 P1 P2 Isend(1) Irecv(2) Barrier Isend Wait (req) Isend(1, req) Irecv(*, req) Barrier No Match-Set Irecv(*) Barrier Barrier Isend(1, req) Barrier Recv(2) SendNext Wait(req) Recv(2) Wait(req) Wait(req) Barrier Deadlock! Isend(1) Wait Wait (req) MPI Runtime

  44. Let us see ISP in action on ‘lucky.c’ and ‘unlucky.c’ • lucky.chas a deadlock that shows upon testing • unlucky.cdoes not reveal a deadlock upon testing • Testing is done using mpicc ; mpirun • Verification is done using ispcc ; isp

  45. Example MPI program ‘lucky.c’ (lucky for tester) Process P0 R(from:*, r1) ; R(from:2, r2); S(to:2, r3); R(from:*, r4); All the Ws… Process P1 Sleep(3); S(to:0, r1); All the Ws… Process P2 //Sleep(3); S(to:0, r1); R(from:0, r2); S(to:0, r3); All the Ws…

  46. MPI program ‘unlucky.c’ Process P0 R(from:*, r1) ; R(from:2, r2); S(to:2, r3); R(from:*, r4); All the Ws… Process P1 // Sleep(3); S(to:0, r1); All the Ws… Process P2 Sleep(3); S(to:0, r1); R(from:0, r2); S(to:0, r3); All the Ws…

  47. Runs of lucky.c and unlucky.c on mpichusing “standard testing” (“lucky” for tester) mpiccunlucky.c -ounlucky.out mpirun -np 3 ./unlucky.out (0) is alive on ganesh-desktop (2) is alive on ganesh-desktop (1) is alive on ganesh-desktop Rank 0 did Irecv Rank 1 did Send Rank 0 got 11 Sleep over Rank 2 did Send (2) Finished normally (1) Finished normally (0) Finished normally [.. OK ..] mpicclucky.c -olucky.out mpirun -np 3 ./lucky.out (0) is alive on ganesh-desktop (1) is alive on ganesh-desktop (2) is alive on ganesh-desktop Rank 0 did Irecv Rank 2 did Send Sleep over Rank 1 did Send [.. hang ..]

  48. Runs of lucky.c and unlucky.c on mpichusing “standard testing” (“lucky” for tester) mpiccunlucky.c -ounlucky.out mpirun -np 3 ./unlucky.out (0) is alive on ganesh-desktop (2) is alive on ganesh-desktop (1) is alive on ganesh-desktop Rank 0 did Irecv Rank 1 did Send Rank 0 got 11 Sleep over Rank 2 did Send (2) Finished normally (1) Finished normally (0) Finished normally [.. OK ..] mpicclucky.c -olucky.out mpirun -np 3 ./lucky.out (0) is alive on ganesh-desktop (1) is alive on ganesh-desktop (2) is alive on ganesh-desktop Rank 0 did Irecv Rank 2 did Send Sleep over Rank 1 did Send [.. hang ..] ispcc ; isp will detect deadlock in both cases !!

  49. With ISP at hand, WE ARE LUCKY IN BOTH CASES Not just ‘feeling lucky’ !! COMMANDS RUN : Ispcclucky.c [ later try unlucky.c ] Isp -n 3 -log /tmp/log1 ./a.out ispUI /tmp/log1 Commands to verify lucky.c or unlucky.c

  50. End of A

More Related