420 likes | 537 Views
HLDVT 2009. Dynamic Verification of Multicore Communication Applications in MCAPI Subodh Sharma, UofU Ganesh Gopalakrishnan, UofU Eric Mercer, BYU. Supported by NSF CCF 0903408/0903491 and SRC 2009-TJ-1993/1994. 1. Concurrency Space in Multicore Era. Concurrency Space in Multicore Era.
E N D
HLDVT 2009 Dynamic Verification of Multicore Communication Applications in MCAPI Subodh Sharma, UofU Ganesh Gopalakrishnan, UofUEric Mercer, BYU Supported by NSF CCF 0903408/0903491 and SRC 2009-TJ-1993/1994 1
Concurrency Space in Multicore Era Requires Standardization Multicore Association (MCA) Effort
What is MCAPI (Multicore Communication API)? • An API specification from MCA (Multicore Association) • To program embedded systems like mobile phones, PDAs, routers, servers, etc. • Not restricted to SPMD (like MPI) or multi threaded style of programming.
Inter API interactions Taxonomy for MCA APIs MTAPI – Task Management API • Specification work yet to begin • Thread Pooling, Work Stealing queues e.g. CILK, TBB, TPL etc. MCAPI – Communication API • Message based • Packet/Scalar Channel based MRAPI – Resource Management API • Semaphores, mutexes • Shared memory segment allocation, deallocation
State of MCA Efforts? • MCAPI standard is released. • MCAPI reference implementation exists • A commercial implementation – Polycore, Inc. • In-house implementation on FPGA in progress (Utah). • MRAPI standard work in progress. • MTAPI efforts yet to begin.
State of MCA Efforts? • Standardization efforts take a long time to mature • Need for FV in MCA API development. • Our effort: Early Engagement of FV in the MCA APIs space • To promote early adoption of the API • To promote better programming practices • To help avoid early pit-falls – e.g. unspecified behaviors
Achievements so far! • Formal Specification work (at BYU) • Permits symbolic analysis • Murphi -> CVC • Query Oracle for scenario outcome calculation. • Platform testing • Dynamic Verification of MCAPI user apps (at Utah) • MCC tool created. • Current capabilities of MCC • Dynamic search of dependencies • Exerting a deterministic runtime control • Algorithmic Advances • Symmetry reduction in MCAPI domain space.
Focus of the talk! • Formal Specification work (at BYU) • Permits execution based symbolic analysis • Murphi -> CVC • Query Oracle for scenario outcome calculation. • Platform testing • Dynamic Verification of MCAPI user apps (at Utah) • MCC tool created. • Current capabilities of MCC • Dynamic search of dependencies • Exerting a deterministic runtime control • Algorithmic Advances • Symmetry reduction in MCAPI domain space.
Related Work • CHESS tool from MSR • Uses preemption bounded search • Cannot be used for message passing programs • ISP tool from Utah • Used for MPI verification • MCC differs in the way determinism is enforced at runtime. • Inspect tool from Utah • Used for shared memory programs • Implements the classical DPOR algorithm. • JPF • A model checker for Java bytecode.
MCAPI Preliminaries • MCAPI Node – • Processes, Threads, OS instance, Processor core, etc. • MCAPI Endpoint – • Socket-like communication termination points • Identified by a tuple <node_id, port_id> • MCAPI Channels • Point to point unidirectional FIFO connections between a pair of endpoints. • A single communication universe
Message received at FIFO receive queue MCAPI Messages Node 1 Node 2 Connectionless Message Port 1 <1,1> Port 1 <2,1> To <2,1> Port 2 <1,2>
Sent over a connected channel • <1,1> can communicate on ONE channel only MCAPI Connection Oriented Communication Node 1 Node 2 Port 1 <1,1> Port 1 <2,1> To <2,1> Port 2 <1,2> ERROR!! To <3,1> Node 3 Port 1 <3,1>
Illustration of MCAPI C program int main () { … for(t=0; t<NUM_THREADS; t++){ rc = pthread_create(&threads[t], NULL, run_thread, (void*)&thread_data_array[t]); } for (t = 0; t < NUM_THREADS; t++) { pthread_join(threads[t],NULL); } } void* run_thread (void *t) { … mcapi_initialize(tid,&version,&status); if (tid == 1) { recv_endpt = mcapi_create_endpoint (PORT_NUM, &status); mcapi_msg_recv(recv_endpt,msg, BUFF_SIZE,&recv_size, &status); mcapi_msg_recv(recv_endpt,msg, BUFF_SIZE, &recv_size, &status); } else { send_endpt = mcapi_create_endpoint (PORT_NUM,&status); recv_endpt = mcapi_get_endpoint (1,PORT_NUM,&status); mcapi_msg_send(send_endpt,recv_endpt, msg,strlen(msg), 1,&status); } mcapi_finalize(&status); return NULL; } Potential deadlock
instrumentation compile request/permit request/permit MCC – MCAPI Checker MCAPI C Program Executable Scheduler Instrumented Program thread 1 thread n MCAPI Library Wrapper Workflow of MCC.
Bug Classes in MCAPI User Applications • Non determinism due to communication race. • The recv() call is passed with only receiving endpoint as a parameter. • The receiving endpoint extracts message from the FIFO receive queue. T0 T1 T2 --- --- --- The send that matches the recv may decide the correctness !! S(e1,e2) R(e2) S(e1,e2) R(e2) POTENTIAL BUG
Bug Classes in MCAPI User Applications • Mismatched Calls Does not exist T0 T1 --- --- R(1) Get_endpoint(2) S(1,2) “get_endpoint” invoked for a non-existent endpoint -- ERROR
Challenge: exponential interleavings T0 T1 T2 T3 T4 T5 Possible Interleaving A B Only these 2 are RELEVANT!!! Dependent MCAPI calls 19
Relevant interleaving exploration by MCC scheduler T0 T1 T2 Intra happens-Before Edge S (ep1,ep3) S(ep2,ep3) R(ep3) R(ep3)
Relevant interleaving exploration by MCC scheduler Set of matching transitions No runnable thread Interleaving 1 T0 T1 T2 S (ep1,ep3) S (ep2,ep3) R (ep3) Issue to the runtime R (ep3) Match sets are computed at “decision points” Enabled Transitions: S(ep1, ep3), S(ep2, ep3), R (ep3) Match Set: <S(ep1, ep3), R (ep3)> <S(ep2, ep3), R (ep3)>
Relevant interleaving exploration by MCC scheduler Interleaving 1 T0 T1 T2 S (ep1,ep3) S (ep2,ep3) R (ep3) Issue to the runtime R (ep3) Enabled Transitions: S(ep2, ep3), R (ep3) Match Set: <S(ep2, ep3), R (ep3)>
Relevant interleaving exploration by MCC scheduler Interleaving 2 T0 T1 T2 S (ep1,ep3) S (ep2,ep3) R (ep3) Issue to the runtime R (ep3) Enabled Transitions: S(ep1, ep3), S(ep2, ep3), R (ep3) Match Set: <S(ep2, ep3), R (ep3)>
Relevant interleaving exploration by MCC scheduler Interleaving 2 T0 T1 T2 S (ep1,ep3) S (ep2,ep3) R (ep3) R (ep3) Issue to the runtime Enabled Transitions: S(ep2, ep3), R (ep3) Match Set: <S(ep1, ep3), R (ep3)>
Relevant interleaving exploration by MCC scheduler Interleaving 2 T0 T1 T2 S (ep1,ep3) S (ep2,ep3) R (ep3) Error Caught R (ep3) Issue to the runtime Enabled Transitions: S(ep2, ep3), R (ep3) Match Set: <S(ep1, ep3), R (ep3)>
Enforcing a deterministic runtime match • For non blocking requests, there is an additional problem • Runtime communication race even after a scheduler decides deterministically. T0 T1 T2 Match-set 1 IS (ep2,ep3) IR (ep3) IS (ep1,ep3) IR (ep3) Match-set 2
Enforcing a deterministic runtime match T0 T1 T2 Match-set 1 RACE IS (ep2,ep3) IR (ep3) IS (ep1,ep3) Not finished yet!! IR (ep3) Match-set 2
Enforcing a deterministic runtime match Solution 1 to enforce a deterministic match. • Addition of Probing function to the MCAPI library • Probes an endpoint’s receive queue for a message • Returns TRUE if the message is found • Scheduler issues the next match-set only when the probe function returns TRUE.
Enforcing a deterministic runtime match Probing Solution T0 T1 T2 Receive Queue IS (ep1,ep3) IS (ep2,ep3) IR (ep3) matched IR (ep3)
Enforcing a deterministic runtime match Probing Solution T0 T1 T2 Receive Queue IS (ep1,ep3) IS (ep2,ep3) IR (ep3) IR (ep3) matched Probes Runtime MCC Scheduler
T1 Enforcing a deterministic runtime match Probing Solution T0 T1 T2 Receive Queue IS (ep1,ep3) IS (ep2,ep3) IR (ep3) IR (ep3) Probes Runtime MCC Scheduler
T0 T1 Enforcing a deterministic runtime match Probing Solution T0 T1 T2 Receive Queue IS (ep1,ep3) IS (ep2,ep3) IR (ep3) IR (ep3)
Enforcing a deterministic runtime match Solution 2 to enforce a deterministic match. • Scheduler induces a “test” call • Scheduler spin-loops on the request handle until the successful completion of the “test” call. We opt Solution 2 in our work as it is non-intrusive.
Enforcing a deterministic runtime match Dummy “wait” Solution T0 T1 T2 IS (ep1,ep3) IS (ep2,ep3) IR (ep3) matched IR (ep3) Enabled Transitions: Send(ep1, ep3), Send(ep2, ep3), Recv (ep3) Match Set: <Send(ep2, ep3), Recv (ep3)> <Send(ep1, ep3), Recv (ep3)>
Enforcing a deterministic runtime match Dummy “wait” Solution T0 T1 T2 IS (ep1,ep3) IS (ep2,ep3) IR (ep3) Test () IR (ep3) Runtime MCC Scheduler
Enforcing a deterministic runtime match Dummy “wait” Solution T0 T1 T2 IS (ep1,ep3) IS (ep2,ep3) IR (ep3) Test() IR (ep3) matched Runtime MCC Scheduler
Concluding remarks and future work • MCC currently supports connectionless API constructs • Checks for safety assertion violations and Deadlocks • Steadily improve MCC over this year • Support for connection-oriented API calls, sanity checks etc. • Release MCC and assess experience of industrial app writers
Happens-Before Relationship – Computed Dynamically T0 T1 T2 IS (ep2,ep3, r3) IR (ep3, r4) IS (ep1,ep3, r1) IS (ep1,ep4, r2) IR (ep3, r5) W(r2) R (ep3)
Review of POR • Dynamic POR Dependencies based on concrete values x=56, y=753 (e.g.) at runtime! t1: a[x] := … t2: … := a[y] t2: … := a[y] t1: a[x] := … t2: … := a[y] Equal ??
Happens-Before Relationship – Example T0 T1 T2 IS (ep2,ep3, r3) IR (ep3, r4) IS (ep1,ep3, r1) IS (ep1,ep4, r2) IR (ep4, r5) W(r2) R (ep3)
OOO completion: Why construct Happens-Before relationship? T0 IS (1GB to T1) Must calls complete in PROGRAM ORDER? IS (1 byte to T2) NO!!! 42