510 likes | 621 Views
Refinement Verification of Concurrent Programs and Its Applications. Hongjin Liang Univ. of Science and Technology of China Advisors: Xinyu Feng and Zhong Shao. Refinement. void main() { print a square; }. void main() { print a rectangle; }. .
E N D
Refinement Verification of Concurrent Programs and Its Applications Hongjin Liang Univ. of Science and Technology of China Advisors: XinyuFeng and ZhongShao
Refinement void main() { print a square; } void main() { print a rectangle; } T S: Thas no more observable behaviors (e.g. I/O events by print) than S.
Concurrent Program Refinement • Compilers for concurrent programs S Multithreaded Java programs Correct(Compiler): S, T. T = Compiler(S) T S Compiler T Java bytecode
Concurrent Program Refinement • Compilers for concurrent programs • Fine-grained impl. of concurrent objects (libraries) • E.g. java.util.concurrent
Concurrent object O Client code C … • … • push(7); • x = pop(); • … • … • push(6); • … void push(int v) { local b:=false, x, t; x := new Node(v); while (!b) { t := top; x.next = t; b = cas(&top, t, x); } } • intpop() { • … • … • } • Whole program C[O] How to specify/prove correctness?
Correctness of Concurrent Objects • Linearizability[Herlihy&Wing’90] • OlinS :correctness w.r.t. functionality • Spec S : abstract object (atomic methods) • Hard to understand/use • Equivalent to contextual refinement[Filipovic et al.] • OctxtSiffC. C[O] C[S]
OctxtSiffC. C[O] C[S] void push(int v) { … } int pop() { … } Concrete obj. O Client C • … • x := 7; • push( x ); • … • … • y := pop(); • print(y); • … • push Abstract obj. S • pop
Concurrent Program Refinement • Compilers for concurrent programs • Linearizability of concurrent objects (libraries) • Impl. of software transactional memory (STM) • Atomic block (transaction) fine-grained impl.
Concurrent Program Refinement • Compilers for concurrent programs • Linearizabilityof concurrent objects (libraries) • Impl. of software transactional memory (STM) • Impl. of concurrent garbage collectors (GC) • Impl. of operating system (OS) kernels Is such a refinement T S general enough & easy to verify?
Problems with T S T1 S1 T2 S2 (Compositionality) T1 || T2 S1 || S2 Existing work on verifying T S : either is not compositional, or limits applications.
Long-Standing Problems in Verifying Linearizability • Objects with Non-Fixed Linearization Points (LPs) • Future-dependent LPs(e.g. lazy set, pair snapshot) • Helping (e.g. HSY elimination-backoff stack) Most existing work : either not supports them, or lacks formal soundness.
Refinement vs. Progress Properties ? • Linearizability • Correctness w.r.t. functionality • Not talk about termination/liveness properties • Progress properties • Lock-freedom (LF) • Wait-freedom (WF) • Obstruction-freedom (OF) • Deadlock-freedom (DF) • Starvation-freedom (SF) Non-blocking impl. Lock-based impl.
Our Contributions (Part 1) • RGSim = Rely/Guarantee + Simulation • Compositional w.r.t. parallel composition • Flexible & applicable • optimizations in concurrent contexts • concurrent GC • fine-grained concurrent obj. • …
Our Contributions (Part 2) • RGSim = Rely/Guarantee + Simulation • A program logic for linearizability • Support non-fixed LPs • Verified 12 well-known algorithms (some are used in java.util.concurrent) • Light instrumentation mechanism to help verification • Formal meta-theory: simulation (extends RGSim) • Establish a contextual refinement
Our Contributions (Part 3) • RGSim = Rely/Guarantee + Simulation • A program logic for linearizability • A framework to characterize progress properties via contextual refinement (CR) • Propose different termination-sensitive CR • Equivalent to linearizability + progress • Unify all five progress properties (LF, WF, OF, DF, SF) • Make modular verification of whole program C[O] easier • Potential to have a generic verification framework for linearizability + progress
Outline • Rely-Guarantee-based simulation for modular verification of concurrent refinement • Logic for linearizability • Progress properties and contextual refinement
Modular Verification of T S T1 S1 T2 S2 (Compositionality) T1 || T2 S1 || S2
is NOT compositional w.r.t. parallel composition: T1 S1 T2 S2 T: local t; t = x; x = t + 1; print( x ); S: x++; print( x ); T1 ||T2 S1 ||S2 We have T S, since output(T) output(S) ; but we do not have T||T S||S .
Existing Proof Methods: Simulation in CompCert [Leroy et al.] • Source state e * * … (S, ) (S’, ’) (S’’, ’’) • observable event (e.g. I/O) • Target state e (T’, ’) (T’’, ’’) (T, ) …
Simulation in CompCert[Leroy et al.] Can verify refinement of sequential programs NOT compositional w.r.t. parallel composition T: local t; t = x; x = t + 1; print( x ); S: x++; print( x ); We haveT S , but notT||TS||S
Simulation in CompCert[Leroy et al.] Can verify refinement of sequential programs NOT compositional w.r.t. parallel composition • Consider NO environments • Simulation in process calculus (e.g. CCS [Milner et al.]) • Assume arbitrary environments • Compositional • Too strong: limited applications
Assuming Arbitrary Environments (S, ) (S’, ’) e * ’ ’ * env … (S’, ’’) (S’’, ’’’) (T’, ’) (T, ) ’ ’ e (T’’, ’’’) env … (T’, ’’) • Too strong to be satisfied, since env. can be arbitrarily bad. • Refinement applications have assumptions • about S & env.
Refinement’s Assumptions • Compilers for concurrent programs • Prog. with data races has no semantics (e.g. concurrent C++) • Not guarantee correctness for racy programs • Fine-grained objects • Accesses use same primitives (e.g. stack: push & pop) • Not guarantee correctness when env. can destroy obj. • More examples are in the thesis … [Boehm et al. PLDI’08] Env. of a thread cannot be arbitrarily bad !
Problems of existing simulations : • Considers noenv. in CompCert[Leroy et al.] NOT compositional w.r.t. parallel composition • Assumes arbitraryenv. in process calculus (e.g. [Milner et al.]) Too strong: limited applications Our RGSim : • Parameterized with the interference with env. • Compositional • More applications • Use rely/guarantee to specify the interference
Overview of Rely/Guarantee [Jones'83] • r: acceptable environment transitions • g: state transitions made by the thread y = y’ x = x’ y = y’ x = x’ ’ ’ ’ ’ Thread1 Thread2 • g2: • g1: • r2: • r1: Nobody else would update y Nobody else would update x I guarantee I would not touch x I guarantee I would not touch y Compatibility (Interference Constraints): • g2 r1 and g1 r2
RGSim = Rely/Guarantee + Simulation * (S’, ’’) R … (S, ) (S’, ’) (S’’, ’’’) e * * G G ≲ ≲ ≲ ≲ (T’, ’) (T’, ’’) r … e (T, ) g g (T’’, ’’’) (T, r, g) ≲ (S, R, G)
Soundness Theorem If we can find r, g, R and G such that • (T, r, g) ≲ (S, R, G) then we have: T S
Parallel Compositionality (T1, r1, g1) ≲ (S1, R1, G1) (T2, r2, g2) ≲ (S2, R2, G2) g1 r2 g2 r1 G1 R2 G2 R1 (PAR) (T1||T2, r1r2, g1g2) ≲ (S1||S2, R1R2, G1G2)
More on Compositionality (T1, r, g) ≲ (S1, R, G) (T2, r, g) ≲ (S2, R, G) (T1;T2, r, g) ≲ (S1;S2, R, G) (T, r, g) ≲ (S, R, G) b B (while b doT, r, g) ≲(while B doS, R, G) … An axiomatic proof system for refinement
We have applied RGSim to verify … • Optimizations in parallel contexts • Loop invariant hoisting, strength reduction and induction variable elimination, dead code elimination, … • Fine-grained impl. & concurrent objects • Lock-coupling list, counters, Treiber’s non-blocking stack, concurrent GCD algorithm, … • Concurrent garbage collectors • A general GC verification framework • Hans Boehm’s concurrent GC [Boehm et al. 91]
Outline • Rely-Guarantee-based simulation for modular verification of concurrent refinement • Logic for linearizability • Progress properties and contextual refinement
Linearizability of Concurrent Objects [Herlihy&Wing’90] • Correctness w.r.t. functionality • OlinS : Every concurrentexecution of object O is “equivalent” to some sequential execution of spec S
Linearizability of Object O A concurrent execution of O: push(7) ret pop() ret (7) Thread 1: Linearization point (LP) push(6) ret Thread 2: Sequential execution of S time push(6), ret, push(7), ret, pop(), ret(7)
t x next Example: Treiber’s Non-Blocking Stack [Treiber’86] Top next next v1 vk … Is it linearizable? v 4 while(!b){ 5 t := Top; 6 x.next := t; 7 b := cas(&Top, t, x); 8 } push(int v): 1 local b:=false, x, t; 2 x := new Node(); 3 x.data := v;
? lin Treiber’s stack O Abstract stack S Stk = v1 :: v2 :: … :: vk Top next next … v1 vk push(v): 1 local b:=false, x, t; 2 x := new Node(v); 3 while (!b) { 4 t := top; 5x.next = t; 6 b = cas(&top, t, x); 7 } PUSH(v): Stk := v::Stk; Not update the shared list “Fixed”: statically located in impl code Line 6: the only command that changes the list LP
? lin Treiber’s stack O Abstract stack S Top next next … Stk = v :: v1 :: v2 :: … :: vk v1 vk v Abstract opr PUSH(v) not done Proved it’s LP LP • 6 b := cas(&Top, t, x); • if (b) • linself; • > • 7 } < • push(v): Execute abstract opr simultaneously 1 local b:=false, x, t; 2 x := new Node(v); 3 while (!b) { 4 t := Top; 5x.next := t; Atomic block - { [PUSH(v)] … } • { [] … } Abstract opr is done
Basic Approach to Verify OlinS Inspired by [Vafeiadis’ Thesis] • Instrument(O) = D with linself at LPs • Verify D in program logic with rules for linself • New assertions [S] and [] • Ensure O’s LP step corresp. to S’s single step Not support non-fixed LPs • Future-dependent LPs • Helping
Challenge 1: Future-Dependent LP Example: Pair Snapshot [Qadeer et al. MSR-TR-2009-142] t2: write(i, d) t1: readPair(i, j) m 0 1 … k write(i, d) updates m[i] to a new value d readPair(i, j) intends to return snapshot of m[i] and m[j]
Pair Snapshot [Qadeer et al. MSR-TR-2009-142] • readPair(inti, j){ • 1 local s:=false, a, b, v, w; • 2 while (!s) { • 3 <a := m[i].d; v := m[i].v>; • 4 <b := m[j].d; w := m[j].v>; • 5 <if (v = m[i].v) s := true>; • 6 } • 7 return (a, b); • } write(inti, d){ 8 <m[i].d := d; m[i].v++>; } LP if line 5 succeeds m 0 1 … k • Future-dependent LP • Not supported by linself d know: m[i] = (a,v) at line 4 v version number Where is the LP ? • Line 4? But line 5 may fail, m[i] and m[j] may be re-read
Solution: Try-Commit • readPair(inti, j){ • 1 local s:=false, a, b, v, w; • 2 while (!s) { • 3 <a := m[i].d; v := m[i].v;> • 4 <b := m[j].d; w := m[j].v; • 5 <if (v = m[i].v) { s:= true; • 6 } • 7 return (a, b); • } speculateat potential LP, keep both result and original - { m[i] = (a, v) [RP, (i,j)] … } [RP, (i,j)] > trylinself; - { m[i] = (a, v) ( [, (a,b)] ) … } } > commit( [, (a,b)] ); - { s [, (a,b)] s …}
Challenge 2: Helping • Example: elimination-backoff stack [Hendler et al. SPAA’04] • t1 finishes t2’s opr t2’s LP is in the code of t1 • Need to linearize a thread other than self • New auxiliary command: lin(t) • New assertions: t S | t • Details are in the thesis…
Our Approach to Verify OlinS • Instrument(O) = D with auxiliary cmds at LPs • linself for fixed LPs • try-commit for future-dependent LPs • lin(t) for helping • Assertions to describe abstract code & states p, q ::= … | t S | t | p q | p q • Verify D in our program logic • Extend an existing logic with rules for aux cmds
Our Logic for OlinS ┝ {p} S {q} ┝ {p} S {q} … ┝ {p (t S)} lin(t) {q * (t )} ┝ {p (cid S)} trylinself{( p * (cid S) ) ( q * (cid) )} ┝ {p q} commit(p) {p} More rules and soundness are in the thesis
Soundness via Contextual Refinement Theorem (equivalence): • • O ctxtS O linS Proof follows [Filipovic et al., 2009] • Intentional • Extensional • “”: all proof methods for ctxtcan verifylin • ctxtis a well-studied concept in PL community(still challenging though) • “”: modular verification (view C[O] as C[S]) • C[S] is simpler to understand/verify
Outline • Rely-Guarantee-based simulation for modular verification of concurrent refinement • Logic for linearizability • Progress properties and contextual refinement
Progress Properties • Describe whether methods eventually return • Defined similarly to linearizablity • Describe objects’ behaviors instead of clients’ • Intentional instead of extensional • E.g. there always exists a method call that’ll return Can we use contextual refinement to define progress properties?
Our Results Linearizability OlinS Progress P(O) • Termination-sensitive contextual refinement OP S ( iffC. ObsBeh(C[O]) ObsBeh(C[S]))
Relationships between Progress Properties WF Wait-freedom LF Lock-freedom SF Starvation-freedom equiv. to Obstruction-freedom OF Deadlock-freedom DF + Linearizability
Conclusion • RGSim= Rely/Guarantee + Simulation • Idea: parameterized with interference with env. • Compositional! • Applications: optimizations, concurrent GC, … • Program logic for linearizability • Light instrumentation to help verification • linself for fixed LPs • lin(t) for helping • try-commit for future-dependent LPs • Verified 12 well-known algorithms