1 / 58

Contributions à la génération de tests à base de contraintes Arnaud Gotlieb

Contributions à la génération de tests à base de contraintes Arnaud Gotlieb INRIA Rennes, France SIMULA Oslo, Norway. Soutenance d’habilitation, 12/12/2011. Embedded Software Testing. Java Card - Oberthur. HVAC - Thales. TCAS - Airbus. Saturn C90 – Cisco Norway.

eshe
Download Presentation

Contributions à la génération de tests à base de contraintes Arnaud Gotlieb

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Contributions à la génération de tests à base de contraintes Arnaud Gotlieb INRIA Rennes, France SIMULA Oslo, Norway Soutenance d’habilitation, 12/12/2011

  2. Embedded Software Testing Java Card - Oberthur HVAC - Thales TCAS - Airbus Saturn C90 – Cisco Norway BCE Rafale – Dassault Electronics • Critical software developmentinvolvesstrong V&V requirements: • At system testinglevel, safety-relatedproperties have to bechecked • Atintegrationtestinglevel, HW/SW integrationfailures must bedetected • At unit testinglevel, programmingfaults must bedetected and removed • Attheselevels, conformance to software certification standards isenforced Several (complementary) techniques at the unit level (code level): software model-checking static-analysis based verification software testing and Constraint-Based Testing… 

  3. Model-based Testing Spec. / Model Test case generation Code-based Testing Correct ? Test set implementation Execution Verdict: pass / fail Software Testing

  4. Spec. / Model Constraint generation Constraint solving Constraint model Test set Implementation Execution Verdict: pass / fail Constraint-Based Testing

  5. Constraint-Based Testing (CBT) Constraint-Based Testing (CBT) is the process of generating test cases against a testing objective by using constraint solving techniques Introduced 20 years ago by Offut and DeMillo in(Constraint-based automatic test data generation IEEE TSE 1991)Developed in the context of code-based testing and model-based testing Lots of Research works and tools !

  6. CBT: main tools CEA - List (Osmose S. Bardin P.Herrmann) Univ. of Madrid (PET M. Gomez-Zamalloa, E. Albert, G. Puebla) Univ. of Stanford (EXE D. Engler, C. Cadar, P. Guo) Univ. of Nice Sophia-Antipolis (CPBPV M. Rueher, H. Collavizza, P.V. Hentenryck) INRIA - Celtique(Euclide, JAUT A. Gotlieb, F. Charreteur) … Tools withexternalindustrial usage : GATEL (CEA B. Marre, since 2004) Test Designer (Smartesting B. Legeard, since 2003) PEX (Microsoft P. de Halleux, N. Tillmann, since 2009) Tools withinternalindustrial usage : InkaV1 (Dassault A. Gotlieb, B. Botella, in 2001) PathCrawler (CEA N. Williams, since 2004) SAGE (Microsoft P. Godefroid, since 2010)

  7. f (int x1, int x2, int x3) { if(x1 == x2 && x2 ==x3) if(x3==x1*x2) ... } The automatic test data generation problem (1) • Select either a path, branch, source code element, or testing criterion • Generate a test input or a test set that covers the element • Predict the expected outputs (i.e., reachability problem in infinite-state systems) Solving this problem would have broad industrial impact: • increase software quality and reliability through better code coverage and more systematic test inputs generation; • decrease testing costs through augmented automation; • automate conformance to Software Certification Standards as they require covering testing criteria (e.g., DO-178C, ISO 2626-1,…) ;

  8. f (int x1, int x2, int x3) { if(x1 == x2 && x2 ==x3) if(x3==x1*x2) ... } The automatic test data generationproblem (2) Given a location k in a program under test, generate a test input that reaches k Reachabilityproblem in infinite-state systemsisundecidable in general! Evenwhenaddingbounds, hard combinatorialproblem UsingRandomTesting, Prob{ reack k} = 2 over 232 232 232 = 2-95 = 0.00000…1. Constraintsolving techniques are required! • Loops (i.e., infinite-state systems) and infeasible paths • Pointers, dynamic structures, higher-order computations (virtual calls) • Floating-point computations, modular computations

  9. Our thesissincefiveteenyears: ConstraintProgramming techniques caneffectively and efficientlyaddresstheseproblems Illustratedwith2selected contributions in mytoday’s talk: 1) Constraint-based program exploration for automatic test data generation 2) Constraints over memorymodelsfor testing pointer programs

  10. Motivations Constraint-based program exploration for automatic test data generation(contribution 1) Constraints over memory modelsfor testing pointer programs(contribution 2) Conclusions and furtherwork Outline

  11. value of i to reach e ? e A reacheability problem a f( int i, … ) { a. j = 100; while( i > 1) b. { j++ ; i-- ;} … d. if( j > 500) e.… t f b … d t f

  12. e Backtrack ! Path-oriented exploration f( int i, … ) { a. j = 100; while( i > 1) b. { j++ ; i-- ;} … d. if( j > 500) e.… a t f b … 1. Path selection e.g., (a-b)14-…-d-e d t 2. Pathcondition generation (via symbolicexec.) j1=100, i1>1, j2=j1+1, i2=i1-1, i2>1,…, j15>500 f 3. Pathcondition solvingunsatisfiable FAIL Even without loops, #pathsis exponential with #decisions

  13. e Constraint-based program exploration f( int i, … ) { a. j = 100; while( i > 1) b. { j++ ; i-- ;} … d. if( j > 500) e.… a t f b … 1. Constraint model generation d t 2. Control dependenciesgeneration; j1=100, i3≤ 1, j3 > 500 f 3. Constraint model solvingj1 j3entailed unroll the loop 400 times  i1 in 401 .. 231-1 No backtrack !

  14. Constraint-based program exploration (Contribution 1) - Based on a constraint model of the whole program (i.e., eachstatementisseen as a relation )- Constraintreasoning over control structures- Requires to builddedicatedconstraintsolvers: * propagation queue management withpriorities * specificpropagators and meta-constraints * structure-aware labelling heuristics Prototype toolimplementation: Euclide (Gotlieb ICST’09)

  15. i1 in -4..2 no i1= 3 ? i1 in -5..3 i*=++i; /* i2 = (i1+1) 2*/ i2in5..16 ? i2 = 9 ? i2 = 7 ? i2 = 16 Assignment as Constraint Viewing an assignment as a relation requires to normalize expressions and rename variables (through single assignmentlanguages, e.g. SSA) i*=++i ;i2 = (i1+1)2 Using finite-domains bound-consistency filtering:

  16. Statements as constraints • Type declaration: signed long x; x in -231..231-1 • Assignments: i*=++i ; i2 = (i1+1)2 • Memory and arrayaccesses and updates: v=A[i] ( or p=Mem[&p] )  variations of element/3 • Control structures: dedicatedmeta-constraints(interface, awakening conditions and filteringalgorithms)Conditionnals (SSA)if D then C1;elseC2 iteLoops (SSA)whileD do C w

  17. Conditional as meta-constraint: ite/6 if( x > 0 ) 0 j1 = 5; 1 2 j2 = 18; 3 = …. j3 … ite( x > 0, j1, j2, j3, j1 = 5, j2 = 18) iff  x > 0  j1 = 5  j3 = j1  (x > 0)  j2 = 18  j3 = j2 ( x > 0  j1 = 5  j3 = j1 )  (x > 0)  j2 = 18 j3 = j2 ( (x > 0)  j3 = j2 )  x > 0  j1 = 5  j3 = j1 Join( x > 0  j1 = 5  j3 = j1 , (x > 0)  j1 = 18 j3 = j2 )

  18. Loop as meta-constraint: w/5 v3 = ( v1 , v2 ) while( Dec ) 2 1 body 3 • w(Dec, V1, V2, V3, body) iff • DecV3V1  bodyV3V1  w(Dec, v2,vnew,v3, bodyV2Vnew) • DecV3V1  v3=v1(DecV3V1  bodyV3V1 )  DecV3V1  v3=v1 • (DecV3V1  v3=v1)  DecV3V1  bodyV3V1  w(Dec,v2,vnew,v3,bodyV2Vnew) • join(DecV3V1  bodyV3V1 w(Dec,v2,vnew,v3,bodyV2Vnew) , DecV3V1  v3=v1)

  19. f( int i ) { j = 100; while( i > 1) { j++ ; i-- ;} … if( j > 500) … no i = 23, j1=100 ? iin 401..231-1 j1 = 100,j3> 500 ? i3 = 10 ? i3 = 1, j3 = 122 • w(Dec, V1, V2, V3, body) :- • DecV3V1  bodyV3V1  w(Dec, v2,vnew,v3, bodyV2Vnew) • DecV3V1  v3=v1(DecV3V1  bodyV3V1 )  DecV3V1  v3=v1 • (DecV3V1  v3=v1)  DecV3V1  bodyV3V1  w(Dec,v2,vnew,v3,bodyV2Vnew) • join(DecV3V1  bodyV3V1 w(Dec,v2,vnew,v3,bodyV2Vnew , DecV3V1  v3=v1) w(i3 > 1, (i,j1), (i2,j2), (i3,j3), j2 = j3 + 1  i2 = i3 - 1)

  20. Features of constraint-based exploration • Specialmeta-constraintsimplementation for ite and w • By construction, w isunfoldedonlywhennecessarybut w may NOT terminate !  only a semi-correct test data generationprocedure • JoinisimplementedusingAbstract Interpretationoperators(e.g., interval-based union, weak-joinoperator, widening in Euclide) • Specialpropagatorsbased on linear-based relaxationsUsingLinearProgramming over rationals(i.e., Q_polyhedra) Abstraction-based relaxations

  21. Duringconstraint propagation, constraintscanberelaxed in Abstract Domains (e.g., Q-Polyhedra, Octagons, …) Z = X * Y, X in a..b, Y in c..d d b a c Abstraction-based relaxations •  { Z - Ya – Xc +ac ≥ 0, • Xd – Z –ad + aY ≥ 0, • bY – bc – Z + Xc ≥ 0, • bd – bY – Xd + Z ≥ 0, • a ≤ X ≤ b, c ≤ Y ≤ d} • To benefitfromspecializedalgorithm (e.g., simplex for linearconstraints) and capture global states of the constraint system • Requiresafe/correct over-approximation (to preservepropertysuch as:if the Q-Polyhedraisvoidthen the constraint system isunsatisfiable) • Q-Polyhedra in Euclide, implementingDynamicLinear Relaxation, propagation queue withpriorities

  22. Joinoperationscanberealized by convexhull, but usuallytoocostly ! In Euclide, wetookadvantage of the weak-join of Q_polyhedra(based on simplex calculations) Abstraction-based relaxations:weak-joinoperator(Sankaranarayanan et al. VMCAI’06) A B

  23. Abstraction-based relaxations:weak-joinoperator(Sankaranarayanan et al. VMCAI’06) A B

  24. Abstraction-based relaxations:weak-join operator (Sankaranarayanan et al. VMCAI’06) A B

  25. Constraint-based program exploration(summary of contribution 1) Foundations of the approach(GotliebBotellaRueher ISSTA’98, CL’2000) Abstraction-based relaxation (DenmatGotliebDucassé ISSRE’07) Global constraint w, extendedwithwidenning (DenmatGotliebDucassé CP’07) Euclide: A Constraint-basedtestingplatform for C (Gotlieb ICST’09) Application on the TCAS case study(Gotlieb KER Journal 2011)*** PhDThesis of Tristan Denmat(Defense in June2007)Post-doc of Pierre Rousseau (2007-2008) *** Prototype toolsimplementation: Taupo(INRIA T. Denmatin 2007), Euclide (INRIA A. Gotlieb in2009)

  26. Constraints over memorymodels for testing pointer programs(contribution 2) How to applyconstraint-basedreasoning over statementlike *p := *p+1 ? p Thenfail or exception a p Thena2 = a1+1 *p := *p + 1 p a Thena2 = a1+1 or b2 = b1+1 b p Thenp2 = p1+1, meaningthatp nowrefers to the nextmemory location

  27. Our propositions How to represent abstract memories and to reason on them ? 1) Constraintreasoning over Memory, as a set of graphs (Gotlieb et al., IST 2007) a2 an a1 .... p Constraintreasoning over Memory, as a structured set of unboundedarrays (Charreteur et al., JSS 2009)

  28. Exploiting points-to informations for modeling use/def of dereferenced pointers  Pointer Static Single Assignmentform x1 p ... xn Use of a dereferenced pointer: Def of a dereferenced pointer: Memory as a set of graphs

  29. Global constraintsto represent pointer dereferencing Def of a dereferenced pointer: combinator_d Use of a dereferenced pointer: combinator_u X = *P ; istrueiffi such as P = Pi  X = Vi *P = Exp; istrueiffi such asP = Pi  Xi = Exp {Xj = Vj} ji NB: When P = 0 (invalidaddress) thenu and dfailor throw an exception

  30. t next Weaknesses of this Memory model • Requires a preliminary points-to analysis that may be too imprecise when dynamic (de-)allocation is involved • Pointers as function inputs, can point to anything on the heap • Some conditions may constrain the shape of dynamic data structures. How to handle this in a constraint solver ? constrains t to int P(struct cell * t) { if( t == t->next ) { …

  31. Memory, as a structured set of unboundedarrays V : integer within a finite domain Type : 16,32,64 bits, signed, unsigned dom : {possible values} Min .. Max TAB : tableau status: closed or not cont. : { @i – Vi, …} V : float within an interval Type : float (32), double (64) dom. : Min .. Max • M : memory • Integers : TABi • Floats : TABf • Pointers : TABp • Structures : [S1,S2,..] V:pointer possibly_null : yes, no dom : {possibles values} nondom : {non-possible values} S : structure status : closed or not cont. : {@i}

  32. C program Constraints store i = i + 1 -----------------------------------> load_elt(@i, I1, M1) I2 = I1 + 1 store_elt(@i, I2, M1,M2) *p = 3 ------------------------------------> load_elt(@p, P1, M2) DP1 = 3 store_elt(P1,DP1,M2,M3) Everystatementisinterpreted as constraints over Memory states

  33. M1 : Status : not closed Includes : i – Vi j – Vj k – Vk … M2 : Status : not closed Includes : i – Vi’ j – Vj’ k – Vk’ … Store_elt P : Domain pointer {i,j} V: Domain Integer 1.. 5 store_elt(P,V,M1,M2)

  34. M1 : Status : not closed Includes : i – Vi  1.. 2 j – Vj  5.. 9 k – Vk 2 … M2 : Status : not closed Includes : i – Vi’  3..6 j – Vj’  7..18 k – Vk’ ? … Store_elt P : Domain pointer {i, j} V: Domain Integer 1.. 5 store_elt(P,V,M1,M2) Automaticdeductionsafter the constraint propagation step: P = i, V = Vi’ in 3..5, Vj = Vj’ in 7..9, Vk = Vk’ =2

  35. Awake Model for the definition of a new constraint success exit SVAR ConstraintsStore Suspend fail failure reduce

  36. store_elt(P,V,M1,M2) If( dom(P) = {p} and p != Null ) then add( M2[p] = V) and .. If( M2[i] != M1[i] ) then add( P = i and M2[i]=V) and .. success SVAR ConstraintsStore Awake Suspend If( dom(V) =  or dom(P) = ) then fail If( dom(P) = {p} and p = Null) then fail reduce failure dom(P)  {i / dom(M2[i]  dom(V) != } dom(V)  idom(P) dom(M2[i]) dom(M1[i])  dom(M2[i])  dom(M1[i])if( i  dom(P) ) dom(M2[i])  dom(M1[i])  dom(M2[i])if( i  dom(P) ) dom(M2[i])  dom(M1[i]  dom(V)) otherwise

  37. Constraints over memorymodels for testing pointer programs(summary of contribution 2) Constraintreasoning over Memory as graphs (GotliebDenmatBotella ASE’05, COMPSAC’05, IST’07) Memory, as a structured set of unboundedarrays(GotliebBotella ICCSEA’06, Charreteur et al. TAIC-PART’07, JSS 2010) Extension to Bytecode Java, inheritance and polymorphism (CharreteurGotlieb, ISSRE’10) *** PhDThesis of Florence Charreteur (Defense in March 2010) *** Prototype toolsimplementation: Inka V1 (Dassault A.Gotlieb, B.Botella, in 2001)InKaV2(Dassault A. Gotlieb, B.Botella in 2006), JAUT (INRIA F. Charreteur, in 2010)

  38. SymmetricTesting: usingsymmetries to test programs (Gotlieb ISSRE’03, Gotlieb Bernard QSIC’06) Constraints over floating-point variables(BotellaGotlieb Michel STVR 2006, Carlier Gotlieb ICTAI’11) Probabilisticconstraintreasoning for statisticaltesting - New probablisticconstraintcombinators(Petit Gotlieb CP’07, QSIC’07) - Statistical test case generation(Gotlieb Petit RT’06, JSS 2009) Explanation-basedgeneralization of infeasiblepaths in DynamicSymbolicExecution(Delahaye BotellaGotlieb ICST’10, TSE in revision) Testingconstraint programs withconstraints(LazaarGotliebLebbah CP’10, Constraints Journal, ICST’11) Whatwasleftapart:

  39. ConstraintProgrammingisworth to Software Testing automation:In this talk, 2 main contributions to the domain of CBT:- Constraint-based program exploration (w)- Constraints over memory models for testing pointer programs Stronglyinvolved in the development of 3 Research prototype tools :Euclide, Inka and FPSE (more than 30KLOC Prolog, Java, C) Researchprojects: ACI V3F, ANR CAT/U3CAT, ANR CAVERN… Applications to the testing of embedded software Conclusions Java Card - Oberthur TCAS - Airbus BCE Rafale – Dassault Electronics

  40. Emerging concept in code- and model-basedautomatic test data gener. ConstraintProgramming techniques offers:- Global constraintsmodelling to handlecontrol and data structures (whilepure SAT-solvingdoes not workwell in thatcontext) - Versatility and flexibility of CP (while pure LP or SMT approaches are veryrigid). Handles non-linearconstraints over finitedomains. - Generic techniques to implement new solvers, with abstraction-based relaxation, even if unsatisfiabilitydetection has to beimproved by combining techniques (e.g., SMT/CP) Mature tools (academic and industrial) alreadyexist, but application on real-sizedindustrial cases still have to bedemonstrated ConclusionsConstraint-BasedTesting

  41. Arrayconstraintsolving.A combined SMT/CP approachfor solvingconstraintswitharrays and arithmetics. Constraintsolver CCFD and large experimental validation over random formulas.joint workwith S. Bardinfrom CEA Constraint-BasedTestingfromfeaturemodels, in the context of the testing of highly-configurable systemsPhD A. HervieuVALVES project Furtherwork HVAC - Thales Saturn C90 – Cisco Norway

  42. Thankyou! • PhDstudentsTristan Denmat, Matthieu Petit, Florence Charreteur, Mickael Delahaye, NadjibLazaar, Aymeric Hervieu • Post-docSandrine Gouraud, Pierre Rousseau, Matthieu Carlier • Co-authorsOlivier Lhomme, Michel Rueher, Claude Michel, Yahia Lebbah, Michel Leconte, Mireille Ducassé, Bernard Botella, Patrick Taillibert, Franck Calvet, Bruno Marre , Benjamin Blanc, Frédéric Dadeau, Nicky Williams, Catherine Dubois, Patrick Bernard, Matthieu Wattel, Benoit Baudry, Sébastien Bardin, Lionel Briand

  43. Constraintsolving over floating-point expr. for testingnumerical programs (Contribution 3) • Numerical programs used in embeddedsystemscontain lots of floating-point computations • - Automatic test data generationthroughsymbolicexecutionischallenging for those programs as: * floating-point computations exhibit round-off errors Absorption (Xf +  == Xf), Cancellation( Xf-Yf==K.( X-Y ) when K>>1* floating-point variables are currentlyhandled as reals or rationals inconstraintsolvers (even if floating-point numbers are used)

  44. On the reals :x  (0,10000) On the floats : no solution ! An illustrative program float foo( float x) { float y = 1.0e12, z ; 1. if( x < 10000.0 ) 2. z = x + y; 3. if( z > y) 4. … Is the path 1-2-3-4 feasible ? Path conditions: x < 10000.0 x + 1.0e12 > 1.0e12

  45. On the reals : no solution On the floats: x  (0, 32767.99…) Conversely, float foo( float x) { float y = 1.0e12, z ; 1. if( x > 0.0 ) 2. z = x + y; 3. if( z == y) 4. … Is the path 1-2-3-4 feasible ? Path conditions: x > 0.0 x + 1.0e12 = 1.0e12 Our approach: to builda dedicatedconstraintsolver over the floats, able to capture floating-point solutions Prototype tool: FPSE (BotellaGotlieb Michel STVR 2006, Carlier Gotlieb 2011)

  46. Notations:with[a add b] representingnear(a + b)Illustrated for the near-to-evenrounding mode only Our approach to solveFP constraints:IntervalPropagation over FP variables a+ x mid(a,a+) near(x) a a-

  47. Floating-point projections - Direct and indirect projections for the assignment:proj(r, r:= a add b) (direct) [r := aaddb] leads to proj(a, r:= a add b) (1st indirect)proj(b,r := a add b) (2nd indirect) - Direct projections (over numericfpnumbers):If Ir= [rl,rh], Ia = [al,ah] and Ib = [bl,bh] then[r := aaddb][rl’,rh’]  [aladdbl, ahaddbh]  [rl,rh][r := asubsb][rl’,rh’]  [al subsbh, ahsubsbl]  [rl,rh] ...

  48. Ex: Direct projection [r := a add b] rh’  min(ahadd bh, rh) impossible rl’  max(aladd bl, rl) impossible Monotony of rounding :r1  r2 near(r1)  near(r2) rh ah bh bl al rl IEEE-754 accuracyproperty

  49. More complex : indirect projections If Ir= [rl,rh], Ia = [al,ah] and Ib = [bl,bh] then 1st indirect projection of [r := a add b] [al’,ah’]  [mid(rl,rl-) subsbh, mid(rh,rh+) subsbl]  [al,ah]1st indirect projection of [r := a subs b] [al’,ah’]  [mid(rl,rl-) addbl, mid(rh,rh+) addbh]  [al,ah] 2nd indirect projection of [r := a subs b] [bl’,bh’]  [alsubsmid(rh,rh+), ahsubsmid(rl,rl-)]  [bl,bh]

  50. Ex: 1st indirect projection [r := a add b] impossible ah’ min(mid(rh,rh+) subsbl, ah) al’  max( mid(rl, rl-) subsbh, al) not optimal, but computable withnear-to-even ah rh+ rh IEEE-754 accuracyproperty bh bl rl rl- al

More Related