Software Testing with QuickCheck

1. FIGYELEM! Software Testing with QuickCheck Lecture 2 Symbolic Test Cases

2. Testing Data StructureLibraries • dict—purelyfunctionalkey-value store • new()—createemptydict • store(Key,Val,Dict) • fetch(Key,Dict) • … • Complex representation… just test the API • ”black box testing” • In contrast to testing e.g. dict invariants

3. A Simple Property • Perhaps the keys are unique… • Cannot test this without a generator for dicts prop_unique_keys() -> ?FORALL(D,dict(), no_duplicates(dict:fetch_keys(D))).

4. Generatingdicts • Black box testing  use the API to createdicts • Wesimulatelazyevaluationwhereweneed it dict() -> oneof([dict:new(), ?LET({K,V,D},{key(),value(),dict()}, dict:store(K,V,D))]). dict() -> oneof([dict:new(), ?LET({K,V,D},{key(),value(),dict()}, dict:store(K,V,D))]). dict() -> ?LAZY( oneof([dict:new(), ?LET({K,V,D},{key(),value(),dict()}, dict:store(K,V,D))]) ). A choicebetween generators Infinite recursion! key() -> oneof([int(),real(),atom()]). value() -> key(). Constant time

5. Let’s test it! • Black box testing: we need to knowhow this wasconstructed!

6. Symbolic Test Cases • {call,M,F,Args} represents a functioncall M:F(…Args…) symbolically. • eval(X) evaluatessymboliccalls in a term X dict() -> ?LAZY(oneof([ {call,dict,new,[]}, {call,dict,store,[key(),value(),dict()]} ])). Test case is nowsymbolic prop_unique_keys() -> ?FORALL(D,dict(), no_duplicates(dict:fetch_keys(eval(D)))).

7. Let’s test again! Shrinks the values, but not the structure! Wecanseeexactlyhow the failingcasewasbuilt up!

8. Shrinking the structure • QuickCheck has manyshrinking operations; hereweuse • Binds X1, X2… to valuesgenerated by G1, G2… • GeneratesResult • Can shrink to any of the Xi ?LETSHRINK([X1,X2,…], [G1,G2,…], Result(X1,X2,…))

9. dict() with Shrinking • ?LETSHRINK makes shrinkingrecursivetypesvery easy dict() -> ?LAZY(oneof( [{call,dict,new,[]}, ?LETSHRINK([D],[dict()], {call,dict,store,[key(),value(),D]})])).

10. Let’s test and shrink! Nice shrinkingresult -1 and -1.0, eh?

11. Testing vs. an Abstract Model • Howshoulddict operations behave? • The ”same” as a list of key-value pairs • Use the proplistslibrary as a reference • Make comparisons in the ”model” world Returns list of key-value pairs model(Dict) -> dict:to_list(Dict).

12. Commuting Diagrams Dict Dict + {K,V} dict:store(K,V,…) model model List [{K,V} | List] model_store(K,V,…) Hoare: Proof of Correctness of Data Representations, 1972

13. Testing store prop_store() -> ?FORALL({K,V,D}, {key(),value(),dict()}, begin Dict = eval(D), model(dict:store(K,V,Dict)) == model_store(K,V,model(Dict)) end). model_store(K,V,L) -> [{K,V}|L].

14. Next Steps… • Writesimilarproperties for all the dict operations • Extend the dict generator to useall the dict-returning operations • Eachproperty tests many operations • …and, of course, correct the specification!

15. Debuggingproperties • Why is a propertyfalse? • Weneedmore information! • ?WHENFAIL(Action,Property) performs Action whenreporting a failingcase • Typicaluse: equal(X,Y) -> ?WHENFAIL(io:format("~p /= ~p~n",[X,Y]), X==Y).

16. Exercises