An Integrated Data Model Verifier with Property Templates

1. An Integrated Data Model Verifier with Property Templates JaideepNijjar Ivan BocicTevfikBultan {jaideepnijjar, bo, bultan}@cs.ucsb.edu University of California, Santa Barbara Department of Computer Science Verification Lab FormaliSE 2013

2. Web Software Everywhere • Commerce, entertainment, social interaction • We will rely on web apps more in the future • Web apps + cloud will make desktop apps obsolete

3. Model-View-Controller Pattern DB • Benefits of the MVC pattern: • Separation of concerns • Modularity • Abstraction Model View • MVC pattern has become the standard way to structure web applications • Ruby on Rails • Zend for PHP • CakePHP • Struts for Java • Django for Python • … Controller

4. A Data Model Verification Approach MVC Design Principles • Bounded and Unbounded Automatic Extraction Add data model properties

5. iDaVer: Integrated Data Model Verifier Active Records Properties Choice of technique (expressed using Templates) Model Extraction formal data model + property + technique Verification Unbounded (SMT Solver) Unbounded (FOL Thm Prover) Bounded (Alloy) Property Verified Property Failed + Counterexample Unknown

6. Outline • Motivation • Overview of Approach • Rails Data Models • Basic Relations • Options to Extend Relations • Formalization of Semantics • Property Templates • Verification Techniques • Case Study • Conclusions and Future Work

7. A Rails Data Model Example Role * class User < ActiveRecord::Base has_and_belongs_to_many :roles has_one :profile, :dependent => :destroy has_many :photos, :through => :profile end class Role < ActiveRecord::Base has_and_belongs_to_many :users end class Profile < ActiveRecord::Base belongs_to :user has_many :photos, :dependent => :destroy has_many :videos, :dependent => :destroy, :conditions => "format='mp4'" end class Tag < ActiveRecord::Base belongs_to :taggable, :polymorphic => true end class Video < ActiveRecord::Base belongs_to :profile has_many :tags, :as => :taggable end class Photo < ActiveRecord::Base ... * User 1 1 * 0..1 1 * Profile Photo 1 1 format=.‘mp4’ * 1 Taggable Video * Tag

8. Rails Data Models • Data model verification: Analyzing the relationships between data objects • Specified in Rails using association declarations inside the ActiveRecord files • The basic relationships • One-to-one • One-to-many • Many-to-many • Extensions to the basic relationships using Options • :through, :conditions, :polymorphic, :dependent

9. The Three Basic Relationships in Rails class User < ActiveRecord::Base has_one:profile end. class Profile < ActiveRecord::Base belongs_to:user end User • One-to-One . • One-to-Many 1 1 Profile class Profile < ActiveRecord::Base has_many:videos end. class Video < ActiveRecord::Base belongs_to:profile end Profile 1 * Video

10. The Three Basic Relationships in Rails class User < ActiveRecord::Base has_and_belongs_to_many:users end class Role < ActiveRecord::Base has_and_belongs_to_many:roles end • Many-to-Many User * * Role

11. Options to Extend the Basic Relationships • :through Option • To express transitive relations • :conditions Option • To relate a subset of objects to another class • :polymorphic Option • To express polymorphic relationships • :dependent Option • On delete, this option expresses whether to delete the associated objects or not

12. The :through Option class User < ActiveRecord::Base has_one :profile has_many :photos, :through => :profile end class Profile < ActiveRecord::Base belongs_to :user has_many :photos end class Photo < ActiveRecord::Base belongs_to :profile end Profile 0..1 1 * 1 Photo User 1 *

13. The :dependent Option class User < ActiveRecord::Base has_one :profile,:dependent => :destroy end class Profile < ActiveRecord::Base belongs_to :user has_many :photos,:dependent => :destroy end • :delete directly delete the associated objects without looking at its dependencies • :destroy first checks whether the associated objects themselves have associations with the :dependent option set User Profile Photo 1 1 0..1 *

14. Formalizing Rails Semantics Formal data model: M = <S, C, D> • S: The sets and relations of the data model (data model schema) • e.g. { Photo, Profile, Role, Tag, Video, User} and the relations between them • C: Constraints on the relations • Cardinality constraints, transitive relations, conditional relations, polymorphic relations • D: Dependency constraints • Express conditions on two consecutive instances of a relation such that deletion of an object from the first instance leads to the other instance

15. Outline • Motivation • Overview of Approach • Rails Data Models • Property Templates • Verification Techniques • Case Study • Conclusions and Future Work

16. Property Templates • User-friendly and makes it easy to express properties • Manually writing properties is a time-consuming and error-prone process • Requires familiarity with input modeling language of solver • Templates are language-neutral • Do not require familiarity with SMT-LIB, Spass and Alloy languages, and understanding of output specifications • Make it easy to rerun the tool and switch the verification technique, by not requiring the user to rewrite the property • Eight property templates available for the most common data model properties

17. Property Templates • alwaysRelated[classA, classB] • To check that objects from classA are always related to objects of classB • someMultipleRelated[classA, classB] • To check that it is possible for objects of classA to be related to more than one object of classB • someUnrelated[classA, classB] • To check that it is possible for an object of classA to not be related to any objects from classB • transitive[classA, classB, classC] • To check that the relation between classA and classC is the composition of the relations between classA and classB, and classB and classC

18. Property Templates • noDangling[classA, classB] • To check that when an object of classA is deleted, there are no objects from classB that are left with a dangling reference to the deleted object • deletePropagates[classA, classB] • To check that when an object of classA is deleted, related objects in classB are also deleted • noDeletePropagation[classA, classB] • To check that when an object of classA is deleted, related objects in classBare not deleted • noOrphans[classA, classB] • To check that deleting an object from classA does not cause related objects in classB to be orphaned • An orphan is an object that is related to no other object

19. Outline • Motivation • Overview of Approach • Rails Data Models • Property Templates • Verification Techniques • Bounded Verification with Alloy • Unbounded Verification with SMT Solver • Unbounded Verification with FOL Theorem Prover • Case Study • Conclusions and Future Work

20. Active Records iDaVer Choice of technique Properties Unbounded Verification Model Extraction Alloy Encoder SMT-LIB Encoder formula formula formal data model + property Bounded Verification Alloy Analyzer SMT Solver instance or unsat instance or unsat or unknown Results Interpreter Results Interpreter FOL Encoder formula Theorem Prover proof found or completion found or timeout Results Interpreter Property Verified Property Failed + Counterexample Unknown

21. Active Records Bounded Verification Choice of technique Properties Unbounded Verification Model Extraction Alloy Encoder SMT-LIBEncoder formula formula formal data model + property Bounded Verification Alloy Analyzer SMT Solver instance or unsat instance or unsat or unknown Results Interpreter ResultsInterpreter FOLEncoder formula TheoremProver proof found or completion found or timeout Results Interpreter Property Verified Property Failed + Counterexample Unknown

22. Alloy Language • A declarative specification language for object modeling • Based on first-order logic • Set-based representation of objects • Defines sets of objects using signatures (sigs) • Defines relations using fields inside the signatures • Add additional constraints about the model as facts • Well-suited for formally specifying data models • Can add assertions to specify properties about the specification

23. Alloy Analyzer • Automated verification of Alloy specifications is undecidable if the domains are not bounded • To ensure decidability, Alloy Analyzer restricts the domains to a finite scope • User-specified • A finite bound on the sizes of the domains • SAT-based bounded verification • Alloy Analyzer translates the Alloy verification query to a Boolean logic formula satisfiability query • Then invokes an off-the-shelf SAT-solver

24. Sample Translation to Alloy class User < ActiveRecord::Base has_one :profile end class Profile < ActiveRecord::Base belongs_to :user end sig Profile {} sig User {} one sig State { profiles: set Profile, users: set User, relation: Profile lone -> one User }

25. Bounded Verification of Data Models Formal data model + Property + Bound • Automatically translate the formal data model extracted from the Active Records and the property to Alloy • User may specify a bound or use the default • Use Alloy Analyzer to perform bounded verification • Possible outputs: • Assertion holds within the given bound • A counterexample proving falsified assertions Bounded Verification Alloy Encoder formula Alloy Analyzer Results Interpreter instance or unsat Property Verifies (within bound) Property Fails + Counterexample

26. Active Records Unbounded with SMT Solver Choice of technique Properties Unbounded Verification Model Extraction SMT-LIB Encoder formula formal data model + property SMT Solver BoundedVerification Alloy Encoder instance or unsat or unknown Results Interpreter formula Alloy Analyzer FOLEncoder instance orunsat Results Interpreter formula TheoremProver proof found or completion found or timeout Results Interpreter Property Verified Property Failed + Counterexample Unknown

27. Satisfiability Modulo Theories (SMT) • SMT-solvers automatically check the satisfiability of first-order formulas with respect to a set of theories • Typical theories include: Linear Arithmetic, Arrays, Bit Vectors, Equality with Uninterpreted Functions • Only implicit universal quantification • Theory of Equality with Uninterpreted Functions. • Language: Variables, Constants, Uninterpreted function symbols, Predicate ‘=‘, and Boolean connectives • Uninterpreted functions have no properties except their signature and functional consistency: a = b => F(a) = F(b) • Example formula: F(x) = F(G(y)) ˅ H(x,y) = 1

28. SMT Solvers • There are many SMT solvers out there that support popular theories • However many of them are not suitable for us • we need support for quantified formulas to handle the property templates • Microsoft’s Z3 supports quantified expressions in the theory of uninterpreted functions • Uses heuristics for eliminating quantifiers in formulas • May return ‘unknown’ during satisfiability check

29. SMT-LIB • Defines a standard input language for SMT Solvers • Defines theories and logics in which formula can be written • Lisp-like format: Specifications are a series of s-expressions • Declare types using declare-sort command • (declare-sort User) • Declare uninterpreted functions using declare-fun command • (declare-fun foo (Domain) Range) • Quantifier commands: (forall )and(exists ) • Add constraints and properties using assert command • Check satisfiability by using (check-sat) command

30. Sample Translation to SMT-LIB class User < ActiveRecord::Base has_one :profile end class Profile < ActiveRecord::Base belongs_to :user end (declare-sort User) (declare-sort Profile) (declare-fun relation (Profile) User) (assert (forall ((p1 Profile)(p2 Profile)) (=> (not (= p1 p2)) (not (= (relation p1) (relation p2) )) ) ))

31. Unbounded Verification using SMT Solvers • Automatically translate formal data model and property into the theory of uninterpreted functions with quantification (SMT-LIB) • Use the SMT solver Z3 to perform satisfiability check • For failing assertion properties, our tool interprets outputs and forms a counterexample • Unknowns (and timeouts) possible since the theory is undecidable Formal data model + Property Unbounded Verification SMT-LIB Encoder formula SMT Solver instance or unsat or unknown Results Interpreter Property Verified Property Failed + Counterexample Unknown

32. Active Records Unbounded Verification Choice of technique Properties Unbounded Verification Model Extraction SMT-LIBEncoder formula formal data model + property SMT Solver BoundedVerification Alloy Encoder instance or unsat or unknown ResultsInterpreter formula Alloy Analyzer FOL Encoder instance orunsat Results Interpreter formula Theorem Prover proof found or completion found or timeout Results Interpreter Property Verified Property Failed + Counterexample Unknown

33. FOL Theorem Provers • Rails data models and properties are expressible in first-order logic (FOL) with equality and quantifiers • Note that this is an undecidable theory • There are automated theorem provers for first-order logic • They use search strategies to find proofs • However due to undecidability of the FOL they cannot always give a definite answer • We use the FOL theorem prover, Spass

34. Modeling Active Records using FOL • Model object types by declaring a unary predicate • Returns true if an object is a member of that class • Model relations between data objects using binary predicates • Returns true if the two objects are related • Axioms are used to express constraints on the data model, e.g. • Specifying cardinality, dependency, and transitivity constraints on relations • Specifying that predicates denoting classes not related by inheritance are mutually exclusive • Conjectures are used to model the property to be checked • To verify the property holds on the data model, send the following formula to the theorem prover: axioms => conjecture

35. Sample Translation to Spass list_of_symbols. sorts[Profile, User]. predicates[(relation, 2)]. end_of_list. list_of_formulae(axioms). formula(forall([Profile(a)], not(User(a)))). formula(forall([User(a)], not(Profile(a)))). formula(forall([a, b], implies( relation(a, b), and(Profile(a), User(b))))). formula(forall([a, b1, b2], implies( and(relation(b1,a), relation(b2,a)), equal(b1,b2)))). formula(forall([a, b1, b2], implies( and(relation(a,b1), relation(a,b2)), equal(b1,b2)))). formula(forall( [Profile(a)], exists([b], relation(a, b)) )). end_of_list.

36. Unbounded Verification of Data Models using Theorem Provers Formal data model + Property • Automatically translate formal data model and property into Spass’s input language (FOL) • Use the theorem prover Spass to see if formula provable • Interpret results to determine whether property holds • Does not produce counterexamples since theorem provers are not designed to do so • If Spass times out (due to undecidability of theory), iDaVer returns ‘Unknown’ Unbounded Verification FOL Encoder formula Theorem Prover Results Interpreter proof found or completion found or timeout Property Verified Property Failed Unknown

37. Outline • Motivation • Overview of Approach • Rails Data Models • Property Templates • Verification Techniques • Case Study • Conclusions and Future Work

38. Case Study • LovdByLess, a social networking application • LOC: 3787 • iDaver input: • Path of the directory containing the Active Record files • Name of the file containing the properties to check (Expressed using property templates!) • Verification technique • NumberActive Record files: 13

39. Case Study • Check: alwaysRelated[Photo, Profile] • Solver: Spass + Unbounded verification − No sample instance − May report unknown or timeout

40. Case Study • Check: someUnrelated[ForumTopic, ForumPost] • Solver: Z3 + Unbounded verification + Sample instance − May report unknown or timeout

41. Case Study • Check: deletePropagates[Profile, Photo] • Solver: Alloy

42. Case Study • Check: deletePropagates[Profile, Photo] • Solver: Alloy + Counterexample data model instance + Always returns a result (for small domains) − Bounded − Slower

43. Summary of Technique Pros and Cons *In particular, Spass was slightly faster than Z3 in our experiments and timed out less frequently

44. Outline • Motivation • Overview of Approach • Rails Data Models • Property Templates • Verification Techniques • Case Study • Conclusions and Future Work

45. Conclusions and Future Work • It is possible to extract formal specifications from MVC-based data models and analyze them • We were able to find data model errors in real-world applications using some of these techniques (ISSTA’11, ASE’12) • Integration of multiple automated verification tools makes overall approach more flexible • Property templates simplify property specification • We have some recent work on • automated property inference (ISSTA’13) • analyzing actions that update the data store (submitted) • Main goal: Verifiable data model specification!

46. Questions?

47. Related Work • Verification of Web Applications • [Krishnamurti et al, Springer 2006 ] focuses on correct handling of the control flow given the unique characteristics of web apps • Works such as [Hallé et al, ASE 2010] and [Han et al, MoDELS 2007] use state machines to formally model navigation behavior • In contrast to these works, we focus on analyzing the data model • Formal Modeling of Web Applications • WebAlloy [Chang, 2009]: user specifies the data model and access control policies; implementation automatically synthesized • WebML [Ceri et al, Computer Networks 2000]: a modeling language developed specifically for modeling web applications; no verification • In contrast, we perform model extraction (reverse engineering)

48. Related Work • Verification of Ruby on Rails applications • Rubicon [Near et al, FSE 2012] verifies the Controller whereas we verify the Data Model • Requires manual specification of application behavior, whereas we verify manually written properties • Limited to bounded verification • Data Model Verification using Alloy • [Cunha and Pacheco, SEFM 2009] maps relational database schemas to Alloy; not automated • [Wang et al, ASWEC 2006] translates ORA-SS specifications to Alloy, and uses the Analyzer to produces instances of the data model to show consistency • [Borbar et al, Trends 2005] uses Alloy to discover bugs in browser and business logic interactions

49. Related Work • Unbounded Verification of Alloy Specifications using SMT Solvers • [Ghazi et al, FM 2011], approach not implemented • More challenging domain since Alloy language contains constructs such as transitive closures • Specification and Analysis of Conceptual Data Models • [Smaragdakis et al ASE 2009, McGill et al ISSTA 2011] use Object Role Modeling to express data model and constraints • Focus is on checking consistency and producing test cases efficiently • Using Patterns to Facilitate Formal Property Specification • First proposed for temporal logic properties [Dwyer et al ICSE 1999] • The templates we present are not temporal and are specific to data model analysis