Evaluating a Formal Methods Technique via Student Assessed Exercises

1. Evaluating a Formal Methods Technique via Student Assessed Exercises Alastair Donaldson, Alice Miller University of Glasgow

2. Outline • Need for evaluation • SymmExtractor • Examples for evaluation – student solutions • Ethical approval • Documentation process • Evaluation results • Future evaluation FM-Ed

3. Need for evaluation • Automated FM tries to solve intractable or undecidable problems • Model checking – quickly becomes intractable • Parameterised model checking – undecidable • Progress made by restricting application domain • “Applicable to C programs without pointers” • “The system must have a fixed no. of components” FM-Ed

4. Need for evaluation • Is restricted application domain still useful? • Need evaluation with users of technique • Can tool do what they want? • Can they change needs easily to fit technique? • Example: symmetry reduction for model checking • Automatic symmetry detection • Exploiting symmetry • Both computationally difficult • Both easy to solve when application domain limited FM-Ed

5. Symmetry reduction for model checking • Replication in topology of concurrent system → replication (symmetry) in state-space • State space partitioned into equivalence classes • Only need to search one state per class • System comprised of n components • Equivalence classes may be as large as n! • Model checking is automatic: • Symmetry must be automatically detected FM-Ed

6. SymmExtractor • Detects symmetry in Promela specifications, for verification with SPIN • Extracts static channel diagram of a specification • Computes symmetries of static channel diagram • Derives state-space symmetries from these • Specification must satisfy certain restrictions • Need evaluation to see how restrictions affect applicability of SymmExtractor FM-Ed

7. Examples for evaluation: submissions to student assessed exercise • Modelling reactive systems • Final year FM course at Glasgow • Main focus: model checking with SPIN • Assessed exercise 2004/2005 • Specification and verification of (3 versions of) a 2-user telephone exchange • Intuitively, underlying state spaces should exhibit one non-trivial symmetry • Can SymmExtractor detect this? FM-Ed

8. Ethical approval • Followed Glasgow Ethics Code and gained ethical approval from faculty • Obtained signed consent forms from all participating students • Ensured evaluation took place after formal assessment of submissions • 17 (out of 35) students gave approval • 51 Promela specifications for input to SymmExtractor FM-Ed

9. Documentation process • For each specification, documented • Size of unreduced state-space (SPIN) • State-space symmetries computed explicitly (SPIN-to-GRAPE) • Symmetry breaking features (experimenter) • Violations of SymmExtractor’s restrictions (SymmExtractor) • Modifications required to fix violations (experimenter) • Symmetries computed by SymmExtractor (SymmExtractor) • Size of quotient state space (TopSPIN) FM-Ed

10. Results • Approx. half specifications had symmetry breaking features • Set of modelling guidelines to avoid common pitfalls • After fixing these: • 23 specifications – symmetry detected • 13 specifications – violated restrictions, needed minor modification for symmetry to be detected • 7 specifications – medium modifications • 8 specifications – major modifications FM-Ed

11. Results • Minor modifications – violation of restrictions which could easily be lifted • Medium modifications – problems due to use of global variables, which SymmExtractor could be modified to cope with • Major modifications – problems involving way arrays indexed by process identifiers are accessed • Serious usability problem due to restrictions • requires further research effort to fix FM-Ed

12. Future evaluation • Benefit here was one-way: students’ assessments used to aid our research • Evaluation took place after completion of course • May be possible to run evaluations during the course • Students apply symmetry detection/reduction to own programs and report results FM-Ed