Engaging Students in Specification and Reasoning: Hands-On Experimentation and Evaluation

1. Engaging Students in Specification and Reasoning: Hands-On Experimentation and Evaluation

2. Murali Sitaraman, Jason Hallstrom, Jarred White, Svetlana Drachova-Strang, Heather Harton, Dana Leonard: Clemson University School of Computing. Joan Krone : Denison University. Rich Pak: Clemson University Psychology. Supported by US NSF #3407328

3. Goal • Reasoning Across the Curriculum • Supporting Tools • Supporting Methods

4. Software Engineering Course • Usual Topics • Requirements analysis • Design and specification • Component-based implementation • Quality assurance • Formal Reasoning

5. Objectives • Read formal specifications • Create test points from the specs • Use component specifications to build larger systems • Work in teams • Carry out formal verification of components • Use automated rules

6. Methods • Collaborative learning • Teams of 2 to 4 members • Read specs • Implement specs • Verify implementations • Build larger systems.

7. Stack Example • Model as a mathematical string of generic entries with an upper limit of Max_Depth. • Initialize as empty string. • Oper Push(eval E: Entry; updates S: Stack); requires |S| < Max_Depth; ensures S = E o #S;

8. Reading Specs • Give students various strings representing abstract stacks and tell what Max_Depth is. • Ask them to show what happens when a Push is called with particular data. • Ex: S = rtb Max_Depth = 5. Push(c,S) Push(w,S)

9. Queue Example • Model as a string of generic entries. • Primary operations: enqueue, dequeue, length. • Secondary operation

10. Test Cases for Rotate • Oper Rotate(updates Q: Queue, restores n: Integer); • requires |Q| <= n; • ensures there exists S, T: Str(Entry) such that |S| = n and #Q = S o T and Q = T o S; • Input: Q = <17, 1, 6, 9, 20> n = 3; • Output: Q = <9, 20, 17, 1, 6> n = 3; • #Q = Input Q, Q = output Q • S = <17, 1, 6> T = <9, 20>

11. Tools • Specification Understanding: Expose students to formal specifications for reasoning. • Team Software Development with Formal Contract Specifications: Enforce contract programming principles. • Verification Condition Generation: Generate clauses equivalent to the correctness of the implementation.

12. Using the Tools • http://www.cs.clemson.edu/~resolve

13. Survey Questions • (S3) The difficulty in understanding and modifying a 10,000 line software system has more to do with the style I which the software is written, and less to do with how smart I am. • (S11) Testing software thoroughly is the most important way to ensure correctness.

14. Results for S11

15. (S18) When working in teams, natural language descriptions of the different components are sufficient for communication among team members. • (S20) Reasoning about programs involving components requires a thorough understanding of pointers and/or references. • Entire survey at http://www.cs.clemson.edu/~resolve

16. Results for S18

17. All Results • http://www.cs.clemson.edu/~resolve/

18. RSRG: Tools for Teaching Mathematical Specification and Reasoning • We are currently documenting tools and results for integrating mathematical specification and reasoning in undergraduate classes with support from an NSF CCLI grant.  For an overview as well as how we have introduced the methods and tools in a software engineering course, please see this technical report: • M. Sitaraman, J. O. Hallstrom, J. White, S. Drachova-Strang, H. K. Harton, D. Leonard, J. Krone, and R. Pak, Engaging Students in Specification and Reasoning: “Hands-On” Experimentation and Evaluation, Technical Report RSRG-08-05, School of Computing, Clemson University, Clemson, SC 29634-0974, December 2008, 6 pages. • For materials on how the principles have been introduced in a sophomore level course on object-oriented development with Java, please visit CP SC 215 course links under teaching at these sites: • Fall 2008: http://www.cs.clemson.edu/~murali • Spring 2009: http://www.cs.clemson.edu/~jasonoh

19. Specification Understanding: One of the software engineering objectives of all undergraduate curricula in computing is to at least expose students to formal specifications of behavior.  These specifications are presented formally in mathematical logic and can be written in a variety of formal specification languages.  For an overview and to download the tool, click here.   Details of the tool are discussed in the following paper: • D. Leonard, J. Hallstrom, and M. Sitaraman, “Injecting Rapid Feedback and Collaborative Reasoning in Teaching Specifications,” ACM SIGCSE 2009 Procs. at http://db.grinnell.edu/sigcse/sigcse2009/Program/Program.asp • Team Software Development with Formal Contract Specifications: RESOLVE analyzer and RESOLVE-to-Java translator is available here to enable formal contract-based software development.  For a technical summary of how we have introduced the ideas in a software engineering course, please see this technical report:: • S. Drachavo-Strang, Teaching Design-By-Contract with Formal Specifications, which is available here. • Verification Condition Generation: For an overview of methods and tools to teach proofs of correctness and formal mathematical reasoning about correctness of programs, please see here.  For technical details on a push-button verification system, please see this technical report:  • M. Sitaraman, B. Adcock, J. Avigad, D. Bronish, P. Bucci, D. Frazier, H. M. Friedman, H. Harton, W. Heym, J. Kirschenbaum, J. Krone, H. Smith, and B. W. Weide, Building a Push-Button RESOLVE Verifier: Progress and Challenges, Technical Report RSRG-09-01, School of Computing, Clemson University, Clemson, SC 29634-0974, January 2009, 34 pages.

20. Demos • http://resolve.cs.clemson.edu/

21. Summary • Importance of Reasoning across the Curriculum • Tools to Support Reasoning • Collaborative Pedagogy includes collaboration between students and between students and faculty