Using software metrics in educational environment

1. Ivan Pribela, ZoranBudimac, GordanaRakić Using software metrics in educational environment

2. Content • Motivation • Assessment process • A case study • Conclusion

3. Motivation • Automated assessment systems are very helpful • Reduce load for instructors • Quicker feedback for students • But have limited scope • Black-box (input-output) tests, unit tests • Lately code style checks • Software metrics are not used • Utilized only by plagiarism detection • Not to help students improve their skills

4. Motivation (cont.) • Software metrics can tell a lot about a program • Have highly diverse methodologies and objectives • Difficult to define quality for various programs in uniform way • What which metric means for particular assignment? • What metrics matter the most? • We explore the usage of software metrics in automated assessment • To cover testing of aspects that demand instructor attention • Algorithm complexity • Number and size of programming units and functions

5. Assessment process Student solution Test report

6. Student solution Student solution MODULE Triplets1; ... VAR x, y, z: INTEGER; BEGIN FOR x := 1 TO Gr DO FOR y := 1 TO Gr DO FOR z := 1 TO Gr DO IF x*x + y*y = z*z THEN WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END END END END Triplets1.

7. Parsing the solution Student solution • eCST generator(the SMIILE tool) • Recognizes input language • Calls appropriate scanner & parser • Generates an eCSTrepresentation • Enriched Concrete Syntax Tree • Modified Concrete Syntax Tree • Enriched with marker nodes • Unit: class, module… • Loop Statement: for, while, repeat… • Branch Statement: if, case, switch… • Independent of programming language • Basis for calculation of software metrics Parse eCST

8. Measuring eCST Student solution • Metrics calculator(the SMIILE tool) • Software metrics tool in the development • Analyzes the eCST representation • Calculates software metrics • Exports results in XML file • Final result is one XML file with values of all calculated metrics • Current prototype of SMIILE tool supports several software metrics • Lines of Code • CyclomaticComplexity • … Parse eCST Metric values in xml Measure

9. Metric values in XML Student solution <?xml version="1.0"?> <sourceElement endLine="24" firstLine="1" name="" type=""> <metrics loc="24" cc="4"/> <sourceElement endLine="24" firstLine="1" name="Triplets1" type="CONCRETE_UNIT_DECL"> <metrics loc="24" cc="4"/> <sourceElement endLine="23" firstLine="9" name="FOR" type="LOOP_STATEMENT"> <metrics loc="15" cc="4"/> <sourceElement endLine="22" firstLine="10" name="FOR" type="LOOP_STATEMENT"> <metrics loc="13" cc="3"/> <sourceElement endLine="21" firstLine="11" name="FOR" type="LOOP_STATEMENT"> <metrics loc="11" cc="2"/> ... </sourceElement> </sourceElement> </sourceElement> </sourceElement> </sourceElement> Parse eCST Metric values in xml Measure

10. Transform XML to properties Student solution • XSL Transformator • Uses XSLT stylesheet • Input XML file • Values of calculated software metrics • Output XML file • Can be manipulated easily inside the testing system Parse eCST Metric values in xml Measure Transform Matric values as properties

11. Metric values as properties Student solution <?xml version="1.0" encoding="UTF-8"?> <metrics> <Triplets1> <loc>24</loc> <cc>4</cc> </Triplets1> </metrics> Parse eCST Metric values in xml Measure Transform Matric values as properties

12. Testing the values Student solution • Testing system (Testovid) • Implemented as a framework for running domain specific testers • Domain specific testers • Written as Apache Ant scripts • Using software metrics • Script runs SMIILE tool • Transforms XML file with metrics values • Loads calculated values • Freely uses them for grading, intelligent advice generation… • Final report • Contains advices • Success/failure information • Presented to the student Parse eCST Metric values in xml Measure Transform Matric values as properties Script with metric control values Test Test report

13. Test report Student solution Course: Data structures and algorithms Assignment: Assignment 1 - Pythagorean triplets Student: John Doe Time: 05.09.2012. 11:15:00 Compilation 100% (from 2 points) All is well, no errors. Correctness 33% (from 6 points) Not all triplets were found, check loop boundaries. Optimality 50% (from 2 points) Try using Euclid's formula. -+-+-+-+-+-+-+-+- Total: 5 out of 10 points. Parse eCST Metric values in xml Measure Transform Matric values as properties Script with metric control values Test Test report

14. A case study • Data structures and algorithms course • Testing efficiency of a Modula 2 program • Using cyclomatic complexity metric • detect loop and branch statements • Created a domain specific tester for Testovid • differentiate between typical student solutions • award points accordingly • The assignment • Write a program which prints Pythagorean triplets, positive integer numbers x, y and z for which x2+y2=z2

15. Solution 1: Naive solution MODULE Triplets1; ... VAR x, y, z: INTEGER; zreal: REAL; BEGIN FOR x := 1 TO Gr DO FOR y := 1 TO Gr DO zreal := REAL(Sqrt(LONGREAL(x*x + y*y))); z := TRUNC(zreal); IF zreal = FLOAT(z) THEN WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END END END Triplets1. • Cyclomatic complexity 3 • Efficiency Average • Points 50%

16. Solution 2: Brute force solution MODULE Triplets2; ... VAR x, y, z: INTEGER; BEGIN FOR x := 1 TO Gr DO FOR y := 1 TO Gr DO FOR z := 1 TO Gr DO IF x*x + y*y = z*z THEN WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END END END END Triplets2. • Cyclomaticcomplexity 4 • Efficiency Bad • Points 0%

17. Solution 3: Using Euclid’s formula MODULE Triplets; ... VAR x, y, z, m, n: CARDINAL; BEGIN FOR m := 1 TO Gr DO FOR n := 1 TO m-1 DO x := m*m - n*n; y := 2*m*n; z := m*m + n*n; WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END END Triplets. • Cyclomatic complexity 2 • Efficiency Good • Points 100%

18. Non solution MODULE Triplets4; ... VAR x, y, z, m, n, w, i, temp : CARDINAL; BEGIN w := 1; n := 0; FOR i := 1 TO Gr DO m := n + w; x := m*m - n*n; y := 2*m*n; z := m*m + n*n; WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2); temp := w; w := 3*w + 4*n; n := 2*temp + 3*n END END Triplets4. • Cyclomatic complexity 1 • Efficiency Excellent • Points 100% • Lose points for correctness

19. Summary • The greater the cyclomatic complexity the worst the solution efficiency • Awarded points should be reverse proportional to the cyclomatic complexity • Inefficient solutions – no points • Average solutions – half the maximum points • Efficient solutions – all the points • Instructor should use knowledge and experience to • Choose metrics • Define minimum and maximum metric values • Define awarded points for those cases • Testing system can automatically • Classify student solutions • Grade them accordingly

20. Conclusion • Utilized software metrics in the assessment process • Increased the scope of aspects that can be covered by automatic tests • Platform and programming language independent • Support a wide range of metrics • Left great flexibility in selecting interesting metrics • Can provide hints and advices to students • Added intelligent assistance • Improved student learning experience

21. Thank you for your attention Questions?