1 / 21

Using software metrics in educational environment

Explore the use of software metrics in an educational setting for automated assessment, covering testing aspects that require instructor attention, like algorithm complexity and program size. Learn how metrics can improve student skills.

Download Presentation

Using software metrics in educational environment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Ivan Pribela, ZoranBudimac, GordanaRakić Using software metrics in educational environment

  2. Content • Motivation • Assessment process • A case study • Conclusion

  3. Motivation • Automated assessment systems are very helpful • Reduce load for instructors • Quicker feedback for students • But have limited scope • Black-box (input-output) tests, unit tests • Lately code style checks • Software metrics are not used • Utilized only by plagiarism detection • Not to help students improve their skills

  4. Motivation (cont.) • Software metrics can tell a lot about a program • Have highly diverse methodologies and objectives • Difficult to define quality for various programs in uniform way • What which metric means for particular assignment? • What metrics matter the most? • We explore the usage of software metrics in automated assessment • To cover testing of aspects that demand instructor attention • Algorithm complexity • Number and size of programming units and functions

  5. Assessment process Student solution Test report

  6. Student solution Student solution MODULE Triplets1; ... VAR x, y, z: INTEGER; BEGIN FOR x := 1 TO Gr DO FOR y := 1 TO Gr DO FOR z := 1 TO Gr DO IF x*x + y*y = z*z THEN WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END END END END Triplets1.

  7. Parsing the solution Student solution • eCST generator(the SMIILE tool) • Recognizes input language • Calls appropriate scanner & parser • Generates an eCSTrepresentation • Enriched Concrete Syntax Tree • Modified Concrete Syntax Tree • Enriched with marker nodes • Unit: class, module… • Loop Statement: for, while, repeat… • Branch Statement: if, case, switch… • Independent of programming language • Basis for calculation of software metrics Parse eCST

  8. Measuring eCST Student solution • Metrics calculator(the SMIILE tool) • Software metrics tool in the development • Analyzes the eCST representation • Calculates software metrics • Exports results in XML file • Final result is one XML file with values of all calculated metrics • Current prototype of SMIILE tool supports several software metrics • Lines of Code • CyclomaticComplexity • … Parse eCST Metric values in xml Measure

  9. Metric values in XML Student solution <?xml version="1.0"?> <sourceElement endLine="24" firstLine="1" name="" type=""> <metrics loc="24" cc="4"/> <sourceElement endLine="24" firstLine="1" name="Triplets1" type="CONCRETE_UNIT_DECL"> <metrics loc="24" cc="4"/> <sourceElement endLine="23" firstLine="9" name="FOR" type="LOOP_STATEMENT"> <metrics loc="15" cc="4"/> <sourceElement endLine="22" firstLine="10" name="FOR" type="LOOP_STATEMENT"> <metrics loc="13" cc="3"/> <sourceElement endLine="21" firstLine="11" name="FOR" type="LOOP_STATEMENT"> <metrics loc="11" cc="2"/> ... </sourceElement> </sourceElement> </sourceElement> </sourceElement> </sourceElement> Parse eCST Metric values in xml Measure

  10. Transform XML to properties Student solution • XSL Transformator • Uses XSLT stylesheet • Input XML file • Values of calculated software metrics • Output XML file • Can be manipulated easily inside the testing system Parse eCST Metric values in xml Measure Transform Matric values as properties

  11. Metric values as properties Student solution <?xml version="1.0" encoding="UTF-8"?> <metrics> <Triplets1> <loc>24</loc> <cc>4</cc> </Triplets1> </metrics> Parse eCST Metric values in xml Measure Transform Matric values as properties

  12. Testing the values Student solution • Testing system (Testovid) • Implemented as a framework for running domain specific testers • Domain specific testers • Written as Apache Ant scripts • Using software metrics • Script runs SMIILE tool • Transforms XML file with metrics values • Loads calculated values • Freely uses them for grading, intelligent advice generation… • Final report • Contains advices • Success/failure information • Presented to the student Parse eCST Metric values in xml Measure Transform Matric values as properties Script with metric control values Test Test report

  13. Test report Student solution Course: Data structures and algorithms Assignment: Assignment 1 - Pythagorean triplets Student: John Doe Time: 05.09.2012. 11:15:00 Compilation 100% (from 2 points) All is well, no errors. Correctness 33% (from 6 points) Not all triplets were found, check loop boundaries. Optimality 50% (from 2 points) Try using Euclid's formula. -+-+-+-+-+-+-+-+- Total: 5 out of 10 points. Parse eCST Metric values in xml Measure Transform Matric values as properties Script with metric control values Test Test report

  14. A case study • Data structures and algorithms course • Testing efficiency of a Modula 2 program • Using cyclomatic complexity metric • detect loop and branch statements • Created a domain specific tester for Testovid • differentiate between typical student solutions • award points accordingly • The assignment • Write a program which prints Pythagorean triplets, positive integer numbers x, y and z for which x2+y2=z2

  15. Solution 1: Naive solution MODULE Triplets1; ... VAR x, y, z: INTEGER; zreal: REAL; BEGIN FOR x := 1 TO Gr DO FOR y := 1 TO Gr DO zreal := REAL(Sqrt(LONGREAL(x*x + y*y))); z := TRUNC(zreal); IF zreal = FLOAT(z) THEN WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END END END Triplets1. • Cyclomatic complexity 3 • Efficiency Average • Points 50%

  16. Solution 2: Brute force solution MODULE Triplets2; ... VAR x, y, z: INTEGER; BEGIN FOR x := 1 TO Gr DO FOR y := 1 TO Gr DO FOR z := 1 TO Gr DO IF x*x + y*y = z*z THEN WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END END END END Triplets2. • Cyclomaticcomplexity 4 • Efficiency Bad • Points 0%

  17. Solution 3: Using Euclid’s formula MODULE Triplets; ... VAR x, y, z, m, n: CARDINAL; BEGIN FOR m := 1 TO Gr DO FOR n := 1 TO m-1 DO x := m*m - n*n; y := 2*m*n; z := m*m + n*n; WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END END Triplets. • Cyclomatic complexity 2 • Efficiency Good • Points 100%

  18. Non solution MODULE Triplets4; ... VAR x, y, z, m, n, w, i, temp : CARDINAL; BEGIN w := 1; n := 0; FOR i := 1 TO Gr DO m := n + w; x := m*m - n*n; y := 2*m*n; z := m*m + n*n; WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2); temp := w; w := 3*w + 4*n; n := 2*temp + 3*n END END Triplets4. • Cyclomatic complexity 1 • Efficiency Excellent • Points 100% • Lose points for correctness

  19. Summary • The greater the cyclomatic complexity the worst the solution efficiency • Awarded points should be reverse proportional to the cyclomatic complexity • Inefficient solutions – no points • Average solutions – half the maximum points • Efficient solutions – all the points • Instructor should use knowledge and experience to • Choose metrics • Define minimum and maximum metric values • Define awarded points for those cases • Testing system can automatically • Classify student solutions • Grade them accordingly

  20. Conclusion • Utilized software metrics in the assessment process • Increased the scope of aspects that can be covered by automatic tests • Platform and programming language independent • Support a wide range of metrics • Left great flexibility in selecting interesting metrics • Can provide hints and advices to students • Added intelligent assistance • Improved student learning experience

  21. Thank you for your attention Questions?

More Related