200 likes | 291 Views
Exploring the Relationship Between Novice Programmer Confusion and Achievement. By: Diane Marie Lee Ma. Mercedes Rodrigo Ryan Baker Jessica Sugay Andrei Coronel. Affective States and Achievement. Recent studies have illustrated the relationships between affective states and achievement
E N D
Exploring the Relationship Between Novice Programmer Confusion and Achievement By: Diane Marie Lee Ma. Mercedes Rodrigo Ryan Baker Jessica Sugay Andrei Coronel
Affective States and Achievement • Recent studies have illustrated the relationships between affective states and achievement • Negative affective states have negative impact on student’s achievement (Craig et al, 2006; Rodrigo, 2009; Lagud, 2010)
Confusion • Double-edged/ Dual Nature (D’Mello 2009) • Harmful • Helpful
Goal • Discovery-with-models approach to finding the relationship between novice programmer confusion and achievement
Data Collection • 149 students enrolled in CS21a – Introduction to Computing I • Four lab sessions • BlueJ IDE • BlueJ Plug-in (Jadud and Henriksen, 2009)
Data Collection • Compilation logs = all submissions made to the compiler • Compilation logs include • Computer number • Timestamp • Code • Error message (if any) • And many more!
Data Collection • Total of 340 student-lab sessions • Total of 13,528 compilation logs collected
Data Labeling • Sorted the compilations by student and by Java class name • Grouped the compilations into clips • Clips = 8 compilations • Total: 2,386 clips • Raters were asked to label a sample of 664 clips
Data Labeling • Used low-fidelity text replays • Maintains good inter-rater reliability and efficient in aiding coders to label student disengagement (Baker et al. 2006) • Labels • Confused • Not Confused • Bad Clip • Cohen’s Kappa between raters: 0.77
Data Labeling • Filter out “bad clips” • Remove clips where raters disagreed on the label • Left with 418 clips for model construction
Model Construction • Used RapidMiner version 5.1 • Used J48 Decision Trees • Features were mined from the clips
Model Construction • Feature set used: • Average time between compilations • Maximum time between compilations • Average time between compilations w/ errors • Maximum time between compilations w/ errors • Number of compilations w/ errors • Number of pairs consecutive compilations ending w/ the same error
Data Relabeling • Model was coded as a Java program • Had the program relabel all the 2,386 clips • Generated three sets of confused-not confused sequences • Correlated the percentage of the sequences of each student to their midterm exam scores
Conclusion • Prolonged confusion has a negative impact on student’s performance • Resolved confusion has a positive impact on student’s performance • A certain amount of confusion is needed for learning
On-going Work • Support the incorporation of tools for automatic detection of confusion in computer science learning environments • Redoing the sampling and clipping method
Thank you Questions?