Stephen Cummins 1 Liz Burd 2 and Andrew Hatch 2 School of Computing University Campus Suffolk 1

1. Using Sharable Feedback Tags to Support Programming StudentsSharing good ideas in CS teaching and learning eventWednesday 6th April 2011 Stephen Cummins1Liz Burd2 and Andrew Hatch2 School of Computing University Campus Suffolk1 Durham University2

2. Overview • Context • The motivation • Overview of terminology • SWATT: Tag based feedback for source code • Overview of Research • Student perceptions and behaviors • Sentiment analysis of feedback tags • Conclusion • Some advice based on the research conducted • Future Work

3. Motivation: The Importance of Feedback • Feedback is crucial to students’ learning (Rowntree1987). • Providing feedback in context is important.

4. Motivation: The Challenges of Learningto Program • Learning programming is difficult. • It takes 10 years to turn a novice into an expert (Winslow1996). • Attempts to solve the problems in teaching programming. • New IDE’s for conventional languages. • New programming languages. • New approaches to assessment.

5. Overview of the Terminology • Tagging – The association of words or short phrases to a physical or virtual entity, providing some extra meaning. • Folksonomies – A system where by users generate tags that apply to one or more virtual entities. • Web 2.0Systems– E.g. wikis, blogs and social networking applications (Facebook). An example tag cloud Tag a photograph with metadata in facebook.

6. SWATT: SoftWare Assessment Through Tagging • New system of tag based feedback. • Based on Web 2.0 ideas. • Enables tag cloud style feedback summaries. • A more interactive style of feedback.

7. SWATT: Feedback Tags as Sharable Learning Resources (1) • Approach: Based on Web 2.0 ideas. • Tag cloudsfor summaries. • Sharable feedback & associated source code. • Learning from others’ mistakes. • More feedback for each student. • Tag frequency analysis. • Allows ‘zooming in’ to deliver in-context feedback.

8. SWATT: Feedback Tags as Sharable Learning Resources (2) • Benefits: • Simple and requires no special training. • Reusable. • Visualisation and analysis tools available. • Sharable. • Disadvantages: • Ambiguity. • Reliance on context information. • No data sanitation.

9. Overview of Research (1) • 3 years of investigation involving 3 different cohorts. • Cohort 1: Summative Group work. • Cohort 2: Formative Individual Work. • Cohort 3: Summative Individual Work. • An important factor is the timing and sequence of feedback release.

10. Overview of Research (2) • Primary research focus: • Student perceptions of tag based feedback. • Student perceptions of sharing feedback. • Secondary research focus: • Sentiment analysis of tag based feedback. • Thematic analysis of tag based feedback.

11. Results: Student Feedback • ‘‘You can see a general theme [from the tag cloud] as to how you’ve done instantly but then drill into certain areas to get more information’’.– First year undergraduate. • 100% of students viewed their feedback. • 94% of respondents thought that the tag based feedback was “Easy” or “Very Easy” to understand. • 81% of respondents said this type of feedback was “Very Useful” or “Useful” in helping them to improve their work.

12. Results: Observed Student Behaviours • Student behaviours were captured and characterised as being one of the following categories: • Explorers. • Informal sharers. • Librarians. • One-off viewers (non-sharers). • One-off viewers (sharers). • Surface users.

13. Results: Motivations For and Against Sharing • 43% of students in all cohorts opted to share their feedback. • The following reasons were provided (Cummins, 2011):

14. http://www.nactem.ac.uk/ Sentiment Analysis • Sentiment Analysis. • Is whether text has a positive, negative or neutral underlying emotion. • Manual and automated methods of analysis. • Commercial Applications. • Applications to feedback. • No one likes entirely negative feedback. • Feedback sandwiches.

15. Sentiment Analysis (2) • The nature of textual feedback makes it ambiguous. • Humans perceive things differently. • Investigations included the perceptions of students, examiners and NaCTeM. • General Results: • Examiners identified more tags as being neutral. • Students identified more positive and negative tags than examiners.

16. Sentiment Analysis (3) • There are clear issues with how people perceive the sentiment of feedback tags. • Some tags have specific meanings within the discipline, for example: • “high coupling” • “duplicate code”

17. Conclusions: Good ideas • Advice: • Do not release marks before feedback. • Release the written feedback asap. • Consider the sentiment of the feedback. • Encourage students to investigate / interact with their feedback. • Students generally perceived feedback sharing as being useful (those who opted-in).

18. Future Work • Investigate the inclusion of sentiment information into tag clouds. (Colour coding) • Investigate the use of tag based feedback in other institutions and levels of education. • Extend tag based feedback to other programming languages / CS disciplines.

19. References • Cummins, S., Burd, L. & Hatch, A. (2011). Investigating shareable feedback tags for programming assignments. Computer Science Education, 21(1), pp81-103. doi:10.1080/08993408.2011.557584 • Cummins, S., Burd, L., and Hatch, A. (2010) 'Using Feedback Tags and Sentiment Analysis to Generate Shareable Learning Resources', Proceedings of the 10th IEEE International Conference on Advanced Learning Technologies, Sousse, Tunisia, July 2010 • Gee, Thomas C., “Students Responses to Teacher Comments”, Research in the Teaching of English 6(2), 1972 • NaCTeM, http://www.nactem.ac.uk/ • Rowntree, Derek, “Assessing Students: How Shall We Know Them?”, Nichols Pub Co, 1987 • Winslow, Leon E, “Programming pedagogy: a psychological overview”, ACM SIGCSE Bulletin 28(3),1996