Computing Education Research

1. Computing Education Research Anders Berglund Informationsteknologi Uppsala universitet Uppsala anders.berglund@it.uu.se

2. Computing education research at Uppsala University • How do our students understand computer science concepts? • How to teach computer science?

3. Computing Education Research This talk • What can we say about how grading in an internationally distributed project-based course? and • Why and how to do research in computing education? My secret agenda: Inspire to Computing Education Research Two parallel stories

4. Computing Education Research • Background • A research approach? • Phenomenography • Results concerning grading • The setting • The teachers’ grades • The students’ peer evaluation • The experience of being graded • Results concerning grading • Summary • Computing Education Research • Some research results • Choosing a research approach

5. What does it mean to learn something? Unfortunately (?)…. • General case: A “meaningless” question • It all depends on “what you mean by learning” or “how you see things”

6. Quantitative/Qualitative research • Quantitative research is grounded on • “… the assumption that features of the social environment constitute an objective reality … collecting numerical data on observable variables” • Qualitative research is grounded on • “… the assumption that individuals construct a social reality in the form of meanings and interpretations. … studying … intensively in natural settings” Implications for the role of the researcher, the concept of evidence, interpretation etc. (Gall, Borg & Gall, 1996)

7. A research approach/methodology/ framework • Offers a way to perform research in learning • Organizes “ways to see things” • A lens with a certain focus • With a specific research approach: Some issues get clearer, others blurred • Offers theoretical stand on learning, ways to see possibilities and limitations, opens to communicate with other researchers etc.

8. Research approach/methodology/frameworkPhenomenography (Marton & Booth, 1997) • Takes the learners’ perspective. • Aims at analysing and describing the variation in students’ experience (understanding, learning). • A empirical, qualitative research approach • Data is often collected through interviews • Outcome: A few qualitatively different ways, in which something is understood within a student cohort Examples: TCP, experience of being graded

9. Phenomenography The students study TCP The researcher studies the different ways in which the students understand TCP Researcher

10. Why the learners’ perspective? • A teacher’s understanding of how her students understand and learn about something (CS concepts, for example TCP) is a good tool for improving teaching. • A change that is not perceived as “good” by the students does not improve learning. • Example: Grades are not the driving force for most students who take (a certain) project course.

11. Computing Education Research • Background • A research approach? • Phenomenography • Results concerning grading • The setting • The teachers’ grades • The students’ peer evaluation • The experience of being graded • Results concerning grading • Summary • Computing Education Research • Some research results • Choosing a research approach

12. Background: The Runestone initiativeA project course in computer systems • A project-based course in distributed systems, real-time programming and computer communication. • Third/fourth year students majoring in CS. • International collaboration for students who do not go on exchanges. • Experience of collaboration over ICT tools.

13. Project course in computer systemsThe Runestone project USA Sweden Communication by e-mail and chat • 3 + 3 advanced CS students per team • 16 teams in total • No lectures • Tutoring by e-mail and chat

14. Student project • Student project: Produce a software system to control a (modified) Brio labyrinth from any Web-browser. • The task demands computer communication solutions. • The task requires collaboration within the team of 6.

15. Grading in Runestone • Both the process and the project are graded, in relation to the teams’ own plan • Process grade is based on weekly meetings • Components of grade: • Team performance, in relation to the team’s own plan • Individual contribution • Peer evaluation • The instructor’s decision. • Team members are graded by “their” instructor • Different grading schemes in Sweden and US • Sweden: pass/fail • US: A to E Problematic???

16. Analysing the grading in Runestone • Teachers’ distribution of grades (quantitative) • Peer evaluation The students’ evaluation of each others’ contributions (quantitative) • Students’ experienced purpose of being graded (qualitative)

17. Grades • Grades awarded by the instructors, according to the Runestone scheme (Max = 100, Pass ≈ 60)

18. Peer evaluation • Each student awarded USD 120.- to his team-mates Then, what is the driving force?

19. The experienced purpose of being graded

20. Results on grading • Getting a good grade is not the driving force for most students in this project. • ”Me in the team” or ”My team in front of other teams” is often important. • How generalizable are these results? • Research in computing education research are normally situationally bound • How can we use this in our teaching? • Both ”hard” results and insights gained by doing the research are useful.

21. Computing Education Research • Background • A research approach? • Phenomenography • Results concerning grading • The setting • The teachers’ grades • The students’ peer evaluation • The experience of being graded • Results concerning grading • Summary • Computing Education Research • Some research results • Choosing a research approach

22. On Computing Education Research • Theoretically sound research in students’ learning in Computing can serve to improve teaching. • Different research approaches offer various contributions to our understanding of students’ learning. • The perspective on ”reality”, what can be studied, what can be known, what the researcher’s role is, how research is performed etc. varies. This talk: Some examples of qualitative research

23. Example, research approach Constructivism • A family of traditions • Jean Piaget, 1896 - 1980 • Reality is rejected or irrelevant • Knowledge is constructed by each individual • No firm methodology • Passive learning will fail • Extremely influential in school teaching

24. Empirical results from constructivism in CS Education • Students construct rules for parameters. They are only sometimes successful. (Fleury, 1991) • Students construct their own understanding of variables. (Paz, 1996; and others) • Software visualization in itself does not help students understanding (Mulholland, 1997)

25. Applications of constructivism in Computer Science Education • Think twice when using visualizations • Explicitly teach the model of the computer • Don’t start with abstractions • Teach planning, teach to avoid “bricolage” • Don’t run to the computer • Organize “closed labs” (from Ben-Ari, 2001)

26. Example, research approach: A socio-cultural perspective Thinking/Learning is not influenced by the environment ⇔ Thinking/Learning is an interaction between the individual and the environment • A family of traditions in research into learning • Common “source of inspiration”: Lev Vygotsky (1896 – 1934)

27. Applications of the soci-cultural tradition • Why do teams of students interpret a programming task so differently? (Holland & Reeves, 1996) • Why do our students hand in “incorrect” programs? (Ben-David Kolikant, 2005) • The example: Open source community - • Linux

28. Example, research approach: Critical enquiry • Critical enquiry (research with “a mission”, often to address power imbalances) • Feminist research • Cannot be defined in ontological stand or research methodology • Instead, it questions fundamental principals and values and aims to implement changes.

29. Feminist research, an example of critical research • Are there factors within computer science itself, that preserves the currently dominating gender structure? (Björkman & Trojer, 2002) “We consider it of vital and decisive importance that gender research is done from within computer science.” “Such research on the core of computer science and its knowledge production would serve to enrich computer science as well as education within computer science.”

30. Selecting a research approach Why phenomenography? A personal view • Appropriate for the research questions • “Complex” “answers” desirable • Statistical “answers” hard or impossible to get • “Close to” learners • Students’ perspective • A way for students to “talk” to teachers/organisers • Data stems from individuals • Computer science is in focus • Results “talk to” computer scientists • Supports deployment of results in teaching • Relevant for recognition of CSE research within CS • Competence at hand/tradition • Shirley Booth • I like it  Create a network! Read!

31. Read! Literature • Clancy, M., Stasko, J., Guzdial, M., Fincher, S., & Dale, N. (2001). Models and Areas for CS Education Research. Computer Science Education, 11(4), 323-341 • Fincher, S., & Petre, M. (2004) Computer Science Education Research, London, UK: Taylor & Francis • Berglund, A., Daniels, M. and Pears, A. (in press). Qualitative Research Projects in Computing Education Research: An Overview. To appear in the Proceedings of the 8th Australasian Computer Science Education Conference, Hobart, Australia. • Berglund, A. 2005. Learning computer systems in a distributed project course: The what, why, how and where. Acta Upsaliensis Universitatis. Uppsala Dissertations from the Faculty of Science and Technology 62

32. Create a network! CeTUSS • CeTUSS ska verka för anpassning av teknikutbildning till samhällets och studenternas behov. • Genom att utveckla och sprida information om lärandemiljöer som är • Personligt meningsfulla • Socialt relevanta • Gränsöverskridande • Baserad på samarbete (lokalt och internationellt) • Workshops, kurser etc.