Assessing critical thinking through scaffolded asynchronous online discussion

1. Assessing critical thinking through scaffolded asynchronous online discussion Yiong Hwee, TEO National Institute of Education Nanyang Technological University, Singaporeyhteo@nie.edu.sg Evaluations & Assessment Conference Adelaide November 24-25, 2003

2. Outline • Introduction • Collaborative critique framework • Socratic questioning in Online Discussion • Using Knowledge Community to scaffold & assess thinking • Conclusion Assessing critical thinking through scaffolded asynchronous online discussion 2

3. Singapore Education Initiatives • Thinking Programme & Project Work (PW) in schools • Masterplan for IT in Education (1997-2002) • In-service teacher training at NIE • Hypermedia design module (Jul 03; 8 weeks) • Design a hypermedia project • Engage in Computer-supported Collaborative Learning (CSCL) using Knowledge Community Assessing critical thinking through scaffolded asynchronous online discussion 3

4. Collaborative Critique Framework • Value of critique of design projects • Use examples of professional work to establish standards • Use examples of students work to define high and low quality work (Thomas & Mergendoller, 2000) • Why use asynchronous online critique? Assessing critical thinking through scaffolded asynchronous online discussion 4

5. Research Question • What types of thinking skills and levels of information processing are exhibited by in-service teachers in a scaffolded online group discussion? Assessing critical thinking through scaffolded asynchronous online discussion 5

6. Collaborative Critique Framework In-service teachers critique past year student hypermedia project in online discussion: 5 – step critique process 1. Identify project process, audience & expertise 2. Justify strengths and suggest improvements 3. Justify weaknesses and suggest improvements 4. Weigh alternatives 5. Transfer to own project Assessing critical thinking through scaffolded asynchronous online discussion 6

7. Thinking Skills • Dimensions of Thinking (Marzano, 1998) • Map of Thinking Domain Framework (Swartz & Parks, 1994) • Socratic Questioning Method (Socrates; Richard Paul, 1995) All reasoning has a purpose, is based on assumptions, done from some point of view, and based on information & evidence; All reasoning leads somewhere, has implications and consequences Assessing critical thinking through scaffolded asynchronous online discussion 7

8. Socratic Thinking Prompts Assessing critical thinking through scaffolded asynchronous online discussion 8

9. Using Knowledge Community to Scaffold Thinking Assessing critical thinking through scaffolded asynchronous online discussion 9

10. Using Knowledge Community to Scaffold Thinking Assessing critical thinking through scaffolded asynchronous online discussion 10

11. Using Knowledge Community to Assess Participation &Thinking • Evaluation of online participation • Statistics on reading & writing behaviour Assessing critical thinking through scaffolded asynchronous online discussion 11

12. Evaluation of types of Socratic thinking • Message units according to Socratic Thinking Types Assessing critical thinking through scaffolded asynchronous online discussion 12

13. Evaluation of types of Socratic thinking • Sorting by Scaffold Prompts Assessing critical thinking through scaffolded asynchronous online discussion 13

14. Evaluation of level ofinformation processing Henri (1992) method of coding • Critical thinking – surface level • “I do not agree that the quit button is redundant” • Critical thinking – in depth level • “A different viewpoint is that the choice of content is excellent. Pupils can relate to the content chosen since it depicts pupil’s own daily experience i.e. doing various activities in a day.” Assessing critical thinking through scaffolded asynchronous online discussion 14

15. Conclusion Preliminary Analysis • Promising potential of ‘scaffolding prompts’ of KC reveals that • More students take a stand • Comments were focused • Comments were longer, detailed, with justifications • Software-scaffolding enable students to use all thinking types BUT they are weak in ‘Probing Assumptions’ (4.3%) & ‘Probing Implications & Consequences’ (5.8%) Assessing critical thinking through scaffolded asynchronous online discussion 15

16. Conclusion • Educators can help students to improve their thinking in certain areas where they are weak and further study can help us design better thinking scaffolds in online discussion forums Assessing critical thinking through scaffolded asynchronous online discussion 16

17. Future Research • How does scaffolding prompts help pre-service teachers in online discussion? • What have pre-service teachers learned and benefited from producing a group educational video? Assessing critical thinking through scaffolded asynchronous online discussion 17

18. THANK YOU! yhteo@nie.edu.sg (Knowledge Community: www.globalkc.net) Assessing critical thinking through scaffolded asynchronous online discussion 18

19. Critiquing Video Cases Research Question 1: How does critiquing professional productions and past student video projects help pre-service teachers’ learning? Method: • Content Analysis of discussion log of both groups • Online survey questionnaire at end of course: MCQ & essay type questions Assessing critical thinking through scaffolded asynchronous online discussion 19

20. Critiquing Video Cases Q1: My knowledge of what makes a good educational video was Assessing critical thinking through scaffolded asynchronous online discussion 20

21. Critiquing Video Cases Q: Seeing & discussing past student projects is a good way to define what low and high quality work look like Assessing critical thinking through scaffolded asynchronous online discussion 21

22. Critiquing Video Cases : Transfer of Learning Q: My group has incorporated some of the insights and knowledge gained from the online video critique discussion in our own video production. Assessing critical thinking through scaffolded asynchronous online discussion 22

23. Critiquing Video Cases :Qualitative Results Research Question 1: How does critiquing professional productions and past student video projects help pre-service teachers’ learning? • Free response in survey questionnaire & Content Analysis • “…have a better idea of how the standards differ and allow me to gauge where my project stands” • “…see clearly what are the certain do’s and don’t’s when producing a video” • “…give direction and pre-empt mistakes in my own production” • “learn from more people’s comments rather than just within my group members” • “Engaging in video critique by looking at people’s comments and critiquing is a fun activity” Assessing critical thinking through scaffolded asynchronous online discussion 23

24. Asynchronous Online Discussion Q: The use of asynchronous online discussion has more advantages over face-to-face discussion. Assessing critical thinking through scaffolded asynchronous online discussion 24

25. Asynchronous Online Discussion Q: After reading my peers’ comments, some of my beliefs and ideas changed. Assessing critical thinking through scaffolded asynchronous online discussion 25

26. Critiquing Video Cases :Qualitative Results Research Question 2 : What benefits do pre-service teachers report that asynchronous online discussion has on their learning? • Free response in survey questionnaire & Content Analysis • Reading peers’ comments broadens point of view and obtaining a different perspective • Confirmed advantages of asynchronous discussion: • tend to contribute more as not afraid of ‘reprisals’ face-to-face • Permanent record encourages reflective thinking • Overcome limitations of distance and time Assessing critical thinking through scaffolded asynchronous online discussion 26

27. Role of Scaffolding Stems Q: The use of scaffolding prompts before posting a note helps me focus my thoughts Assessing critical thinking through scaffolded asynchronous online discussion 27