Introduction

1. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Participatory Assessment for Service Learning 1.0 and Civic Engagement 2.0Daniel HickeyAssociate ProfessorLearning Sciences ProgramIndiana University

2. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Acknowledgements • MacArthur Foundation’s Digital Media and Learning Initiative • 21st Century Assessment Project • IU Office of the Vice President of Information Technologies • E-Portfolio Faculty Scholars Program • US Department of Education Educational Technology Program • Monroe-Eastern Greene Network

3. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Broad Goals for Improving Service Learning and Civic Engagement • Resolve nagging tensions in education • Direct vs. constructivist instruction • Newer participatory approaches can resolve them • Enhance diverse learning outcomes • Engaged communal participation • Individual conceptual understanding • Aggregated achievement • Preserve validity of learning evidence

4. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Research Context • GenScope Assessment Project I (95-2000) • Core discursive formative assessment practice • GenScope Assessment Project II (2000-05) • Alignment across three levels of outcome • NASA COTF (03-05) • Streamlined model • Recent/Current Projects • Quest Atlantis (05-10) • Secondary English & New Media (08- • Learning Sciences Graduate Courses (09--

5. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Current Participatory Assessment Projects • Fan Fiction Module • Classic texts • Microblogging as characters • Characterization &character analysis as fan practices • On-Line Learning Sciences Graduate Courses • P507, Assessment in Education • P540 Cognition in Education • Both taught three times Taiga Ecological Sciences in Quest Atlantis • 20 hour curriculum for sixth graders • Five cycles of refinement from 05-10

6. Introduction Activate Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Research Method • Design Based Educational Research • Transcends fundamental vs. applied dichotomy • Build “local” in the context of iterative refinement • Pay careful attention to contextual framing • Validate principles by testing them in context • Meta-principles (guiding ideas) • Design principles (what ideas mean in general) • Specific design principles (ideas in contexts) • Specific features (used to enact specific principles)

7. Introduction Activate Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Participatory Learning Meta-Principles • Learning is primarily communal participation in knowledgeable discourse • Individual learning is secondary • Dissolves learning versus engagement dichomomy • Contextsgive meaning to concepts • Contextual knowledge is fundamental • This is 21st Century knowledge. • Activities, artifacts, assessments & accountability are all educationaldiscourses • Represent learning at very different timescales

8. Introduction Activate Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Participatory Assessment Design Principles 1. Transform Concepts • Reframe academic knowledge as tools used in contexts. 2. Scaffold Participation • Guide worthwhile discourse around use of tools 3. Assess Reflections • Avoid grading student-produced artifacts directly 4. Control Accountability • Use assessments and tests carefully and narrowly 5. Align discourse • Iteratively refine activities & assessments to indirectly raise achievement.

9. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Specific Assessment Design Principles for Service/Civic Contexts 1. Transform Concepts • Reframe academic knowledge as tools for use in service/civic contexts. 2. Scaffold Participation • Provide informal guidance to help students enlist academic concepts in service/civic contexts 3. Assess Reflections • Have students reflect on how their academic knowledge was transformed in service/civic contexts 4. Control Accountability • Assess individual service/civic learning carefully and discreetly • Measure service/civic programs in the aggregate 5. Align Discourse • Iteratively refine participation in service/civic activities to enhance reflections and indirectly impact distal outcomes

10. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse #1. Transform Concepts: Reframe Academic Knowledge as Tools for Contexts • Contexts give concepts meaning • “Deep” understanding of concepts is relevance across range of contexts. • 21st Century learning is about context • Range of contexts in all educational settings • Across characters, units, students, etc. • Reframe learning as a search for relevance • Learners need personally meaningful contexts • Compare value of contexts for learning concepts

11. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Reframing Context Examples • E-Learning Courses • Each student defines own lesson context • Wikifolios identify most relevant and least relevant implications and relevant specific • Quest Atlantis • Must use knowledge of inquiry and ecological science to participate in game

12. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse E-Learning ClassesStudents First Define Problem

13. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse E-Learning ClassesReading Text as a Search for Relevance

14. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse E-Learning ClassSearch For Relevant Specifics

15. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Ideas for Reframing Academic Knowledge as Tools in Service/Civic Contexts • Service and Civic Learning is about contexts • Void between school learning and contexts • Void is an opportunity rather than an obstacle • Range of contexts civic/service learning • From school to service/civic context • Across students in same service/civic context • Across service/civic contexts • Service/civic learning as search for relevance • How is your academic knowledge relevant? • How is this service/civic context transforming your knowledge?

16. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse #2. Scaffold Participation: Guide Worthwhile Discourse Around Using Tools • Foster conversation about using conceptual tools in contexts • Different context reveal concepts differently • Pay attention to initial participation • Must enlist tools initially before appropriately • Don’t grade or require participation • Encourage lurking • Focus on the trajectory of the discourse • Don’t prematurely focus on accuracy

17. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Scaffolding Participation Examples • E-Learning Courses • Don’t use discussion forums • Students comment on each other’s wikifolios • Quest Atlantis • Reframe Quest submissions as informal discourse • Ensure useful feedback is provided and used

18. wi

19. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Example of Consequential Collaboration in Wiki Commenting • Comments on wikifolios should lead to reflection and revision • Good use of instructor comments. • Assume that physical and intellectual proximity supports engagement • Overcomes abstract and alienating online context

20. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Juana’s Example • Chapter 2 Wiki on modal model of memory • Most relevant implication was information processing is easier when to-be-learned information is distributed in working memory • Linda’s comment suggested misunderstanding • “students often have so much information to manage, that they may need to drop a lot of it in order to focus on the important points. That, to me, would fall more under the description of the 'bottleneck' implication than the 'distributed in working memory' implication, which I interpret to be more about presenting information in both a visual and auditory manner.”

21. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Example Comment on Juana’s Wiki • Linda’s Comment: “The way that I interpreted what you wrote for #1 is that students often have so much information to manage, that they may need to drop a lot of it in order to focus on the important points. That, to me, would fall more under the description of the 'bottleneck' implication than the 'distributed in working memory' implication, which I interpret to be more about presenting information in both a visual and auditory manner.” (emphasis added)

22. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Juana’s Reflection • Consequential Reflection: “After reading everyone comments it seems as though the implications selected are appropriate for my problem except for item number 1 that I have changed from working memory to Resource and Data limitations. (Resource and data limitations constrain learning)This is a much better fit for what my comments were and after rereading that section is clearly the right choice…

23. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Jeraldine’s Case • Example: Jeraldine’s Chapter 2 Wiki • Indicated “All students should be encouraged to ‘manage their resources’” as the most relevant implication • The comments redirected her thinking about relevance in different contexts.

24. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Example Comment on Jeraldine’s Wiki • Emerald’s comment: …almost all of your explanations you wrote about the importance of automaticity, making me wonder why "Automaticity facilitates learning by reducing resource limitations" wasn't your #1 implication.I could understand why you wouldn't choose this if your lesson included more challenging texts, but reading easy texts would require a lot of automaticity even for struggling readers. (emphasis added)

25. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Jeraldine’s Reflection • Collaborative Reflection: “Emerald commented on my use of automaticity throughout my wiki and wondered why the automaticity implication was not my first choice. I reread my reasoning for my first implication and realized that this is my goal for all my students. However, for my instructional problem, automaticity is the key. Hence, this implication should become my top choice. My second and third implications are almost subheadings of this top implication.”

26. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Ideas for Scaffolding Participation in Service/Civic Learning • Anticipate different service/civic contexts in classes • Speculate about relevance early • Articulate connections/voids between school and service/civic contexts • Provide incentives for participation • Bring reflections in early

27. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse #3. Assess Reflection: Avoid Grading Student-Produced Artifacts • Grading artifacts undermines discourse • Requires detailed-but-shallow rubrics and exemplars • Grade reflections as evidence of engagement • Grade infrequently • Aim to foster revision • Focus reflections on engagement • Consequential, critical, and collaborative • Conceptual (i.e., understanding) “comes for free”

28. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Assessing Reflection Examples • E-Learning Courses • Wikifolios are never graded • Grade reflections every five weeks • Quest Atlantis • Added reflections to quest submissions • “How does your report show you have learned?”

29. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Reflect on Specific Types of Engagement • Consequences of key concept in your each context • Also consequences of context for each concept • Critique use of key concepts in each context • Critique context for learning concepts • Collaboration associated with key concepts • Critique classes collaboration?

30. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse

31. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Ideas for Assessing Reflection in Service/Civic Learning • Don’t have student report on what they did or what they learned • Have them report on how they engaged • Articulate the types of engagement for reflection • Consequences of big ideas your context • Consequences of context for big ideas • Critique use of big ideas in each context • Critique context for learning big ideas • Collaboration associated with big ideas • Critique classes collaboration? • Do reflections publically via wikis or forums • Reward reflections that lead to action

32. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Ideas for Assessing Reflection in Service/Civic Learning • Don’t have student report on what they did or what they learned • Have them report on how they engaged • Articulate the types of engagement for reflection • Consequences of big ideas your context • Consequences of context for big ideas • Critique use of big ideas in each context • Critique context for learning big ideas • Collaboration associated with big ideas • Critique classes collaboration? • Do reflections publically via wikis or forums • Reward reflections that lead to action

33. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse #4. Control Accountability: Use Assessments and Tests Carefully • Downplay importance of individual assessments and tests • Align with scaffolding and reflection • Engagement should ensure adequate understanding • Assess and evaluate just a few outcomes • Don’t use program outcomes to assess individuals • Compromises validity

34. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Controlling Accountability Examples • On-Line Learning Sciences Courses • Multiple choice midterm and exam from test bank • Worth only 15% each, use criterion for points • Quest Atlantis • Open ended performance assessment used to evaluate curriculum (not students) • Random sample of achievement items used to evaluate gains over time

35. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Ideas for Controlling Accountability in Service/Civic Learning • Focus on proximal outcomes • Align close level activities to them • Use “distal” individual outcomes in the aggregate • Treat program outcomes as programmatic remote outcomes • Be very focused with surveys • They are not particularly helpful for improvement • Control accountability internally • Don’t let outsiders control it • Assess and measure judiciously • Just because you can does not mean you should!

36. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse #5. Align DiscourseRefine Participation at all Levels • Use design-based research methods • Refine features to validate specific principles. • Build local rather than fundamental theory • Align participation to raise all outcomes • Preserve evidential validity by changing format • Three or more increasingly formal cycles • Design (design activities and artifacts) • Implementation (select activities & artifacts) • Evaluation (compare achievement gains)

37. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Three Levels Of Learning Outcomes • Interpret participation in discourse (“close”) • Foster discourse in real time to increase understanding • Protect from corruption via assessment • Assess individual understanding (“proximal”) • Refine design and choice of activities to raise achievement • Protect from corruption by test • Measure aggregated outcomes (“distal”) • Evaluate increased gains between or across courses • Secure test to preserve validity

38. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Aligning Discourse Examples • Learning Sciences Courses • Refining prompts to maximize quality discussion • Looking for improved reflections • Measure achievement gains from year to year • Quest Atlantis • Continually improve questing discourse directly • Increased gains in understanding (Δ >2 SD) • Slowly raised gains in achievement (Δ >1 SD)

40. Introduction Reframe Concepts Scaffold Participation Assess Reflection Control Accountability Align Discourse Ideas for Aligning Discourse in Civic/Service Learning • Focus on proximal outcomes like reflections on service • Align classes and other activities to proximal assessments • Use program outcomes to evaluate programs rather than individuals • Find valid measures for evaluating program improvement