Assessing Transfer: Using Reflection to Evaluate Transfer of Knowledge at Critical Transitions in Writing Programs

1. Assessing Transfer: Using Reflection to Evaluate Transfer of Knowledge at Critical Transitions in Writing Programs Bob Broad, Illinois State University Heidi Kenaga, Wayne State University Dana Driscoll, Oakland University Joseph Paszek, Wayne State University Wendy Duprey, Wayne State University Liane Robertson, William Paterson University Gwen Gorzelsky, Wayne State University David Slomp, University of Lethbridge Jared Grogan, Wayne State University Kara Taczak, University of Denver Adrienne Jankens, Wayne State University Thomas Trimble, Wayne State University

2. Workshop Overview • Defining & assessing transfer • Key principles for measuring transfer • Case studies on assessing transfer • Working Groups • Critical transitions • Developing goals • Developing assessment plans To access all workshop materials online: http://assessingtransfer.pbworks.com

3. Basic Definitions • Smit (2007) argues that the ability to transfer knowledge is what the term “learning” actually means (p. 130). • The National Research Council (1999) argues that “The ultimate goal of learning is to have access to information for a wide set of purposes—that the learning will in some way transfer to other circumstances” (p. 61). • Other transfer terms: boundary crossing, knowledge building

4. Traditional Cognitive and Task-based Transfer • Definition: “The application of knowledge learned in one situation to a new situation.” • Research method: “improved performance on tasks” (primarily through experimental design) • Research questions: “Was transfer obtained? What conditions facilitate transfer?” (Loboto 2003)

5. Jimmy: Each situation is viewed as unique and nothing is carried to the next situation. Jimmy Rhetorical and Genre Analysis in FYC Product Analysis Whitepaper in Business Class Analysis of Presidential Candidates Briel Briel: Transfers knowledge between tasks and builds her knowledge of analysis.

6. Context-Based Transfer (Activity Theory) Definition: “The personal construction of relations of similarity across activities (i.e., seeing situations as the same).” Research method: “Researchers look for the influence of prior activity on current activity and how actors construe situations as similar.” Research questions: “What relations of similarity are created? How are they supported by the environment?” (Loboto, 2003) • Actor-Oriented Transfer focuses more on the context of learning and works within the realm of activity theory

7. Briel American Civic Engagement Activity System FYC Activity System Political Voting Activity System BUS300 Activity System University Activity System Jimmy

8. Contextual-Dispositional (Hybrid) Approaches • Wells and Driscoll (under review) argue that both task-based and contextual approaches are useful, but provide an incomplete picture of transfer. • Rather, they argue it is the intersection of the task, context, and the individual learner’s dispositions. These include: • motivation, self efficacy, help-seeking, willingness to engage in mindful abstraction, developing a metacognitive mindset, beliefs, attitudes etc. • This approach examines the relationship between learner, the task, and the context and is particularly useful to assessment. • Bio-ecological assessment can also fit within a hybrid approach (as described next)

9. Bio-ecological Theory of Transfer Bronfenbrenner and Morris (2006); Slomp (2012) • Proximal Processes • Processes through which learning occurs (within local environment) Transaction Individual

11. Assessing Transfer Challenges and Considerations

12. 3 Challenges • Choosing a theory of transfer that captures a full picture of the factors that support or inhibit transfer. • Defining/adopting a socially-situated construct model of writing ability. • Overcoming technocentric limitations in current writing assessment practices

13. Challenge 1: Defining a robust theory of Transfer

14. Theories of Transfer • Clearly define how you are conceptualizing transfer: • The theory of transfer you adopt will determine your research/assessment focus. • Carefully consider which theory of transfer you base your assessment on. • Traditional conceptions of transfer have underrepresented what people know and are able to do (Schwartz, Bransford & Sears, 2005).

15. Task-Based Conception of Transfer Task 2: Complete worksheet requiring student to properly insert commas on a page of unpunctuated sentences. Are students able to draw on knowledge of comma rules developed in task 1 to complete task 2? Task 1: Complete worksheet on rules for using commas.

16. Contextual Theories of Transfer Activity Systems: (Wardle, 2007) Context 2: Writing Across the University Students did not generalize from FYC because the activity system did not encourage them to do so. Context 1: FYC Organization skills, process knowledge, critical reading, subject knowledge Do students perceive that what they learned in FYC has helped them with later writing assignments across the university? How does the curriculum structure influence generalization?

17. Contextual theories of Transfer • Transformation and repurposing (Roozen, 2010) Context 1: Religious engagements Prayer journaling, verse-copying, sermon outlining Context 2: Academic engagements Note taking, outlining, and organizing academic arguments. How does prior knowledge inform current practices? How is prior knowledge repurposed when applied in new contexts?

18. Bio-ecological Theory of Transfer Context 2: Writing Across the University Barriers to transfer included: curriculum structures, prior experience, personality issues, challenging home environments. Slomp and Sargent (2009, forthcoming) In what ways do characters of the developing individual, proximal processes, and the ecological systems in which students learn shape their capacity to generalize, repurpose and/or transfer knowledge about writing? Context 1: FYC Writing process knowledge, metacognitive knowledge, discourse community knowledge,

19. Challenge 2: Defining a Construct Model

20. Construct Validity • The construct writing ability is defined through the lenses of developmental theories (Camp 2012):

21. Beaufort’s (2007) Transfer-Oriented Construct Model Metacognitive Knowledge Discourse Community Knowledge Subject Matter Knowledge Writing Process Knowledge Rhetorical Knowledge Genre Knowledge

22. Challenge 3: Overcoming Technocentric Limitations

23. Methodological Considerations • Writing assessment has traditionally been limited by its technocentric orientation (Huot 2002): • Emphasis on achieving high degrees of reliability • Constrained by reliance on current technologies of assessment

24. METHODOLOGICAL CONSIDERATIONS • Assessment-as-research (Huot, 2002): • Focus on defining information needs • Choose assessment methodologies that help you achieve those information needs. • Shift away from technocentric views of reliability toward a rhetorical orientation (Parkes, 2007). • Emphasis on validity

25. Methodological Considerations

26. Ecological Assessment Design Wardle and Roozen (2012)

27. Bio-ecological Assessment design

28. Metacognition, Transfer, and A New Research Paradigm

29. Two Types of Transfer High Road Low Road Results from extended practice Spontaneous, automatic, with little need for reflective thinking Example: driving a car to driving a truck Increased speed and efficiency Decreased long-term memory and analytic reflection Potential for negative transfer • Results from mindful abstraction • Can occur quickly, without long-term practices • Example: applying the “count to ten” rule learned in childhood to inhibit tantrums during adulthood to prevent impulse buying • Promotes greater understanding, reflective evaluation, and conscious adaptation of previously learned concepts and skills - Salomon and Perkins (1989)

30. Mindful Abstraction Mindful Abstraction • Using metacognitive thinking to decontextualize information to construct principles, patterns, strategies, or procedures -- Salomon and Perkins (1989) Abstraction • Identifying key qualities, attributes, or patterns • Decontextualizing information and re-representing it as a set of principles or schemas Mindfulness • Thinking guided by metacognition and conscious reflection on target task, context, known strategies, and potential adaptations

31. Principles for Promoting Transfer through Metacognition • Requiring students to actively monitor their learning • Providing feedback on students’ use of new knowledge • Showing contrasting cases to highlight key features • Foregrounding the transfer potential of new knowledge • Teaching new knowledge in multiple contexts • Moving from specific to general levels • Helping students abstract principles • Balancing specific examples with general principles - --National Research Council (1999)

32. PROMOTING TRANSFER THROUGH REFLECTION Reflection – both a theory and a practice: • From the work of Schon – on reflective practitioners • From the work of Yancey – students as agents in their own learning process

33. PROMOTING TRANSFER THROUGH REFLECTION Reflection’s Connection to Transfer: Our Starting Point Significant research on each separately Absence of research that explicitly explores reflection’s connection to transfer Beaufort’s knowledge domains - reflection discussed as important for metacognition but not explicitly pursued Schon and Yancey - reflection both theory/practice

34. PROMOTING TRANSFER THROUGH REFLECTION Reflection’s Relationship to Transfer: Kara’s Research: questions whether or not reflection is one of the vehicles by which students transfer knowledge and practices of writing to other academic writing situations. Liane’s Research: questions which content transfers effectively and how reflection as a reiterative practice fosters the transfer of that content.

35. PROMOTING TRANSFER THROUGH REFLECTION Definition of Reflection: • We define as: systematic, explicit, intentional • Both an intellectual act and a physical act • aligns with Perkins and Salomon’s claim that “conditions of a classroom affect transfer” • aligns with the notion of mindful abstraction – active self-monitoring arouses mindfulness • Students’ ability to monitor their own thinking processes is what leads to mindful abstraction – alertness to the activity in which they are engaged

36. PROMOTING TRANSFER THROUGH REFLECTION Practice of Reflection: • We use reflection as: reiterative practice • by which students learn to define and apply their own theory of writing as a way to foster transfer of knowledge and practices from one academic writing situation to another. • Theory of Writing – main reflective practice for our FYC course • Students create a framework of writing knowledge • Students begin to develop metacognitive ability

37. PROMOTING TRANSFER THROUGH REFLECTION Our Theory of Writing Component: • Systematic, explicit, intentional • Explicitly encourages transfer • Asks students to be mindful about what they are learning • Reiterative assignment feature - ten writing assignments related to theory of writing throughout the semester • Combines learning about writing theory, through a set of key terms and through reading reflective theory, with the practice of systematic, explicit, intentional reflection

38. PROMOTING TRANSFER THROUGH REFLECTION 3. Ends As Metacognitive thinking; Abstraction Development of a theory of writing knowledge and practice 4. Continues As Application of knowledge and practice in new writing contexts 1. Begins As • Practice more than theory • Not mindful • No abstraction, direct application 2. Progresses Toward • Increased active self-monitoring • Key terms understood as writing concepts • Mindfulness develops

39. PROMOTING TRANSFER THROUGH REFLECTION Sample Reflective Activities: • Early Guided Reflection • Reiterative Reflection

40. Motivation and Dispositions Factors Teaching Approaches Devising challenging but do-able assignments Designing tasks that demonstrate relevance Providing social support for risk-taking • Task difficulty level • Perceived relevance • Performance vs. learning orientation -- National Research Council (1999)

41. Developing the New Research Paradigm: Integrating Individual, Social, and Developmental Aspects PFL Enables Writing Studies Researchers to: • Evaluate students’ metacognition in relation to writing studies conceptual and procedural knowledge • Evaluate how students use this knowledge to learn about unfamiliar genres and rhetorical situations • Triangulate data from assessments of individual factors like dispositions and motivations and of social factors like curriculum Preparing for Future Learning (PFL) • Focusing on Interpretive Knowledge • Evaluating Interpretive Knowledge • Incorporating opportunities for learning into assessment • Focusing on both “transfer in” and “transfer out” -- Schwartz, Bransford, and Sears (2005)

42. Developing the New Research Paradigm: Integrating Individual, Social, and Developmental Aspects • Pre- and post-semester surveys evaluating students’ motivation and dispositions related to writing instruction • Text-based interviews that ask students to describe their drafting choices, particularly the conceptual and procedural knowledge they used • Textual analyses comparing discourse features of students’ reflective writing with those of their academic writing • Textual analyses of students’ reflective writing on their uses of conceptual and procedural knowledge about writing.

43. Tasks used and knowledge required • Learners’ prior experiences and dispositions • Classroom and curricular contexts Assessing Transfer Develop a hybrid, locally focused model that takes into account all of the factors affecting transfer.

44. References Driscoll, D. and Wells, J. (under review). Toward a Dispositional Model of Writing Transfer: The Impact of the Individual Learner. Loboto, J. (2003). How Design Experiments Can Inform a Rethinking of Transfer and Vice Versa. Educational Researcher, 32(1), 17-20. National, R. C. (1999). How people learn: brain, mind, experience, and school. . Washington D.C.: National Academy Press. Royer, J. M., Mestre, J. P., & Dufresne, R. J. (2005). Introduction: Framing the transfer problem. . Greenwich, CT: Information Age Publishing. Salomon. D. N., and Perkins, G. S. a (1989). Rocky Roads to Transfer: Rethinking the Mechanisms of a Neglected Phenomenon. Educational Psychologist, 24(2), 113-142. Smit, D. (2007). The End of Composition studies. Carbondale, IL: Southern Illinois University Press. Tuomi-Grohn, T., & Engestrom, Yrjo (2003). Between school and work : new perspectives on transfer and boundary-crossing (1st ed.). . Boston, MA: Pergamon.