Promoting Transfer through Program Assessment: The Role of Reflection

1. Promoting Transfer through Program Assessment: The Role of Reflection Gwen Gorzelsky, Wayne State University g.gorzelsky@wayne.edu

2. WSU Composition Program Assessment Project Research-based approach Strategies for teaching students to “inventory and monitor their own skills,” to “seek assistance as needed to increase their success,” and to cultivate conditional knowledge Transfer studies, psychology (Salomon & Perkins, 1989; Schraw & Denison, 1994; Schwartz, Bransford, & Sears, 2005) Transfer studies, composition (Anson, 2008; Beaufort, 2007; Bergmann & Zepernick, 2007; Downs & Wardle, 2007; Driscoll & Wells, 2012; Taczak, 2011; Wardle, 2009) Assessment scholarship, composition (Broad, 2003; O’Neill, Moore, & Huot, 2009)

3. WSU students: SES, finances • 90% tri-county residents; 20% of those from Detroit • 41% people of color; 25% African American • 50% family income < $50K • 70% financial aid • 86% work (FT or PT) • Many 1st-generation college students

4. Initial Data- Collection Methods Assessment Focus Guiding Question To what extent, if any, are students transferring writing-related knowledge and skills across Composition Program courses and from these courses to Writing Intensive (WI) courses in the disciplines? Student surveys Student and instructor focus groups Instructor text analysis sessions: criteria for evaluating students’ writing

5. Student Focus Group Findings ENG 3010 Writing Intensive Courses Very diverse range of genres required Limited description of skills developed by IC courses Need for collaborative work is discipline-specific Perceived connections between IC and WI courses depend on whether the courses share either: genres, or complex, highly defined skills. • Improved confidence in finding and using scholarly research • Appreciation for explicit instruction in rhetorical moves of academic writing • Appreciation for accountability and input • Unimproved recognition of genre features of writing in their disciplines

6. Instructor Focus Group & Text Analysis Findings ENG 3010 Writing Intensive Courses Diverse genres and priorities Wide range of disciplines, shared evaluative criteria Implicit but strong concern with genre features “Critical thinking” – key criterion, competing definitions: complexity and depth vs. categorizing, generalizing, synthesizing, and recognizing the progression of science • Tension: students as critics vs. students as apprentices

7. Pilot 3010 Sections: The Research Study Two WAW sections, Fall 2011 Three WAW + reflection sections, Fall 2011 Learning outcomes & course designs incorporate 3010 & WI instructors’ suggestions Two control sections (existing 3010 curriculum), Fall 2011 Portfolio evaluation: White’s reflective letter to show achievement of learning outcomes Participating students: N=115 Pilot 3010 Sections: Curricular Design – Writing About Writing (WAW) Shift of focus: from writing in students’ disciplines to learning how to write in any future context Content: writing studies concepts, including discourse community, genre, rhetorical, and writing process knowledge Emphasis: cultivating students’ metacognitive strategies, especially their conditional knowledge on when and how to use prior content and procedural knowledge about writing and, perhaps most importantly, how to adapt that knowledge in new writing contexts

8. ENG 3010 Learning Outcomes #1 Produce writing that demonstrates your ability to identify, describe, and analyze various occasions for writing, genres, conventions, and audiences in your discipline or profession from a rhetorical perspective. #2 Produce an extended writing project that uses research methods and research genres to explore a topic applicable to the course and that draws substantively on concepts from primary AND/OR secondary sources. #3 Produce writing that shows use of a flexible writing process (generating ideas, drafting, substantive revision, and editing) and shows your ability to adapt this process for different writing situations and tasks. #4 Produce writing that shows how you used reflection to make choices and changes in your writing and that explains how you would use reflection and the other skills taught in this course to approach a completely new writing task.

9. Fall 2011 WSUPortfolio Scores • N = 64 • Reflection scores slightly lower than other categories (1=high; 5=low)

10. WSU Portfolio DebriefingResults • Readers indicated that successful reflective arguments included concrete specifics in discussing all learning outcomes, including #4. • Readers believed the majority of portfolios did not demonstrate adequate reflection, regardless of curriculum taught. • Readers saw greatest need for improvement in specificity in students’ discussions of potential future use of writing strategies and processes. • Readers recommended significantly expanding resources for teaching reflection throughout the semester.

11. WSU Assessment Findings: Needs & Challenges • Pilot 3010 instructors reported high level of challenge in teaching new curriculum. • Pilot instructors and portfolio readers recommended more attention to crafting and teaching reflective assignments. • The wider 3010 instructor cohort needed grounding in writing studies concepts at the heart of the revised curriculum, findings of the efficacy of supporting metacognition, and strategies for teaching reflection to promote metacognitive thinking. • Ambivalence about a non-humanities/non-literary focus for 3010 among many instructors suggested the need to provide opportunities for instructors to work collaboratively with each other to implement the curriculum in a way that allowed them to take some ownership of it.

12. Assessment Findings: Use for programmatic revisions • Online 3010 resource site that foregrounds profiles of six instructors and provides rationales for and explanations of key concepts, bibliographies, and templates, along with sample syllabi, assignments, scoring rubrics, class plans, and student work • Teaching workshops and orientation sessions on writing studies concepts and teaching reflection to support metacognition • Teaching circles using a communities of practice approach to encourage instructor exchange and collaboration

13. What can we learn from considering our portfolio scores and readers’ debriefing feedback in context of findings from “The Writing Transfer Project”? • How might those of us at WSU use these findings to improve curriculum, pedagogy, and professional development at our home institution? The Writing Transfer Project The cross-institutional perspective

14. Code: Connection to future educational experience • Homework 2 (N=11) • WSU 45.5% (5) • SHU 0% (0) • GWU 9.1% (1) • OU 45.5% (5) • Final reflection (N= 68) • WSU 54.5% (36) • SHU 4.5% (3) • GWU 7.6% (5)/12.1% (8) • OU 21.2% (14) • Homework 1 (N=3) • WSU 33.3% (1) • SHU 0% (0) • GWU N/A • OU 66.7% (2) • Reflection 1 (N=39) • WSU 33.3% (13) • SHU 5.1% (2) • GWU 15.4% (6)/17.9% (7) • OU 28.2% (11)

15. Code: Connection to futureprofessional life • Homework 1 (N= 9) • WSU 37.5% (3) • SHU 0% (0) • GWU N/A • OU 62.5% (5) • Homework 2 (N=10) • WSU 60% (6) • SHU 0% (0) • GWU 0% (0) • OU 40% (4) • Reflection 1 (N=17) • WSU 64.7% (11) • SHU 0% (0) • GWU 0% (0) • OU 35.3% (6) • Final reflection (N=47) • WSU 66% (31) • SHU 6.4% (3) • GWU 2.1% (1) • OU 25.5% (12)

16. Using Knowledge Codes Connection to Current Educational Life Connection to Current Professional Life Year 2 Interview (N=12) WSU 16.7% (2) SHU 0% (0) GWU 0% (0) OU 83.3% (10) Year 2 Interview (N=76) • WSU 17.1% (13) • SHU 34.2% (26) • GWU 23.7% (18) /10.5% (8) • OU 14.5% (11)

17. Recap: WSU Year 1 & Year 2 Code Applications Connection: Future Profession Homework 1 (N= 9): 37.5% (3) Homework 2 (N=10): 60% (6) Reflection 1 (N=17): 64.7% (11) Final reflection (N=47): 66% (31) Connection: Current Profession Year 2 Interview (N=12): 16.7% (2) Connection: Future Education Homework 1 (N=3): 33.3% (1) Homework 2 (N=11): 45.5% (5) Reflection 1 (N=39): 33.3% (13) Final reflection (N= 68):54.5% (36) Connection: Current Education Year 2 Interview (N=76): 17.1% (13)

18. Vague WSU Reflection Excerpt • As a result of the mastery of all four learning objectives, I have acquired all the criterial knowledge banks that come together to form writing expertise as Anne Beaufort argues, and I believe this writing expertise coupled with my newfound metacognative skills that allow me to analyze and adapt quickly to differing sets of conventions will carry me far in the various writing situations I will encounter in my future studies and beyond. Without a doubt, this course has permanently changed my thought processes and overall approach to writing in ways that are genuinely invaluable, and I believe that completing this sort of work – writing about writing, and consequent reflection – is possibly one of the most enlightening experiences a developing writer could wish for.

19. Specific WSU ReflectionExcerpt #1 As for trying to apply my writing strategies to future situations, I strongly believe that there is so much that I can take with me from this course. Especially from writing the literature analysis portion of my Synthesis Project, I feel that will be a crucial step in my future writing classes. I even point out in my Reflection Paper that, “While discussing this project with an older sibling, she explained that literature reviews come up frequently when trying to achieve a master’s degree in later years” (p. 20). Also, in a broader sense, I pointed out that, “Now, when writing papers for my other classes, I am constantly thinking about which paragraph should go in which spot to help further my thoughts and perhaps my argument, depending on the context of the paper” (p. 19). This also proves my mastering of Learning Outcome #4, because I am fully aware of how I can apply the skills I learned in this pilot course to future situations. I know what I am taking away from this class, and these skills are imbedded permanently from the repetition and frequent use of all of them.

20. Specific WSU ReflectionExcerpt #2 As for transferring the writing skill to other fields, I have not been doing that many writing since this semester besides this class. I did, however, learned to analyze some art historian’s writings and bettered my essay for my Art History class. For example, in one essay, I was analyzing Venus of Willendorf (a famous monumental sculpture from Austria). I said, “Her (Venus of Willendorf) breast and genitalia areas are exaggerated to show fertility.” I only knew this express after reading several art historians’ writings from the textbook and try to apply that formal, proper language in my own writing. This is the genre analysis ability I took away from this English 3010 class.

21. Implications of the Cross-Institutional Study for WSU’s Assessment Project Local insights Use of cross-institutional findings Our assessment team will review excerpt sets from relevant codes in “The Writing Transfer Project” WSU data to obtain strong, weak, and average samples of reflective writing. We’ll use these sample sets to augment our existing resources, including our online resource site, teaching orientation sessions and workshops, and teaching circles. • From the WSU perspective, not all instances of transfer-focused thinking seem adequately specific and elaborated to be likely to prepare students to use and adapt knowledge from 3010 in future writing contexts. In fact, many instances seem too vague to do so.

22. References Anson, C. (2008). Closed Systems and Standardized Writing Tests.. College Composition and Communication, 60(1), 113-128. Beaufort, A. (2007). College Writing and Beyond: A New Framework for University Writing Instruction. Logan, Utah: Utah State UP. Bergmann, L. & Zepernick, J. (2007). Disciplinarity and transfer: Students’ perceptions of learning to write. WPA: Writing Program Administration.31(1-2), 124-149. Broad, R. (2003). What we really value: Beyond rubrics in teaching and assessing writing. Logan, UT: Utah State UP. Downs, D., & Wardle, E. (2007). Teaching about Writing, Righting Misconceptions: (Re)Envisioning "First-Year Composition" as "Introduction to Writing Studies". College Composition and Communication, 58(4), 552-584. Driscoll, D. L., & Wells, J. (2012). Beyond knowledge and skills: Writing transfer and the role of student dispositions. Composition Forum, 26. National Research Council (1999). How people learn: brain, mind, experience, and school.Washington D.C.: National Academy Press. O’Neill, P., Moore, C., & Huot, B. (2009). Guide to college writing assessment. Logan, UT: Utah State UP. Salomon, G. & Perkins, D.N. (1989). Rocky Roads to Transfer: Rethinking Mechanisms of a Neglected Phenomenon. Educational Psychologist.24(2), 113-142. Schraw, G., & Dennison, R. S. (1994). Assessing Metacognitive Awareness.Contemporary Educational Psychology, 19, 460-475. Schwartz, D. L., Bransford, J. D., and Sears, D. (2005). Efficiency and innovation in transfer.” Transfer of Learning from a Modern Multidisiplinary Perspective. Ed. Jose P. Mestre. Information Age Publishing, 1-51. Taczak, K. (2011). "Connecting the Dots: Does Reflection Foster Transfer?". Ph.D., Flordia State University, Tallahassee. Wardle, E. (2009). "Mutt genres" and the goal of FYC: Can we help students write the genres of the university? College Composition and Communication, 60(4), 765-789. White, E.M. (2005). The scoring of writing portfolios: Phase 2. College Composition and Communication, 56(4), 581-600.