Using Portfolio Assessment to Discover Student Learning

1. Using Portfolio Assessment to Discover Student Learning Barbara J. D’Angelo, Ph.D. Arizona State University Association for Business Communication Annual Convention November 2009

2. Background: Written Communication for Managers • Junior-level business writing course • Multi-section course • 7-10 sections per semester + summer • On-campus, hybrid, and online • Redesigned Summer 2007 • Industry focus groups • Business and writing faculty input • Role playing scenario: students create a company • Standardized syllabus and assignments • Portfolio for assessment • End-of-semester self-evaluation of learning • Scored by 1 writing faculty and 1 business faculty Barbara J. D'Angelo, ABC Convention November 2009

3. Research Questions • Which assignments do students choose to include in their portfolios? • How do students organize their portfolios as an argument to demonstrate their learning? • What persuasive techniques do students use to support their argument for learning? • What tone and style do students use? • Do students present an argument for their learning or to enhance their grade? Barbara J. D'Angelo, ABC Convention November 2009

4. Methods • Random sampling • 3 portfolios from each section Fall 2008 and Spring 2009 • 34 portfolios analyzed • Simple counting of assignments • Content analysis of narrative statements • Categories derived from • Bower (2003): rhetorical appeals and argumentation • Scott (2005): form, tone, and appeal • Post-semester feedback from instructors • See proceedings for details (or talk to me later or email me) Barbara J. D'Angelo, ABC Convention November 2009

5. Assignments Included in Portfolios Barbara J. D'Angelo, ABC Convention November 2009

6. Assignments • Assignment inclusion not related to grading scale • Inclusion seems based on • Substantiality of writing or traditional genres • Instructor emphasis: instructor feedback indicates this as high • De-emphasis of meeting documents • Lack of understanding or emphasis • Increased emphasis during spring = higher inclusion = awareness of importance? • Process documents • Reluctance to show drafts or other “in progress” work? Barbara J. D'Angelo, ABC Convention November 2009

7. Portfolio Organization • 26 organized generically • Introduction: reflection about course • Body: described artifacts, claims and evidence • Conclusion: summed up • 7 organized contextually • Body contained more context • Best vs. worst • Strengths vs. weaknesses • Course objectives • Had no influence on raters’ scores Barbara J. D'Angelo, ABC Convention November 2009

8. Persuasive techniques Barbara J. D'Angelo, ABC Convention November 2009

9. Use of Persuasion • Nomos and pathos not linked to “schmooze” • Not directly linked to a claim or evidence • More expression of student “feelings” • Awareness of audience and course values • Majority (23) of students supported claims • 23 used logos to create claim with evidence to support it • 13 supported all claims • Unlike findings with FYC students • Indication of more sophisticated writing from upper level students? • 8 did not support any claims Barbara J. D'Angelo, ABC Convention November 2009

10. Tone and Style • 24 students used positive tone • Excitement about role playing (pathos) • 3 students used negative tone • Dis-satisfied with grade or with instructor? • 7 students used intermediate tone • Neutral descriptive tone • All students changed from “business” style to “student” style when completing the portfolio Barbara J. D'Angelo, ABC Convention November 2009

11. Learning or enhance grade? • Learning • 28 students made statements related to their growth • Only 1 student directly referred to their grade • Use of logos and support for claims indicates argument was for learning • Enhance grade • 19 portfolios potentially included “schmooze” • 2 flattered course or instructor • Rest thanked instructor • 2 students appealed directly to instructor (using “you”) • Result differs from FYC studies Barbara J. D'Angelo, ABC Convention November 2009

12. Use for course improvement • More emphasis on meeting documents • Importance to our employer stakeholders • Additional requirement to write agendas added for each Manager Roundtable • Requirement to evaluate agendas • Change MRT topics to more completely support written assignments • Instructors believed not enough time to go over concepts needed for assignments • Strengthen research summary to emphasize tie between finding information-analysis-use for persuasion Barbara J. D'Angelo, ABC Convention November 2009

13. Future • Potential change: portfolio scenario to shareholder or other external audience • Maintain emphasis on business style • Continue to de-emphasize mechanics • But help needed for ESL students • Continue assessment to aggregate additional data • Confirmation of findings related to use of logos/evidence • Add industry evaluators to assessment to include external stakeholders Barbara J. D'Angelo, ABC Convention November 2009

14. Questions? Presentation available at: http://www.public.asu.edu/~bdangelo/presentations/abc09.ppt Course syllabus and assignments available at: http://techcomm.asu.edu/curriculum/twc347 Email: bdangelo@asu.edu Barbara J. D'Angelo, ABC Convention November 2009