Portfolio Assessment: A literature review

1. Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong

2. Overview • Historical perspectives • Defining the portfolio • Purposes of portfolios • Issues in portfolio assessment • Research in portfolio assessment • My own possible research avenues

3. Historical perspective • Portfolios widely used for many years • Late 80s interest in portfolios for assessment (Belanoff and Dickson 1991) • 90s saw advent of eportfolios • A shift in emphasis away from assessment to learning?

4. Definitions • “collection of student work that demonstrates achievement or improvement” (Stiggins 1994) • “a portfolio is a collection of evidence that is gathered together to show a person’s learning journey over time and to demonstrate their abilities” (Butler 2006)

5. Definitions • “…student writing over time, which contains exhibits showing the stages in the writing processes a text has gone through and the stages of the writer’s growth as a writer, and evidence of the writer’s self-reflection on her/his identity and progress as a writer” (Hamp-Lyons 1996)

6. Definitions • portfolios are “…prepared with a particular audience in mind”, “…are selective” and “call for judgments” (Calfee and Freedman 1996)

7. Definitions • “…a purposeful collection of student work that illustrates efforts, progress, and achievement in one or more areas [over time]. The collection must include: student participation in selecting contents, the criteria for selection, the criteria for judging merit, and evidence of self-reflection” (The Northwest Evaluation Association cited in Barret 2005)

8. Definitions – main characteristics • They are collections of work, different from a single timed impromptu essay or a class essay carried out over a semester. • They are purposeful in that they “demonstrate”, “exhibit” or provide “evidence” of “achievement”, “improvement”, “the writer’s self reflection”, “the writing process” and “the writer’s growth”. • The degree to which these characteristics are evidenced in portfolios largely depends on their purpose.

9. a process portfolio a showcase portfolio an assessment portfolio A dossier portfolio A reflective portfolio A classroom portfolio A positivist portfolio A constructivist portfolio A personal portfolio A structured portfolio An employment portfolio A working portfolio Types of portfolio

10. Where are portfolios used? • Primary and secondary classrooms • In tertiary settings: • Teacher education • Medicine • Nursing • Engineering • Dentistry • Psychology • ESP/EAP writing classes

11. How are portfolios used? • In a class • Across more than one class • Statewide • Across a university curriculum

12. Purposes • Accountability; evaluating program or curriculum effectiveness • Evaluating individual student progress; grading, certifying student accomplishment • Diagnosing students’ needs; informing classroom instructional planning

13. Purposes • Encouraging teacher efficacy; encouraging reflective practice at the school and classroom levels; supporting teachers’ professional development • Encouraging student efficacy; promoting student self-assessment; motivating student performance (Herman, Gearhart and Acshbacher, 1996)

14. Issues • Different audiences (who is the portfolio for?) • Grading – who grades? (fairness) • Learning and reflection get lost in drive to measure competency (Herman and Winter 1994)

15. Issues • Gap between psychometrics and collaborative nature of the revision process (Song and August 2002,Hamp-Lyons and Condon 2000 ) • Time needed for both teachers and students (Callahan 1995, Herman and Winters 1994)

16. Common problems • Purposes – clear to teacher and student? (Callahan 1995) • Mismatch between assessment criteria and goals of programme of study • Student anxiety and confusion (Butler 2006)

17. How do portfolios function best?

18. Research in portfolio assessment A slim collection?

19. Research in portfolio assessment • Validity and reliability • Fairness • Impact

20. Validity • Vermont program • Correlation ranging from .47 to .58 between writing portfolio scores and direct writing assessments • Similar correlation between portfolio scores and multiple-choice maths test scores (Koretz 1993 cited in Herman and Winter 1994)

21. Validity • Gearhart and others (1993) cited in Herman and Winter (1994) found: • No relationship when comparing writing portfolios with standard writing assessments • Two thirds of students classified as “masters” on the portfolio assessment would not have been so classified on the standard assessment.

22. Validity • Gearhart and others (1993) cited in Herman and Winter (1994) also found: • When portfolios were scored in two different ways (holistic and individual pieces scored) correlations were in the .6 range • Half the students who would have been classified as masters on the single portfolio score would not have been so classified when individual pieces were averaged

23. Which assessment best represents an enduring capability?

24. Validity • CUNY (Song and August 2002) • 2 groups, 1 assessed by portfolio and writing test, the other only a writing test • Students twice as likely to move to the next course when evaluated by portfolio • At the end of the next course the pass rate and grade distribution for the two groups were nearly identical

25. Reliability • Vermont interrater reliability of .28 to .60 • Pittsburgh portfolio system ranged from .6 to .7 • Herman et al. (1993)found correlations of .82 in an elementary school portfolio containing final drafts of writing

26. Reliability • Little work on other sources of portfolio reliability • Score stability over time • Stability across different rater groups • The portfolio set in which a particular portfolio is rated (Herman and Winter 1994, pg. 51)

27. Reliability • Heller, Sheingold and Myford (1998) think-aloud protocol on portfolio raters to see if they fit process model of portfolio rating • Found score validity was threatened when a major process was omitted or extraneous assessment criteria were applied

28. Reliability • Nystrand, Cohen and Dowling (1993) found reliability could be significantly improved if: • Raters scored each task in response to a prompt before moving to the next task and • Raters read several examples together to decide how they were to be rated

29. Impact • Herman and Winter (1994) based on self-reports from teachers and others implementing portfolios appears to have positive effects on instruction • Vermont principals affirmed that the portfolio assessment program had beneficial effects on curriculum and instruction

30. Impact • Aschbacher’s (1993) action research cited in Herman and Winter 1994 suggests teacher’s instructional practices and their attitudes towards students changed. • Reported ways they thought about their own teaching • Two-thirds of teachers expected higher level of performance from students

31. Impact • Hirvela and Sweetland (2005) used 2 case studies showing the 2 students did not strongly endorse the portfolios as used in 2 different courses. • Seemed to need more explanations of what portfolio approaches were meant to achieve • Even with a 5% final course grade students saw the portfolio as essentially summative in nature

32. Impact • Richardson (2000) study involved classroom observations teacher and student interviews and examination of student writing and teacher response. • Found that students regard teacher responses as directives. Were not prepared to make independent judgments largely because of the threat of grades

33. The future? • More technical quality • Issue of fairness needs to be addressed • More research on impact • A move away from psychometric measurement?

34. Potential research • Equity of portfolio assessment • Validity v reliability • Nature of feedback • Portfolios for employers

35. References • Abrami, P. C., & Barrett, H. (2005). Directions for Research and Development on Electronic Portfolios. Canadian Journal of Learning and Technology, 31(3). • Belanoff, P., & Dickson, M. (Eds.). (1991). Portfolios : process and product. Portsmouth, N.H.: Boynton/Cook Publishers. • Calfee, R. C., & Freedman, S. W. (1996). Classroom Writing portfolios:Old, New, Borrowed, Blue. In R. C. Calfee & P. Perfumo (Eds.), Writing Portfolios in the Classroom. MahWah, N. J.: L. Erlbaum Associates. • Callahan, S. (1995). Portfolio expectations: Possibilities and limits. Assessing writing, 2(2), 117-151. • Case, S. H. (1994). Will mandating portfolios undermine their value? Educational Leadership, 52(2), 46-47.

36. References • Desmet, C., & Cummings, R. (2004). Negotiating the Teaching-Assessment Cycle in Writing Programs with XML. Paper presented at the World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2004 Washington DC. • Frederiksen, J. R., Sipusic, M., Sherin, M., & Wolfe, E. W. (1998). Video Portfolio Assessment: Creating a Framework for Viewing the Functions of Teaching. Educational Assessment, 5(4), 225-297. • Hamilton, S. J. (2006). A Principle-Based ePort Goes Public (and Almost Loses its Principles). In A. Jafari & C. Kaufman (Eds.), Handbook of Research on ePortfolios (pp. 434-446). Hershey: Idea Group Reference. • Hamp-Lyons, L. (1990). Second language writing: assessment issues. In B. Kroll (Ed.), Second language writing : research insights for the classroom. Cambridge: Cambridge University Press.

37. References • Hamp-Lyons, L., & Condon, W. (2000). Assessing the Portfolio: Principles for Practice Theory and Research. Cresskill, NJ: Hampton Press. • Heller, J. I., Sheingold, K., & Myford, C. M. (1998). Reasoning about Evidence in Portfolios: Cognitive Foundations for Valid and Reliable Assessment. Educational Assessment, 5(1), 5-40. • Herman, J. L., Gearhart, M., & Aschbacher, R. (1996). Writing portfolios in the classroom : policy and practice, promise and peril. In R. C. Calfee & P. Perfumo (Eds.), Writing portfolios in the classroom : policy and practice, promise and peril • (pp. x, 374 p.). Mahwah, N.J.: L. Erlbaum Associates. • Herman, J. L., Gearhart, M., & Baker, E. L. (1993). Assessing writing portfolios: Issues in the validity and meaning of scores. Educational Assessment, 1(3), 201-224.

38. References • Herman, J. L., & Winters, L. (1994). Portfolio research: A slim collection. Educational Leadership, 52(2), 48-55. • Hirvela, A., & Sweetland, Y. L. (2005). Two case studies of L2 writers' experiences across learning-directed portfolio contexts. Assessing writing, 10, 192-213. • Holt, D., & Baker, N. W. (1991). Portfolios as a follow-up option in a proficiency-testing program. In P. Belanoff & M. Dickson (Eds.), Portfolios: process and product. Portsmouth NH: Boynton/Cook. • Jafari, A., & Kaufman, C. (Eds.). (2006). Handbook of Research on ePortfolios. Hershey: Idea Group Reference. • Murphy, S., & Camp, R. (1996). Moving towards systemic coherence: A discussion of conflicting perspectives on portfolio assessment. In R. C. Calfee & P. Perfumo (Eds.), Writing portfolios in the classroom: policy and practice, promise and peril. Mahwah, N.J.: L. Erlbaum Associates.

39. References • Nystrand, M., Cohen, A. S., & Dowling, N. M. (1993). Addressing reliability problems in the portfolio assessment of college writing. Educational Assessment, 1(1), 53-70. • O'Brien, K. (2006). ePortfolios as Learning Construction Zones. In A. Jafari & C. Kaufman (Eds.), Handbook of research on ePortfolios (pp. 74-82). Hershey: Idea Group Reference. • Ostheimer, M. W., & White, E. M. (2005). Portfolio assessment in an American college. Assessing writing, 10, 61-73. • Pullman, G. (2002). Electronic Portfolios Revisited: The efolios Project. Computers and Composition, 19, 151-169. • Richardson, S. (2000). Students' conditioned response to teachers' response: Portfolio proponents, take note! Assessing Writing, 7, 117-141.

40. References • Song, B., & August, B. (2002). Using Portfolios to Assess the Writing of ESL Students: A Powerful Alternative? Journal of Second Language Writing, 11, 49-72. • Spalding, E., & Cummins, G. (1998). It was the best of times. It was a waste of time: University of Kentucky students' view of writing under KERA. Assessing Writing, 5(2), 167-199. • Stiggins, R. J. (1994). Student-centred classroom assessment. New York: Merrill. • Wagner, M., & Lamoureaux, E. (2006). Implementing an Outcome-Based Assessment ePortfolio. In A. Jafari & C. Kaufman (Eds.), Handbook of Research on ePortfolios (pp. 539-550). Hershey: Idea Group Reference. • Weigle, S. C. (2002). Assessing writing. Cambridge, UK Cambridge University Press.

41. References • Yancey, K. B. (1996). Portfolio as genre, rhetoric as reflection: Situating selves, literacies, and knowledge. WPA 19(3), 55-69. • Yao, Y., Thomas, M., Nickens, N., Downing, J. A., Burkett, R. S., & Lamson, S. (2008). Validity evidence of an electronic portfolio for preservice teachers. Educational Measurement: Issues and Practice, Spring 2008, 10-24.