Evaluating Portfolios: Mirrors Maps and Sonnets

1. Evaluating Portfolios:Mirrors Maps and Sonnets Simon Cotterill Sue Gill Jamie Thompson EPICS ePortfolios project & PDP Forum http://www.epics.ac.uk

2. The story so far • Simon meets Jamie • ePortfolio meets PDP • Debate widens • EPICS and the NE Regional Forum • Philosophy, Practice and Evaluation • Symposium

3. EPICS - North East regional collaboration around e-portfolio progression pathways with illustrative studies Part of the JISC Distributed e-Learning Programme. Partners include: 5 HEIs FE colleges (OWL network) JISC Regional Centre Case studies -demonstrate feasibility / scalability -supporting life-long learning http://www.epics.ac.uk

4. Before we begin… What do we mean by “Evaluation” & “Research”? • Evaluating which ePortfolio product to use -checklists / criteria eg. Richardson & Ward, 2005 • Formative Evaluation (independent or practitioner) • Action Research (practitioner) • Summative Evaluation – making a judgment; did the intervention achieve its stated goal(s) • Research (= rigorous scholarly evaluation?) • tends to be more grounded in theory • narrow, well defined research question • prospective research plan • aims to be reproducible (difficult in complex educational environments) } to improve (iterative)

5. Clarity of Purpose Back to basics: • Evaluation should address the main aims / purpose(s) of your portfolio / PDP • What is the purpose? • How will you know if it has been achieved? i.e. measure it? (quantitatively and/or qualitatively) • Use this to inform your evaluation strategy • Are there existing (validated) research instruments that will address your questions? e.g. ELLI (life-long learning) e.g. ASSIST (approaches to learning) e.g. Bloom’s taxonomy (reflective learning)

6. Diez, M. (1994) The Portfolio: Sonnet, Mirror and Map in Burke, K. (1996) Professional Portfolios. Arlington Heights: IRI Skylight

7. See also • Story, Journey, Laboratory (P and L Paulson) • Positivist v Constructivist (Wilkerson and Lang) • My digital clone, work companion, butler, dashboard, planner, IPR Management assistant (Serge Ravet) • Toothbrush, Caterpillar, Kaleidoscope, Window

8. Paulsons (1994) • Positivist Portfolios“The purpose of the portfolio is to assess learning outcomes and those outcomes are, generally, defined externally. Positivism assumes that meaning is constant across users, contexts, and purposes… The portfolio is a receptacle for examples of student work used to infer what and how much learning has occurred.” (p.36) • Constructivist Portfolios“The portfolio is a learning environment in which the learner constructs meaning. It assumes that meaning varies across individuals, over time, and with purpose. The portfolio presents process, a record of the processes associated with learning itself; a summation of individual portfolios would be too complex for normative description.” (p.36) • Tension between two approaches“The two paradigms produce portfolio activities that are entirely different...“The positivist approach puts a premium on the selection of items that reflect outside standards and interests....“The constructivist approach puts a premium on the selection of items that reflect learning from the student’s perspective.” (p.36)

9. Map Mirror Sonnet Where am I going and how will I get there? Reflecting … beyond surface learning … making meaning. Presenting my truth. Who I am and how I know. Iterative PDP / ePortfolio

10. Stakeholder Mode

12. Evaluating a ‘MAP’ type portfolio Example: ePortfolio for year 4 Medicine, Newcastle University Cotterill SJ, Aiton J, Bradley PM, Hammond GR, McDonald AM, Struthers J, Whiten S. A flexible component-based ePortfolio: adapting and embedding in the curriculum. In: In Jafari A, Kaufman C, ed. Handbook of Research on ePortfolios. Pennsylvania: Idea Group Inc, 2006 Mapping/Planning Recording/Evidencing Reflection

13. Fixed-response questions (eg Likert scale) n = 157 Q. Building the ePortfolio was a useful learning experience 80% thought it was a useful learning experience Q. Having clearly defined intended learning outcomes influenced the way in which I approached the [placement] 72% felt that the LOs influenced their approach 83% felt they had recorded good evidence 93% reflected on their learning after the [placement] n=157 (84% response rate)

14. “It encouraged me to really give thought to what I wanted to achieve during the [placement], which was especially useful as this was my first [placement]. As a result of the portfolio I think I got much more out of the [placement] than I would have otherwise.” “Most of the things I learned couldn't be 'measured/quantified' so I felt it was of little value to try and invent a way in which they could.” “It made me concentrate on creating aims at the start of the [placement] and allowed me to plan the [placement] with my supervisor in a defined way. Overall it made my learning for the [placement] more organised and focused.” …but simple open-ended questions can be very illuminating: eg. what did you most like [dis]like about using the portfolio?

15. ELLI Achievement v Personal development Characteristics of learning power Measurable change ELLI and HE 2006 – 2008 Situated research Range of contexts Interested in participating? Measuring Development

16. Stakeholder Mode

17. Group Discussion Critique of the model • In terms of your own experience and use of PDP/ePortfolio; what do you think of the MMS metaphor? Which other perspectives are important? Developing the model • Identify a type of portfolio and a perspective relevant to your practice or experience; share and discuss relevant research or evaluation questions and methodology

18. Evaluation of achievement not appropriate (positivist assumptions) Causative connections between PDP/ePortfolio difficult to establish Notion of situational evaluation The problem of evaluation

19. Thank you for your participation! Contacts: Jamie.Thompson@northumbria.ac.uk S.J.Cotterill@newcastle.ac.uk http://www.epics.ac.uk EPICS ePortfolios project & PDP Forum http://www.epics.ac.uk