Evaluating learning about partnership with service users using Concept Mapping

1. Evaluating learning about partnership with service usersusing Concept Mapping Roxana Anghel roxana.anghel@anglia.ac.uk Joanna Fox joanna.fox@anglia.ac.uk

2. Level 2b outcome (Kirkpatrick, 1997; Barr, 2000) – Acquisition of knowledge – concepts, procedures and principles of working with service users and carers (in Carpenter, 2005:6) • Focus: • Evaluate the progress in learning about working in partnership with service users throughout the first two years of the BA course

3. Rationale • The BA (Hons) SW degree - strong emphasis on training focused on partnership with service users (DoH, 2002) • ‘All social workers will learn and be assessed on partnership’ (p16) • Core expectation that defines ‘good practice’ (NOS, GSCC, QAA) • Service users and carers consider ‘very important’ for students to learn how to involve them in assessing their needs, and how to treat them with respect (Barnes et al, 2000; Anghel & Ramon, 2009) • However a complex concept with varying definitions, in practice and education often implied rather than made explicit (Taylor & Le Riche, 2006)

4. Generally taught in an embedded or discrete way – ‘no defined partnership curriculum’, not explicitly defined as an outcome (Taylor & Le Riche, 2006). • Anglia Ruskin - embedded and discrete approaches to learning (involvement of service users in curriculum delivery, and teaching about aspects of partnership on various modules). • As learning about partnership is spread across various learning opportunities on the BA it is important to assess the students' gradual understanding of the concept and acquisition of a meaningful framework before becoming practitioners.

5. David Ausubel (1968, 2000) – Meaningful Learning (opposed to rote learning) is achieved when • the learner assimilates new concepts into a pre-existing conceptual framework stored in the long-term memory • the learner deliberately seeks the logical connections between new and old learning thus choosing a meaningful learning mindset • the concepts progress in acquiring meaning as new concepts are added • retention depends on the strength of the meaningful connections between the concepts stored • meaningful learning is personal • Joseph Novak (1984) – Concept Mapping

6. Concept Mapping (CM) • A visual map of the summary of an individual's understanding of a knowledge domain (Novak & Canas, 2006) • Concepts – perceived regularities in events or objects, or records of events or objects, designated by a label. • Events, objects, even emotions or feelings (Freeman & Jessup, 2004) • Differs from brainstorming and mind maps by making explicit the concept-links • Initially designed as a research instrument used mainly in science education, but increasingly being used in a variety of teaching and research fields including social work (Anghel & Fox, 2008; Webber et al., unpublished)

7. T1 map before MA module Cross link Concept link

8. Research Design • Before and After quasi-experimental longitudinal design - three measurements - two academic years • CM and questionnaire at T1 (induction week BA); T2 (end year 1); T3 (end year 2) • Questionnaire • Demographic details • Students’ previous experience of social work and • Rating possible impacting factors • Non-academic (media, workplace, personal experience) • Classroom-based (various modules, lecturer teaching style, discussing with colleagues) • Non-classroom-based academic factors (practice/ observational placement) • ‘no impact’, ‘little impact’, ‘some impact’, ‘great impact’.

9. Research Process • University colleagues consulted on the idea of outcomes evaluation, on the research design, and later on using CM. • Some allies but initially difficult to propose an outcomes evaluation culture • Advisory group – mixed – working mostly on the development of the CM analysis method • 2 service users • 2 year three SW students • 3 researchers (one familiar with SPSS analysis) • 3 senior lecturers • Students and service users paid for their time and travel (additional £5000 secured from University fund)

10. Research Procedure • During induction week BA students received one hour instruction on CM, a handout with the instructions, and the participant information sheet • Two days later they were invited to participate by signing a consent form – participants given 30 mins (negotiation with lecturers) to complete handwritten CMs individually. Student with dyslexia supported. • Instruction: complete the conceptual map of ‘working in partnership with service users’ • Questionnaire completed after each CM • Before the subsequent measurements the students received 15 mins of instruction as reminder

11. Sample • At T1 – 13 (17% of BA cohort) • At T2 – 6 – attrition 53% • T2 sample • Females • 4 White British • 1 student with dyslexia • 3 students with previous experience of social work (employment, personal experience) • 1 Asian – previous experience of social work (volunteering) • 1 Black Caribbean – no previous experience • Age ranges: 36-45

12. Analysis • Novak (1984) – structural method – scoring the maps by giving specific values to hierarchies, concept-links and cross-links. • Arbitrary values • Mechanical counting of correct links • Hierarchies may not be relevant to all types of knowledge (Freeman, 2004) • West el al. (2000) – relational method – based on degree of foundational or core relationship to the main concept.

13. Scoring instructions developed by the team • Combination of structural and relational methods • Acknowledges the importance of the theoretical basis of concept-links and cross links, but rejects the value of hierarchies and the mechanical counting of concepts • Acknowledges the importance of the degree of core relevance of the ideas included in the map to the main concept

15. Developing the Scoring instructions • Advisory group – service user involvement • ‘Blind’ scoring • 2 raters – scoring independently, meeting to discuss were wide differences in scoring (only a few) • Instructions altered three times and scoring repeated with three different sets of raters • The final version of the scoring method was then applied to the entire set of maps in one go by two new raters: the lead researcher and a senior lecturer • Pearson (0.96) and Spearman (0.87) – Strong inter-rater reliability • Wilcoxon (r=0.854) – Temporal stability

16. Findings • Increased knowledge demonstrated • Significant limitation – only 6 pairs of maps, then 4 Map Scores • Most impacting sources of learning at T1: Ethics and Values; observational practice; Poverty, Social Exclusion and Social Work, and academic reading. • At T2:Social Work with Children and Families, Social Work with Adults, practice placement; Principles and Skills of Social Work; academic reading; and discussing with colleagues

17. Student 4 profile • Female 36-40 • White British • No disability • Previous contact with social work through employment in NHS – generally good appraisal of her experience • Scores • T1: 35.5 • T2: 80.5 • T3: 136.5

18. T1: 35.5

19. T2: 80

20. T3: 136.5

21. Student 3 profile • Age range – 36-40 • Asian ethnic origin • Previous volunteering work – appraised as ‘excellent in opening my eyes to a lot of issues concerning social services’. • Scores • T1: 29.5 • T2: 26.5 • T3: 48

22. T1: 29.5

23. T2: 26.5

24. T3: 48

25. Conclusions • Some evidence of significant increase in knowledge – however, inconclusive due to small number of maps – thus CM as method for measuring social work education outcomes needs to be explored further • In particular the balance between quantitative and qualitative evaluation of maps has to be strengthen. • CM – has potential for being used routinely as evaluation method – validity depends on familiarity with the tool and adequate time given to learn it and complete it. • Being based on the theory of meaningful learning CM pertains to a wide range of subject matters in social work and other fields. • Workshop: CM as formative evaluation method, learning and teaching aid, and empowering tool in social work practice

26. Selected Bibliography • Anghel R. & Ramon S. (2009) Service users and carers’ involvement in social work education: lessons from an English case study. European Journal of Social Work, 12 (2), p. 1 – 15 . • Ausubel D. (1968) Educational psychology: a cognitive view. New York: Holt, Rinehart and Winston • Carpenter J. (2005) Evaluating Outcomes in Social Work Education, SCIE, SWAP and SIESWE, London • DoH (2002) Requirements for social work training [online]. • Cmap software FREE download -http://cmap.ihmc.us/download/ • Freeman L.A. & Jessup L.M. (2004), The power and benefits of concept mapping: measuring use, usefulness, ease of use, and satisfaction, International Journal of Science Education, 26(2), 151-169 • Novak J.D. & Gowin D.B. (1984), Learning how to learn, Cambridge University Press, Cambridge • Novak J. & Canas, A.J. (2008) The Theory Underlying Concept Maps and How to Construct and Use Them, http://cmap.ihmc.us/Publications/ResearchPapers/TheoryCmaps/TheoryUnderlyingConceptMaps.htm • Taylor I. & Le Riche P. (2006) What do we know about partnership with service users and carers in social work education and how robust is the evidence base?, Heath and Social Care in the Community, 14(5), 418-425 • West D.C., Pomery J.R., Park J.K., Gerstenberger E.A., & Sandoval J. (2000), Critical thinking in graduate medical education – a role for concept mapping assessment?, American Medical Association, September 6, 284(9), p 1105-1110 • West D.C., Park J.K., Pomeroy J.R. & Sandoval J. (2002) Concept mapping assessment in medical education: a comparison of two scoring systems, Medical Education, 36, 820-826