Evaluation: Find out what visitors think Improve what we do

1. Evaluation: Find out what visitors think Improve what we do Nicola Bell, MA culture ~ evaluation ~ learning nicola@nicolabell.co.uk www.nicolabell.co.uk

2. What is Evaluation? Evaluation is the systematic collection of information about the content, characteristics, and outcomes of a programme to make judgements about it, improve its effectiveness, and/or inform decisions about future activities. (Kirby, P., and Bryson, S. (2002) Measuring the Magic? Evaluating and Researching Young People’s Participation in Public Decision Making, London: Carnegie Young People's Initiative) Consulting with young people - literature review http://www.creative-partnerships.com/data/files/consulting-young-people-13.pdf

3. What is evaluation? • It is Quality Control, to find out: • Is it good? • How can we make it better?

4. Why do we want to carry out • an evaluation? • To find out how to develop a project • To find out if the project was successful • To find out if the money was well spent • To get money for future activities • To find out what non-visitors think

5. If you don’t evaluate, you might: • waste time • waste money • produce something that you can't change • lose interest from your target audience • lose an opportunity to learn something useful • lose an opportunity to tell people about the good work you are doing • lose funding

6. When planning an evaluation, remember: • Plan the evaluation at the beginning of the project • Allow enough time to collect and analyse the data • Allow enough money for the evaluation • Consider using a freelance evaluator who can bring an independent view to the project

7. Who is the evaluation for? • Museum staff • Project partners • Project participants • Funders • Local or national government, future partner organisations • Visitors

8. Types of evaluation • Front end - e.g. planning new exhibit • Prototyping - e.g. for a new interactive • Baseline - e.g. visitors’ knowledge at the beginning • Formative - e.g. how the programme is developing • Summative - e.g. was the programme effective? • Usually evaluations only cover a short time; longitudinal studies (over several years) would be good

9. Methodology: • Link the evaluation to the project’s aims and objectives • Aims and objectives must be made clear at the start • Aim = what you are going to do? What changes will happen? (to participants, to the museum, to staff) • Objectives = how you are going to do it? • Objectives should be SMART

10. SMART objectives: S - specific M - measurable A - agreed R - realistic T - time-based

11. Outputs, Outcomes and Impact Logic Framework model: Inputs > activities > outputs > outcomes > impacts Examples:

12. Outcomes can be: Positive Negative Expected Unexpected They can happen: During the project At the end of the project

13. What sort of data do you need to collect? • Quantitative data: • How much? • How many? • Qualitative data: • What was it like? • Opinions and views

14. Decide how many people you need to gather data from: (assuming 3% error) http://www.custominsight.com/articles/random-sampling.asp

15. Sampling • The sample should be representative of the whole population (e.g. everyone who attended the exhibition) • Random sampling: e.g. every 5th or 15th person who leaves the exhibition • Convenience sampling: e.g. people who are available • Snowball sampling: e.g. for people who are hard to reach - ask some people, then ask them to suggest other people (e.g. their friends) to interview

16. Make sure that the data collection method is suitable for the people and the information • Triangulation = using several different methods, to get strong evidence • http://www.creative-partnerships.com/data/files/consulting-young-people-13.pdf • Make sure that you have appropriate permissions for photography, audio recording, interviewing children

17. Methods of collecting data: • Mediated (with a member of staff): • Questionnaires (with interviewer) • Interviews • Focus groups • Observation or tracking • Participant observation • Video • Personal Meaning Mapping (concept maps)

18. Need not (necessarily) be mediated: • Questionnaires (self-completion) • Diaries, journals, scrapbooks • Comments books • Comments cards • Drawings • Mind maps • On-line surveys (e.g. www.surveymonkey.com) • Blogs • SMS, Facebook, Twitter

19. Concept map: visitors write what they know etc about the topic, before the visit the exhibition. Afterwards, they write (in a different colour) what they now know, think, feel etc. Analyse comments using e.g. Generic Learning Outcomes. What do you know, think, feel about: Etnografski Muzej Zagreb

20. Make an Evaluation Plan • Who are the participants? • What is the activity? • Who is the evaluation for? • What are the aims and objectives? • Inputs, activities, outputs, outcomes, impacts? • Baseline, formative, summative? • Which methods? for each part of the project • for each group of people • (participants, staff, artists etc) • How will you present the evaluation?

21. Analysing the data • The analysis must reflect the evaluation aims and objectives • Allow plenty of time (and therefore money) for analysis • A written report is usually necessary for managers, funders and other stakeholders • The report should give recommendations for the future • The report can be in alternative formats …

22. Alternative formats: • Website, blog • DVD or video • Performance • Presentations • Set of reference cards • Book

23. What will you do with the evaluation when it is finished? • Present it in a suitable format • Tell the relevant people about it • Ensure that recommendations are acted upon • Don’t leave it on the shelf!

24. Best Practice in Educational and Cultural Action • http://ceca.icom.museum/node/208 • Conception of the project • Achievement • Evaluation • Remediation / Improvement

25. Inspiring Learning for All • http://www.inspiringlearningforall.gov.uk/ • Generic Learning Outcomes • Generic Social Outcomes • Ways of measuring hard-to-measure things • Shows to other organisations what museums do, to benefit their visitors

26. Generic Learning Outcomes

27. Generic Social Outcomes

28. Working with community groups • Youth groups • Services for disadvantaged people • Services for people with disabilities • Communities of place • Communities of interest • Work with partner organisations, who already have a good relationship with local people

29. Why talk to teenagers and children? United Nations Convention on the Rights of the Child, article 12: “assure to the child who is capable of forming his or her own views the right to express those views freely in all matters affecting the child, the views of the child being given due weight in accordance with the age and maturity of the child.” Every child should be able to express their views and influence decisions affecting them. “Child” is anyone under 18 years old, so includes teenagers.

30. Why talk to teenagers and children? • They have views in their own right - they are not just part of a family, part of a school, or on the way to being an adult • It develops a relationship between the teenagers and the museum • They can learn skills, e.g. decision making, interviewing, analysing data • They can have a positive influence on their community • It can increase their confidence and social skills

32. Ways of involving teenagers in the • museum: • Volunteer tour guides • Provide training for them as volunteers in shop and café - good skills for jobs • Young people’s forum • Curating exhibitions • Creating interpretation (e.g. film, guide) for an exhibition • Projects to develop skills (e.g. reading, writing, art)

33. Young people as evaluators • It is better if young people are involved in planning and doing the evaluation • They can learn, and grow as a person • Everyone (any age) brings their own knowledge and world-view • Everyone makes their own meaning • UNICEF guide to participatory evaluation with teenagers: • http://www.artemis-services.com/downloads/tools-for-participatory-evaluation.pdf

34. Maslow’s Hierarchy of Needs

35. Outcomes Star: a way to measure things like increase in self-confidence http://www.outcomesstar.org.uk/

36. Do… • Make it fun! • Explain the process and what influence they can have • Provide a welcoming place • Provide food which young people like • Respect young people’s views; confidentiality, privacy • Celebrate achievements • Treat young people as individuals, not as a group • Use e.g. artists to help with creative evaluation • Be aware of needs of people with physical / learning disabilities • Tell young people the outcomes of the project • Provide help with transport to the museum

37. Don't… • Make assumptions - about what they know (or don’t know), want or can do • Promise too much • Forget about child protection (especially in mixed age groups) • Do adult things e.g. boring meetings • Consult too much - people will get bored • Do it too quickly – good consultation takes time