Evaluating the success and sustainability of curriculum and pedagogical innovation

1. Evaluating the success and sustainability of curriculum and pedagogical innovation Elizabeth Godfrey

2. Evaluation – gathering of information to understand and make judgements about the project and its outcomes • Periodic assessment of results • Appropriateness, effectiveness, efficiency, impact, sustainability • Identifies intended and unintended effects • Identifies what worked and what didn’t • Provides level of judgement about the overall worth of intervention • Key to project improvement • Influence decision-making or policy formulation through the provision of empirically-driven feedback  sustainability

3. What is the purpose and scope of the evaluation? • Why is the evaluation being done? • How will the information be used? • What evaluation form and approach will be most suitable for this study? Formative – monitoring progress to improve approaches Summative – overall perspective – focus on value or worth of project and designed for accountability or continuation purposes

5. Evaluation and Assessment • Evaluation and assessment have many meanings…one definition: • Assessment - is gathering evidence • Evaluation - is interpreting data and making value judgements • Examples of assessment and evaluation • Individual’s performance(grading) • Program’s effectiveness(accreditation) • Project’s progress and success(monitoring and validating)

6. Evaluation and Project Goals/Outcomes • Effective evaluation starts with carefully defined project goals and expected outcomes • Goals provide overarching statements of project intention • What is your overall ambition? • What do you hope to achieve? • Expected outcomes identify specific observable results for each goal • How will achieving your “intention” reflect changes in student behavior? • How will it change their learning and their attitudes?

7. Goals, Expected Outcomes, and Evaluation Questions • Goals → Expected outcomes → Evaluation questions • Questions form the basis of the evaluation process • Evaluation process collects and interprets data to answer evaluation questions

8. Evaluation Questions like… • To what extent has the project been implemented as planned? • How well has the project been co-ordinated across different institutions? • To what extent have the intended outcomes been achieved? • Were there any unintended outcomes? • How well have the needs of staff been met? • To what extent have the intended student learning outcomes been achieved? • What measures if any have been put in place to promote sustainability of the project’s focus and outcomes? • Are students better able to describe the effects of changing some feature in a simple problem as a result of the intervention? • What lessons have been learned from this project and how might these be of assistance to other institutions?

9. How will the information be collected, analysed and presented? What type of data is most appropriate? How will data be analysed and presented? What are the most appropriate methods of data collection? What ethical issues are involved in the evaluation?

10. Examples of Tools for Evaluating Learning Outcomes  • Surveys • Forced choice or open-ended responses • Observations • Actually monitor and evaluate behavior • Interviews • Structured (fixed questions) or in-depth (free flowing) • Concept Inventories • Multiple-choice questions to measure conceptual understanding • Rubrics for analyzing student products • Guides for scoring student reports, test, etc. • Focus groups • Similar to interviews but with group interaction

11. A NZ exemplar An Intelligent Tutoring System for Engineering Mechanics TanjaMitrovic, Charles Fleischmann, Brent Martin, PramudiSuraweera The planned evaluation will focus on the following questions: • Is the developed ITS effective? • Does it support learning better than the traditional approach? • Does the system increase students� motivation? We will compare performances of two groups of students. One group will learn the material in the traditional way, via lectures and tutorials. The other group will attend lectures, but the tutorials would be replaced by interaction with the system. Time will be controlled. We will require students to sit pre- and post-tests, to measure their knowledge. We will also collect data about their actions while solving problems, and analyze the data.

12. What are the criteria for making judgements about the project and its outcomes? • Judgements will be required for each key evaluation question • Need to develop indicators or targets - standards and levels regarded as acceptable • Could use benchmarks – location or source of best practice for comparison

13. What resources and skills are required to conduct the evaluation? Who? key question underpinning the budget • Should it be an individual or a team? • Insiders or outsiders? • Skills – qualitative and/or quantitative • Prepare an evaluation brief or terms of reference • When to involve the evaluator?

14. Other Sources • NSF’s User Friendly Handbook for Project Evaluation http://www.nsf.gov/pubs/2002/nsf02057/start.htm • Australian Learning and Teaching Council Grants Scheme – Evaluating Projects http://www.altc.edu.au/extras/altc-gsep/evaluation_plan.pdf • Student Assessment of Their Learning Gains (SALG) http://www.salgsite.org/

15. Summary • Think about evaluation, purpose and scope from the beginning of the project • Ensure appropriate data is collected eg pre/post intervention • Evaluation provides supporting evidence to influence decision-making or policy formulation through the provision of empirically-driven feedback  sustainability

16. Questions…

17. Thank you Elizabeth Godfrey Engineering Education Research and Project Management lizgodfrey@gmail.com