Measuring the impact of a pedagogical change or innovation

1. Measuring the impact of a pedagogical change or innovation Strategies for Dissemination Anastassis Kozanitis Educational consultant Polytechnique Montréal

2. Do youknowanyone in thisroomwho has innovated? • 1- no • 2- yes • 3- I’m not sure

3. Haveyouinnovatedin thepast 2 years? • 1- no • 2- yes • 3- I’mnotsure • 4- I would of liked to, but I lacktheressources

4. Whydidyou (not) innovateorwhy do you (don’t) want to innovate? • Activity 1: Think-pair-share

5. Supporting elements? Institutional or departmental levels • Pedagogical culture • Pedagogical training for teachers • Educational support center • Valuing teaching (support of the institution or department) • Tangible recognitions (awards, $, career progress) • Educational leadership of the authorities (Dean, Head of department, etc.) • Support (human, material and financial resources) • Work collaboratively within a program approach

6. Elements that inhibit or push back • Doubt that it will work. • Negative reaction from students. • Sensing that there was no added value. • Material, financial or contextual constraints (e.g. group size), lack of time or work load too large. • Unsuccessful blind trial and error.

7. Supporting elements? Individual level • Strength of your personal beliefs • Desire to make your work more interesting • Professionalization of teaching (3 levels of intensity) • 1) Reflective practice (Schön, 1983) • 2) Scholarly teaching (Richlin, 2001) • 3) Scholarship of teaching and learning (Potter & Kustra, 2011)

8. Outline and goals • Choose what elements to measure as a result of a pedagogical innovation implementation • Identify how to measure and analyze its impact on students • Develop strategies to disseminate the findings and motivate colleagues to do the same. • Commit to practice a scholarly teaching

9. Circle of innovation

10. pedagogical innovationDefinition: • A deliberate activity that aims to introduce novelty in a given context, and seeks to substantially improve the learning of students in a situation of interaction or interactivity (Béchard & Pelletier, 2002)

11. Initiatives for innovationswith or without ICT • Institutional • Changes in software platform(Moodle) • Curricular changes (competency-based curriculum) • Departmental • Curricular changes(program-approach) • Structural changes (course fusion) • Individual • Changes to active learning methods

12. The Active learning continuum* Prolonged time or large scale activities Limited time or small scale activities Interactive lectures Student centered Instructor centered Think-pair-share Case study Project-basedlearning Roleplaying *Adaptedfrom Michael Prince (2011). Active cooperativelearning. BucknellUniversity. http://www.asee.org/documents/conferences/annual/2011/plenary-michael-prince.pdf

13. Teachingmethods Instructor centered Student centered • Problem-based learning • Guided discovery • Case study • Project-based learning • Role playing • Simulation • Tournament • Peer instruction • Team-based learning • Laboratory • Discussions • Lecturing • Podcasting • Video-conferencing • Demonstration

14. is there something you wish to change in your teaching? • Whatproblemshaveyouobservedconcerningyourstudents’ learning? • Whatsolutionshaveyoutried and whatweretheresults? • Example: implementinga new method • Problem-basedlearning • Project-basedlearning • Case studies • Team-basedlearning • Other?

15. relevant Pre-active Questions • How can I fix my mistakes from last semester? • How can I get my students’ attention? • What learning activities should I provide to help students understand the subject? • How can I arouse their interest and motivation? • What content-specific resources do I need? • How will I consider student heterogeneity and their different ways of learning? • What are my expectations about student productions?

16. Activity 2 current situation – Desired situation • Step 1. • Identify, in your practice, a specific situation you want to improve or a recurring problem you wish to solve. • Step 2. • Using one or two words, post your situation or problem on a piece of paper for everyone to see. • For example: • Classroom participation • Motivation • Critical thinking • Student attitude

17. Scholarship of teaching and learning (SoTL) • Definition • A careful analysis from the teacher about students’ learning, before and/or after a systematic and intentional change of pedagogy, with an evaluation of the results of this change, and dissemination of these results (Potter y Kustra, 2011).

18. Steps to Sotl • 1 – write a research question 2 - design the study 3 - collect data 4 - analyze the data 5 - present and publish the results

19. Specific aspects to be measured • For example, if a teacher says he has problems with teaching large groups, you may look at the following indicators : • Absenteeism rate • Verbal or non-verbal behavior of students in the classroom • Lack of student participation during activities • Grades on tests and assignments • Results of teaching evaluations • Student comments

20. Activity 3choosing the specific aspects to measure or assess • Identify the specific aspects of the general theme you want to measure. • You have to be very accurate. For example, it is not enough to say I want to measure motivation. But rather what aspect of motivation (interest in the course or learning activity, perception of usefulness of what they are learning, amount of time or effort invested in the course, the perception of self-efficacy, etc..). • You can measure multiple aspects, however make sure that the information you seek is: available, valid, accurate, and true.

21. Activity 4identify how to measure or assess the specific aspects • It is the procedural or operational part • What instrument or process do you need in order to measure the specific aspects • 2 options:1) You can choose and adapt an existing instrument 2) You can build or create an instrument • In both cases you must validate the instrument • How will you do the validation? • Who are your sources? • With how many participants are going to validate the instrument?

22. methods or tools for measuring the impact of the pedagogical change • Quantitative methods • Quasi-experimental design • Likert type questionnaire • Existing tools like the NSSE • Comparing means (ANOVA, t-test) • Regressions • SEM o Path analysis • Qualitative methods • Case study • Semi-structured interviews (NVivo) • Focus group • Quality circle • In situ observations using a form

23. Examples of methods • A valid questionnaire on motivation; • Some open-ended questions regarding opinions; • A test or an examination of knowledge or skills; • An observation form of classroom behavior; • Self-assessments, peer-assessments, student evaluations; • Etc.

24. dissemination • How to get your colleagues involved? • How to convince them to change? • How can you teach them what you have learned? • What are the usual arguments for inaction?

25. dissemination cognitive affective

26. Factors related with pedagogical innovation development (Béchard & Pelletier, 2002) • 4 categories : • External environment • Institution • Department • Classroom

27. Estrategias

28. opportunities and threats

29. Strategies for dissemination • Organizing an event for presentation of best practices (institutional or department level) • Formal and informal pedagogical training • Mentoring – coaching • Teamwork with a colleague giving the same course • Participating in educational conferences (STLHE, AERA, CHEP, Lilly conference, ISSOTL, etc.) • Funding pedagogical initiatives (specific criteria)

30. What can lead to innovations? 1 - Personal, informal initiatives 2 - Evaluation of teaching 3 – Work in collegialityto prepare assessment activities 4 - Work in collegially to prepare course material 5 - Promotion of educational initiatives (funds, special events e.g. Teaching day) 6 - Scholarship of teaching and learning (SoTL) 7 - Training and pedagogical support for teachers 8 - Paradigm shift (from teaching to learning) 9 - International collaboration projects

31. Activity 7 • Share with a neighbor what other strategies are adequate given your context. • Think of any resistance you anticipate.

32. conclusion • Degree of satisfaction with the workaccomplished? • Degree of satisfaction with the help? • Any questions remain? Obrigado!

33. References • Béchar, J-P. & Pelletier, P. (2002). Dynamique des innovations pédagogiques en enseignement supérieur: à la recherche d’un cadre théorique. Available online at:http://neumann.hec.ca/oipg/fichiers/2002-002_-_Dynamique_des_innovations_pedagogiques_en_enseignement_superieur.pdf • Cornet, J. & Voz, G. (2010). «Comment former de futurs étudiants réflexif?» Education and formation, [En ligne], octobre, e-294, http://ute3.umh.ac.be/revues/include/download.php?idRevue=10&idRes=81. • Hannan, A., English, S. & Silver, H. (1999). Whyinnovate? Somepreliminaryfindingsfrom a researchprojecton “Innovations in teaching and learning in highereducation”. Studies in HigherEducation, 24(3), 279-289. • Kapur, M. (2012). Productive failure. Available online at: http://www.isls.org/icls2012/downloads/K2Kapur.pdf • Lenoir, Y. (2012). Analyse réflexive : un outil de questionnement sur la gestion des activités d’enseignement-apprentissage, Outil 2. Vivre le primaire, 25(4). • O’Brien, M. (2008). Navigating the SoTLLandscape: A Compass, Map and Some Tools for GettingStarted. International Journal for the Scholarship of Teaching and Learning, 2(2), 1-20. • Perrenoud, P. (2001). Développer la pratique réflexive dans le métier d'enseignant: professionnalisation et raison pédagogique: ESF éditeur. • Potter, M. K., & Kustra, E. (2011). The relationshipbetweenscholarlyteaching and SoTL: Models, distinctions, and clarifications. International Journal for the Scholarship of Teaching and Learning. • Schön, D.A. (1983). The Reflective Practitioner. San Francisco: Jossey-Bass.