PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY

1. PhD, ProfessorKATRIN NIGLAS TALLINN UNIVERSITY VI KONFERENCJA EWALUACYJNA WARSZAWA, 13-14.12.2010 Podejścia mieszane – przyczynek do modelu „piątej generacji” badań ewaluacyjnych Some Visions for FifthGenerationEvaluation

2. Some background information • My interest in research methodology goes back to the beginning of 1990s. • Since 1994 I have been a lecturer in data analysis; since 2004 also lecturer in research methods. • My (5y+2y) Master’s thesis, defended in 1996, was about teaching statistics in the framework of research methods courses. • Got interested in mixed methods research in 1998 during my one-year study in Cambridge University • In the doctoral dissertation published in 2004 I focused on the combined use of quantitative and qualitative approaches in educational research • Belonging to the editorial board of Journal of Mixed Methods Research and International Journal of Multiple Research Approaches

3. Further reading related to the topic of speech • Niglas, K. (2010). The multidimensional model of research methodology. An integrated set of continua. Tashakkori, A., Teddlie, C. (Eds). Handbook of Mixed Metods Research. 2nd Ed. Sage Publications Ltd • Niglas, Katrin (2008). How the novice researcher can make sense of mixed methods designs. International Journal of Multiple Research Approaches, 1, 13 - 33. • Niglas, K.; Kaipainen, M.; Kippar, J. (2008). Multi-perspective exploration as a tool for mixed methods research. . M.M. Bergman (Ed). Advances in Mixed Methods Research: Theories and Applications. Sage Publications Ltd • Niglas, Katrin (2007). Spreadsheet Software Can Facilitate Mixed Methods Research – Using Old Tools in a New Context! Journal of Mixed Methods Research, 1(3), 297 – 29 • Niglas, K. (2004) The Combined Use of Qualitative and Quantitative Methods in Educational Research. Tallinn Pedagogical University. Dissertations on Social Sciences. Short version: http://www.tlulib.ee/files/arts/95/nigla32417030233e06e8e5d471ec0aaa32e9.pdfPS! Published version of the dissertation (book) is available by request (e-mail: katrin@tlu.ee)! • Several papers available at Education Line http://www.leeds.ac.uk/educol/

4. Change in the aims and forms of evaluation Guba and Lincoln (1989) describe 4 generations in evaluation of social and political programs: • Evaluation ≈ measurement, testing • Evaluation ≈ description, mapping • Evaluation ≈ substantive evaluation, judgement, advise • Evaluation ≈ collaboration, negotiation, initiation and guidance of change

5. From positivism to constructivism …Critique of the first three generations by G & L (1989) • A tendency towards managerialismmanagers who organise evaluation as well as evaluators are outsiders and their relationship with/to stakeholders is disempowering • A failure to accommodate value pluralismscientific mode comes with the claim of the value-freedom, but as evaluation is essentially about valuing, the value-free in practice means imposing the values of one group and not accommodating value differences • An overcommitment to the scientific paradigm of inquirycontext stripping; overdependence on quantitative measurement; belief in scientific truth leaves no room for negotiation or alternative explanations and frees evaluators from moral responsibility

6. From positivism to constructivism …Main principles for 4th generation evaluation by G&L (1989) 4th Generation evaluation • is organised by the claims, concerns, and issues of stakeholding audiences • utilises the methodology of the constructionist paradigm • is a process whereby evaluators and stakeholders jointly and collaboratively create (or move towards) a consensual valuing construction of some evaluand This process is: local, sociopolitical, emergent, continuous, recursive, and divergent, involves teaching/learning of all parties and sets them into hermeneutic dialectic relationship

7. From positivism to constructivism …Where are we 20 years after manifestation of 4th Generation? • Are we there? (read: has constructivist paradigm won the paradigm wars and we all work within this framework?) • Do we want to be there? (read: are we convinced that constructivism as The paradigm is a desired end for our journey on conceptualising evaluation and it’s methodology?) • Do we have an alternative?(read: is our choice constrained by positivism and constructivism or …?)

8. From positivism to constructivism …My answers to the three questions set on the last slide: • Are we there? (read: has constructivist paradigm won the paradigm wars and we all work within this framework?) No!– a quick glance at the evaluation practice tells us that wealth of evaluation studies are not conceptualised and carried out in this framework

9. From positivism to constructivism …My answers to the three questions set on the last slide: • Do we want to be there? (read: are we convinced that constructivism as The paradigm is a desired end for our journey on conceptualising evaluation and it’s methodology?) No, not all of us!– Even though there is a lot to learn from constructivist tradition many of us resist the forced choice between positivism and constructivism and do not accept all the assumptions that constructivism brings with it (i.e relativist ontology, strong belief in and either or stance on paradigms, overemphasized belief in impossibility of representativeness and generalizations as well as that change cannot be engineered, etc.)

10. From positivism to constructivism …My answers to the three questions set on the last slide: • Do we have an alternative?(read: is our choice constrained by positivism and constructivism or …?) Yes!We can:– replace the bi-polar paradigmatic view by the conceptualization of research methodology as a multidimensional integrated set of continua – reason that the research/evaluation practice is inevitably lead by sophisticated and changing personal worldviews or mental models of the parties rather than well defined and incommensurable paradigms – set facilitating change as the sovereign aim and the staple framework for the evaluation without having to declare either the stakeholders’ or the managers/experts’ position, needs, views, etc. being always paramount

11. The idea of incommensurable paradigms • The Conventional and Constructivist Belief Systems(Adapted from Guba and Lincoln 1989)

12. Common dichotomies in methodological literature

13. Arguments to support rejection of the idea of paradigms and moving towards the ideaof continuum • view which divides social research into limited number of paradigms, whether based on philosophical categories or methodological distinction, is misleading • research practice is much more variegated and complicated than that proposed by a paradigmatic view • misguided attribution of two distinct and often opposite sets of qualities to the two large families of methods

14. Arguments to support rejection of the idea of paradigms and moving towards the ideaof continuum • misrepresentation of complex philosophical ideas • major differences in philosophical as well as methodological preferences within the camp of qualitative researchers as well as within the ranks of quantitative researchers • it is rather a complex and multifaceted individual mental model, which is formed by many factors like education and training, personal values, disciplinary perspectives, philosophy of science, etc., than formalized paradigm that guides the work of social inquirers and evaluators

15. On the way towards the idea of continuum • Extension of the paradigm model to 5 paradigms • by Guba and Lincoln (2001/2005/2008) • The idea of mixed methods as an interactive continuum by Ridenour & Newman (2008) • QUAL-MM-QUAN continuum • Multidimensional continuum of research projects • by Teddlie & Tashakkori (2009)

16. On the way towards the idea of continuum • Integrated Multidimensional Continuum of Research Methodology (adapted from Niglas 2001) • not static or “all-inclusive” • functions as an illustration supporting an argument against validity of paradigmatic approach to methodology • helps to organize our thinking about methodology and thereby serves educative purposes

17. Original figure available at: www.tlu.ee/~katrin/mmrm/

19. Experiences from the field ... “[...] experience in conducting research to bring about social change and influence social policy indicates that the most persuasive policy research includes both elements: numbers that define the scope and patterns of the problem, and a story that shows how the problem works in daily life and provides for emphatetic understanding. These two elements stem from quantitative and qualitative research. Integrated approaches are effective for [...] analysis of programs, policies, or actions that helped to overcome persistent inequalities.” Spalter-Roth, 2000

20. The choice of methodology and methods for a study depends on the purposes and research questions

21. Best fit between aims/problems and designs/strategies(adapted from Creswell & Plano Clark 2007)

22. Facilitating change as sovereign aim for evaluation - practice and change oriented research approaches • Actionresearch • Design research Cyclical nature of the research process: • Diagnose / Analyze • Plan action / design & develop • Act / test / implement • Evaluate / Learn Need for variegated empirical data –> best results will be often achieved by utilizing mixed methods approach • Monitor

23. Traditional model of methodological aspects is here incorporated into more general framework! • Research problem(question, hypothesis, purpose, …) • Strategy (case study, survey, experiment, grounded theory, action research …) • Sampling(random sample, one case, purposefully chosen cases, … ) • Data collection(structured questionnaire, unstructured/open interview, …) • Data analysis(statistical methods, open coding, discourse analysis,… ) • Interpretation and conclusions (descriptions, empirical generalisations, …)

24. What is design research? • Design research investigates the process of designing with the overall aim to develop an accessible, robust body of knowledge that enhances our understanding of design processes, applications, methods and contexts. Often, this knowledge helps to define best practice and workable methods in dealing with design and design related problems. • Alternative approaches/terms used: design-based research, design-science research, design study, design experiment, developmental research, development study, formative research, ...

25. What is design research? • Design research is distinguished from design by the production of interesting (to a community) new knowledge • Research driven • Systematic documentation • Formative evaluation • Generalization • DR strives to combine the creativity of design communities with appropriate adherence to standards of traditional research

26. Characteristics of good design research • Development and research takes place through continuous cycles of design, enactment, analysis, and redesign • The central goals of designing artifacts or environments and developing theories about design process but also about the related phenomena are intertwined • DR must lead to sharable theories that help communicate relevant implications to practitioners and other designers • Research must account for how design functions in authentic settings • The development relies on methods that can document and connect processes of enactment to outcomes of interest

27. Design research – the methodological cycle Indicating the need for development and related knowledge -> -> initial plan of the study • Analysis of the problem(needs, goals, pre-existing knowledge, ... ) • Designing / monitoring / documenting – planning of the design&development process (work allocation, schedule, applicable methods, ... )– monitoring the d&d process(memos, team discussions, …) – describing the result of d&d process (sketches, alternatives, resulting design and/or artifact, …) • Evaluation (incl initial implementation)(testing the design result, evaluation according to standards, feedback from users and/or experts, ... ) • Generalisations(suggestions, requirements,standards, concepts, models, ontologies, theories,...)

28. Our design research project:Induction year for novice teachers in Estonia 4 Evaluating the model and theoretical standpoints 5. Shortcomings of the model and Generalizations 3. Applying the model (being reflective) Novice teacher 1. Analysing problem, collecting and syntezising theoretical information 0. Need to design a support system 2.Design process, constructing/ improving the model

29. Induction year for novice teachers in EstoniaPartners and implementation activities: • Schools (mentors, novie teachers, school leaders) creating propitious environment for professional development mentoring beginning teachers • Universities (re-)developing theoretical model for mentoring mentor training support programme for beginning teachers • Ministry developing policy ensuring resources • Joint expert group (re-)designing implementation model facilitating networking quality assurance

30. Induction year for novice teachers in EstoniaDesign research cycles Design cycles of our project can be divided into two general phases: • Cycles I and II - emphasis on planning, designing and re-designing implementation model for induction programme Analysis of theoretical and secondary materials Focus group interviews with experts, novice, mentors and school leadres Survey for beginning teachers Semi-structured questionnaire for mentors Open interviews for the school leaders Monitoring and self-evaluation (diaries, peer visits, peer groups, ...) • Cycles III, IV, etc – emphasis on implementation of the induction programme - Constant evaluation and improvement of the model => monitoring and research activities continued

31. Experiences from our design research project Well established cyclical process of design research where QUAL and QUAN data is systematically collected and analysed in order to inform design & development process, empovered by strong partnership links provide good basis for evidence based educational restructuring ensuring • Continuity of the process - ongoing evaluation and development - ongoing partnership • Sustainability of the project - acceptance by the practitioners - wide-scale implementation

32. Towards Fifth Generation Evaluation …some conclusions (read: visions) The 5th Generation Evaluation can be characterised by: Conceptualising evaluation as a tool for facilitating development and therefore utilizing cyclical methodological designs which place evaluation into the process of continuous pursuit for improvement (evaluation not as a purpose of the project per se but implemented within a project to serve wider purposes) Valuing initiation from stakeholders as well as from experts to start and lead systematic process of evidence-based development (utilizing designs from action research to design research with various hybrids in between)

33. Towards Fifth Generation Evaluation …some conclusions (read: visions) Open and inclusive stance towards various worldview positions and methodological approaches where paradigmatic confrontation is commuted for continuum model accommodating overlaps between and being enriched by the differences there are between traditions (dignifying personal mental model instead of paradigm, following the principle of informed creativity, choosing methods and techniques in concord with specific aims and questions considering QUAN, QUAL as well as MM approaches) Acceptance that evaluation can generate results that apply to the unique local settings as well as results that servemore general shared needs for development (accepting and utilizing different forms of generalizations)

34. Thank you ! Katrin NiglasPhD,ProfessorInstitute of Informaticskatrin@tlu.ee; www.tlu.ee/~katrin/