Promoting a Culture of Research in Aid of Policy Formulation

1. Diosdado M. San Antonio DepED Region 4A (CALABARZON) Promoting a Culture of Research in Aid of Policy Formulation

2. Clarifying key concepts The need for evidence-informed policy Reasons why education research is not used extensively in policy formulation Linking educational research to policy Ways research can support educational practices Research synthesis The policy cycle Social impact analysis Talk Outline

3. Research

4. Three Realms of Research in Education

5. Two Categories of Research Workers

6. Education vs Educational Research? • Educational research - studies for education • consciously geared towards improving policy and practice • Education research - studies of education • additional substantive value independent of its policy-relevance Clark, C. (2011). Education(al) Research, Educational Policy-Making and Practice, Journal of Philosophy of Education, Vol. 45, No. 1, 2011, pp. 37-57

7. an approach which ‘helps people make well informed decisions about policies, programs and projects by putting the best available evidence at the heart of policy development and implementation’ (Davies 1999 in Gough et al, 2011). Evidence informed policy…. Gough D, Tripney J, Kenny C, Buk-Berge E (2011) Evidence Informed Policy in Education in Europe: EIPEE final project report. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London.

8. McColskey & Lewis (ND) Making Informed Decisions About Programs, Policies, Practices, Strategies, & Interventions

9. Evidence-Based Decision Making CycleMcColskey & Lewis (ND) Making Informed Decisions About Programs,Policies, Practices, Strategies, & Interventions

10. a greater concern with student achievement outcomes; a related explosion of available evidence due to a greater emphasis on testing and assessment; more explicit and vocal dissatisfaction with education systems, nationally and locally; increased access to information via the Internet and other technologies; and resulting changes in policy decision-making. Factors driving need for evidence-informed policy Tracey Burns and Tom Schuller, OECD Centre for Educational Research and Innovation, http://www.oecd.org/edu/ceri/47435459.pdf

11. Economic imperative to justify public spending • Moralimperative to ensure those providing services do so informed by the best possible evidence (e.g. Oakley, 2000); • Academic imperative – The importance of maximizing research use Maximising research use in policy and practice in education Judy Sebba Professor of Fostering and Education University of Oxford Department of Education judy.sebba@education.ox.ac.uk

12. Policy makers rank academic research well below special advisers (media background), experts and think tanks as sources of evidence (Campbell et al 2007; Rich 2004; Rigby 2005); • Policy makers often regard research findings as impenetrable, ambiguous, conflicting, insignificant, untimely or only partially relevant. In turn, they display confusion about what constitutes evidence and its role (Brown, 2012; Rickinson, Sebba & Edwards 2011). • Confusion about evidence is rife among the public What is the problem?: The lack of evidence-informed policy and practice Maximising research use in policy and practice in education Judy Sebba Professor of Fostering and Education University of Oxford Department of Education judy.sebba@education.ox.ac.uk

13. Numbers to be influenced by evidence? More than half a million teachers in the Philippines; Practitioners are too busy, cannot locate relevant andaccessible evidence, lack confidence to ‘judge’ research; ‘Expert systems such as EBP [evidence-based practice] are attempts to manufacture trust as a legitimating exercise for the mandate of professional authority in social work’ (Webb, 2002) What counts as evidence, the nature of evidence & how it is used in decision-making is highly contested. What stops evidence being used? Maximising research use in policy and practice in education Judy Sebba Professor of Fostering and Education University of Oxford Department of Education judy.sebba@education.ox.ac.uk

14. Lack of rigor Failure to produce cumulative research findings Theoretical incoherence Ideological bias Irrelevance to schools Lack of involvement of teachers Inaccessibility and poor dissemination Poor cost effectiveness Common Issues against Education(al) Research Whitty, G. (2006). Education(al) research and education policy making: is conflict inevitable?, British Educational Research Journal Vol. 32, No. 2, April 2006, pp. 159–176

15. Research to Policy and Practice:Some Issues • On the processes and mechanisms through which research-based knowledge may be transferred into policy and practice • On the question of appropriate relationships between research, policy and practice. Ozga, J. (2004). From Research to Policy and Practice: Some Issues in Knowledge Transfer, accessed 12 April 2015 from http://www.ces.ed.ac.uk/PDF%20Files/Brief031.pdf

16. On transferring research-based knowledge into policy and practice Ozga, J. (2004). From Research to Policy and Practice: Some Issues in Knowledge Transfer, accessed 12 April 2015 from http://www.ces.ed.ac.uk/PDF%20Files/Brief031.pdf

17. On relationships between research, policy and practice Ozga, J. (2004). From Research to Policy and Practice: Some Issues in Knowledge Transfer, accessed 12 April 2015 from http://www.ces.ed.ac.uk/PDF%20Files/Brief031.pdf

18. Linking Educational Research to Policy Research does not logically or psychologically provide a basis or starting point for policy. Re-frame political and educational expectations of the research/policy relationship in favor of more realistic and sophisticated models of how policy is developed. Values, normativity and ideology are legitimately central to policy making. There is a role for research in refining, critiquing, and developing these elements within a structure of intelligent argumentation. Policy can and should be informed by the full range of intellectual resources available in the research community and not just a narrowly empiricist selection. Influence development of ways in which the high quality work of scholarship can inform policy – and work to remove current restrictions on what is admitted. There is no simple algorithm for translating research into policy: ‘Impact’ depends on social practices which bring political, democratic and research voices together in a shared conversation and process of mutual influence. Create conversational communities around central policy issues as a vehicle for mutual information and influence, and not necessarily for decision-making or even agreed understanding. Bridges, D. (2009). ‘Evidence based policy’ What evidence? What basis? Whose policy? Teaching and Learning Research Briefing No. 74, www.tlrp.org

19. Make use of ‘best available evidence’ a requirement in professional standards & build into infrastructure of policy-making; Improve access to synthesized, quality assured evidence in priority areas – open access; Support practitioners to use research (and in some cases to engage in research through closer collaboration of researchers and professionals); Most importantly, interrogate research use and evaluate any initiatives designed to increase impact – only then can we really know what is achieved. Optimizing use of research in crafting educational policies Maximising research use in policy and practice in education Judy Sebba Professor of Fostering and Education University of Oxford Department of Education judy.sebba@education.ox.ac.uk

20. encouraging an active community of educational researchers; promoting cooperation and discussion—with policy makers and practitioners, as well as national and international associations in education and related subject areas; encouraging and supporting debate about the quality, purpose, content and methodologies of educational research; developing and defending an independent research culture committed to open inquiry and the improvement of education; Promoting a Culture of Research Whitty, G. (2006). Education(al) research and education policy making: is conflict inevitable?, British Educational Research Journal Vol. 32, No. 2, April 2006, pp. 159–176

21. Push - incentivize producers (researchers) to undertake relevant, robust research; 2. Pull - incentivize users/practitioners Better articulation of benefits to funders (e.g value-added, prestige); research ‘training’ for policy officials ; role of ‘insider-researchers’in government, two-way secondments; • Networks & brokerage - bring together researchers, users and policy makers - influence on design, research questions, verifying findings, on-going dialogue without losing research integrity. But not all research shows us the way forward e.g. attainment gap (Lavis et al 2003, Levin 2011, Nutley et al 2007, etc) Models of research impact

22. Weaknesses in quality of research in education and reporting of it – ‘descriptive validity’ (Farrington 2003) • Features of high quality research: • clear questions (that address a need) • methods selected that are ‘fit for purpose’ • methods executed properly e.g. reliability • use multiple sources of data (integration of quantitative & qualitative?) • multidisciplinary research needed for complex questions • These are all characteristics assessed through systematic reviewing. Building a higher quality evidence base for the future Maximising research use in policy and practice in education Judy Sebba Professor of Fostering and Education University of Oxford Department of Education judy.sebba@education.ox.ac.uk

23. Randomly controlled trials – Interrogating large databases e.g. on educational outcomes & longer term employment, health etc Longitudinal studies – Mixed methods – to inform us of ‘what’ and ‘how’ Quality assurance, synthesis and scaling up of practitioner inquiry. Improving the future evidence base Maximising research use in policy and practice in education Judy Sebba Professor of Fostering and Education University of Oxford Department of Education judy.sebba@education.ox.ac.uk

24. Research Excellence Framework (REF), UK; Research publications assessed on quality, originality & significance (impact); Impact separately assessed through case studies; Knowledge mobilization work; Research Supporting Practice in Education (OISE) - interrogating research impact. Assessing research and its impact Maximising research use in policy and practice in education Judy Sebba Professor of Fostering and Education University of Oxford Department of Education judy.sebba@education.ox.ac.uk

25. Research use in secondary schools & districts (LAs). Used knowledge claims as basis for intervention – ‘mediated’head teacher study groups, resources on web. Had little impact; • KM in universities - Interviewed18 education faculties in leading research universities worldwide regarding the role of KM - modest in most faculties, done by individual faculty members rather than at institutional level; • Survey of 500 grant-holders to determine extent and nature of their KM efforts - tools and techniques used, mediators, linkage activities, project funding earmarked for KM. Interrogating research use empirically: Research Supporting Practice in Education (RSPE), OISE, UoThttp://www.oise.utoronto.ca/rspe/ Maximising research use in policy and practice in education Judy Sebba Professor of Fostering and Education University of Oxford Department of Education judy.sebba@education.ox.ac.uk

26. Website analysis – developed metric for assessing organizational KM strategies (different types, ease of use, accessibility, focus of audience) >100 education organisations in Canada, UK, US & Australia: national/ local govt depts., universities, funders & ‘knowledge brokers’. Limited evidence of activities that build interpersonal connections that are known to lead to greatest research impact. • Facts in Education: service to counter press reporting, correct significant factual errors about education that appear in various news media across Canada, providing the source & empirical evidence basee.g. class size. • Education Media Centre in England is brokering service between journalists and researchers offering timely evidence & access. Research Supporting Practicein Education continued.. Maximising research use in policy and practice in education Judy Sebba Professor of Fostering and Education University of Oxford Department of Education judy.sebba@education.ox.ac.uk

27. Mediation is undertaken by funders, media, policy analysts, educators, lobby groups, think tanks, policy advisers, etc; • Knowledge brokering links decision makers and researchers, facilitating their interaction …to better understand each other's goals and professional cultures, influence each other's work, forge new partnerships, and promote the use of research… (Canadian Health Services Research Foundation n.d.) • Mediators have multiple positionsas trustees for each others’organizations, sit on each others’ councils, write, speak and ‘appear on platforms’ at each other’s events (Ball & Exley 2010, p.155); • dedicated individual liaison between policy makers and researchers during commissioning/reporting (Martinez and Campbell, 2007); • problem definition,….expansion of public debate, innovation & knowledge brokerage (McNutt and Marchildon 2009); • linking researchers with users throughout the research process increases research impact (e.g. Rickinson et al, 2011; Ward et al, 2009). The role of research mediation in maximizing research use

28. Media presented all the think tanks as credible sources of research, facts, and figures on education, regardless of the extent to which each think tank emphasized policy and political advocacy over the professional norms of academic research e.g. peer-reviewing (Haas 2007) The media and think tanks

29. collective term for the family of methods for summarizing, integrating and, where possible, cumulating the findings of different studies on a topic or research question. Narrative reviews Vote counting reviews Meta-analysis Best evidence synthesis Meta-ethnography Types of Research Synthesis Davies, P. (2000). The relevance of systemic reviews to educational policy and practice. Oxford Review of Education, 26 (3-4), pp. 365-378

30. Attempt to identify what has been written on a subject or topic, using which methodologies, on what samples or populations, and with what findings. There is usually no attempt to seek generalization or cumulative knowledge from what is reviewed. Rather, the task is to identify the range and diversity of the available literature, much of which will be inconclusive, and to find a gap which new research might attempt to fill. Traditional qualitative literature review Narrative reviews Davies, P. (2000). The relevance of systemic reviews to educational policy and practice. Oxford Review of Education, 26 (3-4), pp. 365-378

31. Attempt to accumulate the results of a collection of relevant studies by counting ‘how many results are statistically significant in one direction, how many are neutral (i.e. ‘no effect’), and how many are statistically significant in the other direction’ (Cook et al., 1992, p. 4). The category that has the most counts, or votes, is taken to represent the modal or typical finding, thereby indicating the most effective means of intervention. Vote counting reviews Davies, P. (2000). The relevance of systemic reviews to educational policy and practice. Oxford Review of Education, 26 (3-4), pp. 365-378

32. ‘the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings’. ‘combines the individual study treatment effects into a ‘pooled’ treatment effect for all studies combined, and/or for specific subgroups of studies or patients, and makes statistical inferences’ (Morton, 1999) Meta-analysis Davies, P. (2000). The relevance of systemic reviews to educational policy and practice. Oxford Review of Education, 26 (3-4), pp. 365-378

33. Reviewers apply consistent, well justified, and clearly stated a priori inclusion criteria’ of studies to be reviewed. Uses guiding principles for choosing a priori criteria, including that primary studies should be germane to the issue at hand, should be based on a study design that minimizes bias, and should have external validity. Best evidence synthesis Davies, P. (2000). The relevance of systemic reviews to educational policy and practice. Oxford Review of Education, 26 (3-4), pp. 365-378

34. Attempts to summarize and synthesize the findings of qualitative studies, especially ethnographies and interpretive studies. • ethnographic, interactive, qualitative, naturalistic, hermeneutic, or phenomenological. • seek an explanation for social or cultural events based upon the perspectives and experiences of the people being studied. Meta-ethnography Davies, P. (2000). The relevance of systemic reviews to educational policy and practice. Oxford Review of Education, 26 (3-4), pp. 365-378

36. includes the processes of analyzing, monitoring and managing the intended and unintended social consequences, both positive and negative, of planned interventions (policies, programs, plans, projects) and any social change processes invoked by those interventions. • Its primary purpose is to bring about a more sustainable and equitable biophysical and human environment. Social Impact Assessment (SIA) http://www.socialimpactassessment.com/

37. Analytical tools Community-based methods Consultation methods Observation and interview tools Participatory methods Workshop-based methods Social Impact Assessment tools and methods http://www.unep.ch/etu/publications/EIA_2ed/EIA_E_top13_hd1.PDF

38. Stakeholder Analysis addresses strategic questions, e.g. who are the key stakeholders? what are their interests in the project or policy? what are the power differentials between them? what relative influence do they have on the operation? Gender Analysis focuses on understanding and documenting the differences in gender roles, activities, needs and opportunities in a given context. Secondary Data Review of information from previously conducted work is an inexpensive, easy way to narrow the focus of a social assessment. Analytical tools

39. Participatory Rural Appraisal (PRA) covers a family of participatory approaches and methods, which emphasizes local knowledge and action. It uses to group animation and exercises to facilitate stakeholders to share information and make their own appraisals and plans. • SARAR is an acronym of five attributes -- self-esteem, associative strength, resourcefulness, action planning and responsibility for follow-through -- that are important for achieving a participatory approach to development. • seeks to optimize people's ability to self-organize, take initiatives, and shoulder responsibilities. Community-based methods

40. Beneficiary Assessment (BA) is a systematic investigation of the perceptions of a sample of beneficiaries and other stakeholders to ensure that their concerns are heard and incorporated into project and policy formulation. • The purposes are • (a) undertake systematic listening, which "gives voice" to poor and other hard-to-reach beneficiaries, highlighting constraints to beneficiary participation, and • (b) obtain feedback on interventions. Consultation methods

41. Participant Observation is isbased on looking, listening, asking questions and keeping detailed field notes. Semi-structured Interviews are a low-cost, rapid method for gathering information from individuals or small groups. Focus Group Meetings are brief meetings -- usually one to two hours -- with many potential uses, e.g. to address a particular concern; to build community consensus about implementation plans; to cross-check information with a large number of people; or to obtain reactions to hypothetical or intended actions. Village Meetings allow local people to describe problems and outline their priorities and aspirations. Observation and interview tools

42. Role Playing helps people to be creative, open their perspectives, understand the choices that another person might face, and make choices free from their usual responsibilities Wealth Ranking (also known as well-being ranking or vulnerability analysis) is a visual technique to engage local people in the rapid data collection and analysis of social stratification in a community (regardless of language and literacy barriers). Access to Resources is a tool to collect information and raise awareness of how access to resources varies according to gender, age, marital status, parentage, and so on. Analysis of Tasks clarifies the distribution of domestic and community activities by gender and the degree of role flexibility that is associated with each task. Participatory methods

43. Mapping is useful for collecting baseline data on a number of indicators as part of a beneficiary assessment or rapid appraisals, and can lay the foundation for community ownership of development planning by including different groups. Needs Assessment draws out information about people's needs and requirements in their daily lives. Pocket Charts are investigative tools, which use pictures as stimulus to encourage people to assess and analyze a given situation. Tree Diagrams are multi-purpose, visual tools for narrowing and prioritizing problems, objectives or decisions. Information is organized into a tree-like diagram. Participatory methods

44. Objectives-Oriented Project Planning is a method that encourages participatory planning and analysis throughout the project life cycle. • A series of stakeholder workshops are held to set priorities, and integrate them into planning, implementation and monitoring. Building commitment and capacity is an integral part of this process. • TeamUPwas developed to expand the benefits of objectives-oriented project planning and to make it more accessible for institution-wide use. • PC/TeamUPis a software package, which automates the basic step-by-step methodology and guides stakeholders through research, project design, planning, implementation, and evaluation. Workshop-based methods

45. “The research we do at the local level - collaboratively - is what makes formal, outside research work. Outside research cannot be installed like a car part - it has to be fitted, adjusted, and refined for the school contexts we workd in.” ― Mike Schmoker

46. diosdado.sanantonio@deped.gov.ph Thank you and Mabuhay!!!