Research Agenda Workshop

1. Research Agenda Workshop [District/State Name] Session 1: [Date and time] Session 2: [Date and time] [Facilitator Name]

2. Welcome and Purpose Our purpose in this workshop series is to • Engage in a collaborative process to identify district/state priorities, and • Develop a set of research questions that comprise a coherent research agenda that will be the work of our district/state for next 3-5 years.

3. Goals for the Session • Session 1 (today) will focus on reviewing different types of research, including levels of evidence, making causal claims, and linking questions to methods. We will leave today having prioritized initial research topics. • Next session, we will generate research questions, and refine and prioritize our identified topics and research questions. We will leave with an initial research agenda for our district/state.

4. Agenda for the Session • Introduce workshop participants • Investigate types of research and levels of evidence, including reviewing the pre-work and additional research examples • Discuss examples of researchable questions and research agendas • Review district/state goal and generate some possible topics • Prioritize possible research topics • Closing

5. Introductions • Please share your: • Name • Agency/where you work • Your role there • [if there are pictures of the participants you can insert an additional slide with the name and pictures while introductions are taking place].

6. Primary Types of Research • Different types of research allow us to answer different types of questions and to make different types of claims. • What’s Happening • Descriptive • What’s Known • Descriptive • Making Connections • Correlational • Making an Impact • Impact/Causal

7. What’s Happening • Descriptive studies do not examine cause and effect but look at what is happening: trends; baselines; and experiences of individuals, groups, or programs • What is the trend in science NECAP scores over past 5 years? • What are the certification characteristics of beginning teachers in the region? • What are the features of district teacher evaluation systems? • Methods tend to include secondary data analysis, document and records review • Tend to use descriptive statistics: averages, frequencies, and percentages

8. What’s Known • Another type of descriptive study focusing on reviews of the literature and other research that has already been conducted. • Literature review • Meta-analysis

9. Making Connections • Correlational studies look at the relationship between two or more variables but one cannot infer causality. • How do math course taking patterns vary race and gender? • What factors are related to differences in graduation rates across districts? • Methods tend to be secondary data analysis of administrative or other state, district, or school datasets • Statistical methods include: differences in means, ANOVA, multivariate regression models

10. Making an Impact • Causal/Impact studies usually examine questions about impact, building on descriptive studies • Does this afterschool program increase student engagement? • Does this math intervention lead to improved math achievement? • Do new teacher evaluation policies improve student achievement outcomes? • Many “opportunistic” possibilities in implementation • According to IES, an impact study: • examines the effectiveness of a particular policy, program, or practice • can be small-scale or large-scale • can be used in a formative way (e.g. a short-term pilot study) or a summative way

11. Making an Impact • Only certain types of designs can be used to assert causal claims or claims about program impact • IES-funded impact studies must meet What Works Clearinghouse standards • Randomized Control Trials (RCTs) = “the gold standard” • Randomization of treatment and control groups (e.g. those who get the intervention and those who don’t) • Other designs, including quasi-experimental, matched comparisons or others may not yield unbiased estimates of impact

12. Investigating Research Examples • Refer to the 2 research summaries you read during pre-work • You will be placed into breakout rooms where small groups will focus on one study and discuss • Research questions • Data sources • What was answered/found • Study design—discuss and add this information • What else there is to learn/ future research questions

13. Discussion: Research Design • Back in the large group we will share out from our small group discussions • What were the studies about? • What is and is not answered by research questions/designs? • What related research questions might you ask?

14. Researchable Questions • What is a researchable question and is not? • Reasonable • Appropriate • Answerable • Measurable • How do you develop researchable questions? • Questions, concerns, and values of stakeholders/community • Important issues raised in the field or literature • Professional standards or guidelines • Views and knowledge of experts • One’s own views and judgement

15. Research Agendas • What is a research agenda? • What does a research agenda look like? • Linear • Topical • Examples

16. Example: Research agenda that is less coherent • TOPIC: COLLEGE READINESS • Descriptive study of high school dropout rates for key subgroups • Descriptive study of policies for assigning students to college math vs. general math • RCT of the impact of classroom coaching for new high school English teachers in 5 large urban districts (Source: REL Kickoff Meeting)

17. Example: Coherent, linear research agenda • TOPIC: HIGH SCHOOL GRADUATION • Action Goal: Increasing the percentage of District X students who graduate from high school • Descriptive study of high school dropout rates for key subgroups • Descriptive study of re-enrollment rates in traditional high schools and education trajectories of re-enrollees • RCT of the impact of alternative programs for degree completion on high school graduation (Source: REL Kickoff Meeting)

18. Example: Coherent, topical research agenda • TOPIC: MATHEMATICS LEARNING • Action Goal: Increasing mathematics achievement in the Northeast region • RCT of the impact of student access to Algebra Iin grade 8 • Descriptive study of math education practices for students with disabilities in six schools in two Northeast states • Descriptive studies of math performance patterns for students with disabilities in MA and NY

19. Our Process to Identify Topics • Work from district/state goal to identify topics and narrow focus to two topics • Identify subtopics and questions within top two topics • Prioritize questions [or subtopics]

20. Example: Gender in Schools Research district/state Goal: Provide research that informs and promotes gender equity in schools

21. Moving from Subtopics to Questions

22. Moving from Subtopics to Questions A fully specified research question A general question that can be used to develop a set of research questions

23. District/State Goals

24. Goals & Research Topics • Reflecting on our goals, brainstorm topics that are important to you and that fit under the goals. • Record in your workbook on page 5. • If you are unsure whether a topic fits under the goal, include it in the “possible” column of the page. • The ideas don’t need to be refined, but we will be narrowing the list later in the session, so focus on topics that are most important to you. • You will have 5 minutes to think and write by yourself.

25. Research Topics-Breakout Rooms • You will be placed into a virtual breakout room to discuss the topics you brainstormed (15 minutes) • Share your topics with your small group • On your breakout room whiteboard: • Write the topics that you would like to bring back to the whole group for consideration as priority topics • These topics will be brought back to the whole group, when the breakout time ends.

26. Grouping our topics • Together, we will engage in a process of grouping the topics. • Our identified topics are diverse but some are related. • What topics generated in the groups do you think go together? • After we review the grouping of topics we will name the topics. • The name becomes the “topic” and the items grouped under that topic name are “subtopics”

27. Narrowing the List of Topics • The Goal is to narrow the list of topics to 2 priority topics for our work—we will do this using a modified Focusing Four process • Brainstorm: We did earlier and grouped our ideas into topics • Clarify: We will review the groupings. • We can clarify what any topic is, suggest moving topics or splitting groupings. • Advocate: Think about narrowing to 2 topics that could comprise a good agenda when taken together. • Canvass: You will each have 3 votes to use to indicate the topics that you think should be a research priority for our district/state.

28. Clarify • Review the “groupings” of topics • Do any topics on any list need to be clarified? • Do the groupings make sense? Should we move any items or split any groupings? • Can anything be combined or removed?

29. Advocate • Think about which cluster of topics, when taken together, provide the best research direction for the district/state. • Share your preferred priority topic and provide a few positive reasons why you think the topic grouping should be a priority for the district/state.

30. Break 5–10 minute break

31. Polls • Vote once in each Poll to indicate your priority research topic for the district/state. • You can vote for the same topic in each poll if you like, or vote for different topics. • At the conclusion of the canvassing, we will review the results of our Polling and make some decisions about our 2 topics. • This will be your take-away from Workshop session 1.

32. Prepare for Next Session • For one or both of the topics, think about: • What research questions interest you under the priority topic areas? (these do not have to be at the level of researchable questions but they should be questions that are a finer grain of detail than the topics themselves). Email questions to facilitators before Session 2. • Begin to think about the data sources that are appropriate to answering these questions and where they reside, if they exist.

33. Thank you • Thank you for your participation and engagement today. • See you [INSERT DATE AND TIME OF NEXT SESSION]!

34. Research Agenda Workshop Session 2 [District/State Name] Session 1: [Date and time] Session 2: [Date and time] [Facilitator Name]

35. Welcome and Purpose Our purpose in this workshop series is to • Engage in a collaborative process to identify district/state priorities, and • Develop a set of research questions that comprise a coherent research agenda that will be the work of our district/state for next 3-5 years.

36. Goals for the Session • Session 1 focused on reviewing different types of research, including levels of evidence, making causal claims, and linking questions to methods. We left with some prioritized topics. • This session, we will: • Generate, refine and prioritize research questions, and • Leave with an initial coherent research agenda

37. Agenda for the Session • Review topics from session 1 • Generate questions (review questions sent in by email and generate additional questions) • Prioritize questions • We will prioritize all the questions generated in our two topic areas. • Generate preliminary research agenda • Next steps

38. Review Research Topics and Begin to Generate Questions • Highlights of topics from last session and review of research questions sent in by email. • In virtual breakout rooms, work on the white boards for 30–40 minutes to generate questions related to the priority topic (and any subtopics within it) • Groups should think comprehensively within their topic/subtopic

39. Share Out Small Group Work/Refine Questions • Re-group into whole group and share white boards from small group work. • Discuss questions generated and refine by: • Edit, add, and/or delete questions • Check: any additional questions—is something missing?

40. Prioritize Research Questions • Focusing Four process • Brainstorm: Review our list of questions, generated earlier • Clarify: Do any questions on the list need to be clarified? • Advocate: Think about narrowing a few questions in each topic/subtopic area that could comprise a good agenda • Last step will be done after the break

41. Break 5–10 minute break

42. Poll • Poll: Vote for the research questions. • Vote once in each Poll • You can vote for the same question in each Poll or different questions

43. Preliminary Research Agenda • Review identified priority topic, subtopics, and questions • How many top rated questions should be on the agenda? Consider: • Types of work (data alignment, workshops, studies, etc.) • Sequence • Short- and long-term nature of work • Remember…

44. Research Agenda- continued • Refinement of the research agenda should continue over time. • The agenda should: • encompass the work that is most important to the district/state, • Align with the district/state goal and • Be the work that the district/state will do over the next 3-5 years.

45. Next Steps • Continued refinement of agenda • Involvement in prioritization and study development • Work to conduct research or identify research support • Use agenda to recruit research and funding partners for this work.

46. Thank you Thank you for your participation and engagement today.