Research Agenda Workshop

1. Research Agenda Workshop [Insert relevant information. Consider: • Alliance or group name • Facilitator’s name(s) • Date • Location]

2. Welcome and purpose Workshop objectives: • Engage in a collaborative process • Identify alliance research priorities • Develop a set of research questions • Develop a coherent research agenda for the next 3–5 years

3. Session goals • Morning: • Review different types of research • Identify and prioritize research topics • Generate related research questions • Afternoon: • Refine and prioritize research questions to form initial research agenda

4. Agenda • Introduce participants • Identify possible priority research topics • Investigate types of research and levels of evidence • Explore research questions and research agendas • Prioritize research topics • Generate research questions • Share, refine, and prioritize research questions • Put it all together into a research agenda

5. Introductions • Name • Where you work • Your role

6. Alliance goal [Insert your alliance or group’s goal. For example: The USVI College and Career Readiness Research Alliance will support the Virgin Islands Department of Education in their efforts to prevent and reduce the number of students dropping out of schools by providing applied research and analytic technical support on how best to use available data to both establish robust early warning systems and identify interventions to help improve outcomes for students at risk.]

7. Alliance goal and research topics • Record in your workbook on page 5: • Topics that fit under your alliance’s goal • If you are unsure of a topic, include it in the “possible” column • Focus on topics that are most important to you. • You will have five minutes to think and write by yourself.

8. Small-group discussions • Break up into small groups • On large Post-It notes, write down the topics your group would like considered as priority topics • One topic per Post-It note

9. Grouping our topics • Affinity grouping process. • What topics generated in the groups go together? • Review the topic groupings and name them. • Items grouped under each topic name are now considered subtopics.

10. Types of research Typology used by the REL program at IES includes: • What’s Happening • Descriptive • What’s Known • Descriptive • Making Connections • Correlational • Making an Impact • Impact/causal

11. What’s Happening • Descriptive studies look at what is happening: trends; baselines; and experiences of individuals, groups, or programs • Methods include secondary data analysis and document and records review • The studies use descriptive statistics: averages, frequencies, and percentages

12. What’s Known • Another type of descriptive study focuses on reviews of the literature and other previous research. Examples include: • Literature reviews • Meta-analyses • What Works Clearinghouse reviews • The What Works Clearinghouse is an IES-funded initiative that provides reviews of existing education research and evaluates the credibility and reliability of the research evidence.

13. Making Connections • Correlational studies look at the relationships among two or more variables or characteristics but do not imply that one causes another. • Correlational research tests whether relationships between variables are “statistically significant,” meaning they are not likely due to chance • Methods include analysis of existing data from administrative or other state, district, or school datasets.

14. Making an Impact • Causal/impact studies usually examine questions about impact; they can build on descriptive studies • Look for “opportunistic” possibilities • An impact study: • Examines the effectiveness of a particular policy, program, or practice • Can be small or large • Can be used in a formative way (for example, a short-term pilot study) or a summative way

15. Making an Impact • Only certain types of designs can assert causal claims or claims about program impact • REL impact studies must meet What Works Clearinghouse standards • Randomized controlled trials (RCTs) = the “gold standard” • Other designs, including quasi-experimental and matched comparisons, may not yield unbiased estimates of impact

16. Investigating research examples • Review the research summary you read during prework (5 minutes) • Divide into small groups by summary and discuss (15 minutes) • What the research questions were • What data sources were used • What the research design was • What else there is to learn • What you would need to answer the remaining questions (data, RCT, etc.)? • Whole-group discussion (10 minutes)

17. Break • 15 minutes

18. Researchable questions • What is a researchable question? • Reasonable • Appropriate • Answerable • Specific • Where do researchable questions come from? • Questions, concerns, and values of stakeholders • Important issues in the field or research literature • Professional standards or guidelines • Views and knowledge of experts • One’s own views and judgment

19. Research agendas • What is a research agenda? • Identifies research priorities and questions for group • Is coherent • Leads to rigorous and relevant research that is actionable • May include current or future project • What does a research agenda look like? • Linear • Topical

20. Example: Research agenda that Is less coherent • Topic: College readiness • Research questions: • What are the high school dropout rates for particular subgroups of students? • Does coaching high school English teachers impact their students’ performance? • Studies: • Descriptive study of high school dropout rates for key subgroups • Descriptive study of policies for assigning students to college math versus general math • RCT of the impact of classroom coaching for new high school English teachers in five large urban districts

21. Example: Coherent, linear research agenda • Topic: High school graduation • Research questions: • What are the high school dropout rates for key subgroups of students? • What are the effects of alternative high school programs on students’ degree completion and graduation? • Studies: • Descriptive study of high school dropout rates for key subgroups • Descriptive study of re-enrollment rates in traditional high schools and education trajectories of re-enrollees • RCT of the impact of alternative programs for degree completion on high school graduation

22. Example: Coherent, topical research agenda • Topic: Mathematics learning • Research questions: • What is the impact of grade 8 students’ access to algebra I on their math achievement? • How do students with disabilities perform in math? • Studies: • RCT of the impact of student access to algebra Iin grade 8 • Descriptive study of math education practices for students with disabilities • Descriptive study of math performance patterns for students with disabilities

23. Prioritizing research topics Narrow the list of research topics to 24 priority topics for our alliance, using a modified Focus Four: • Brainstorm • Clarify • Advocate • Canvass

24. Example: Gender in Schools Research Alliance • Alliance goal: Provide research that informs and promotes gender equity in schools 24

25. Moving from subtopics to questions

26. Moving from subtopics to questions A fully specified research question A general question that can be used to develop a set of more specific research questions

27. Generating research questions • In pairs, work for 30 minutes to generate research questions related to identified priority research topic(s). • Think about: • What are some possible research questions for these topics and subtopics? • What data would be needed to research these questions?

28. Lunch break • 45 minutes

29. Sharing questions • Post newsprint with questions from each group • Review the posted newsprint and consider: • What would you change or edit? • What would you delete? • What would you add? • Briefly discuss with a partner

30. Refining questions • Whole group (small groups if whole group is large): • Edit, add, delete, and combine questions as necessary • What kinds of studies would result from the questions (descriptive, correlational, impact evaluation)? • In pairs, discuss: • What is important to you? • What research topic and questions can provide a meaningful research agenda?

31. Prioritizing research questions • Prioritize the research questions in each priority topic area • Focus Four process • Brainstorm • Clarify • Advocate • Canvass

32. Preliminary research agenda • Review identified priority topics and questions • Any surprises? • How many top-rated questions should be on the agenda? Consider: • Alliance goal • Types of research • Sequence • Short- and long-term nature of questions

33. Research agenda template • Turn to research agenda template on page 19 • Use template to organize research agenda • Document and summarize the main research topics and subtopics identified, along with research questions

34. Next steps • Summary document from workshop • Iterative review and refinement of agenda • Agenda as a “living document” • Collaborative work to: • Identify appropriate existing data or data collection for studies • Identify existing research relevant to agenda priorities and questions • Develop study designs • Conduct research or identify research support

35. Thank you • For more information contact: • [insert contact information or customize with any special closing information]