Scientifically Based Research: A Guide for Educational Publishers and Developers Practical Issues

1. Scientifically Based Research:A Guide for Educational Publishers and DevelopersPractical Issues Interactive Educational Systems Design, Inc., 2003

2. Overview • Cost considerations • How long will it take? • Securing school participation in research projects • Developing research white papers and portfolios Interactive Educational Systems Design, Inc., 2003

3. Cost Considerations Rule of thumb: • About 10% of a development project’s budget should be set aside for studying impact. Interactive Educational Systems Design, Inc., 2003

4. Cost Considerations • There are no general rules of thumb about the relative cost of different research designs. Interactive Educational Systems Design, Inc., 2003

5. Cost Considerations Typical labor costs: • Senior researcher time • Project manager time • Site coordinator time • Data collector time • Data analyst time Interactive Educational Systems Design, Inc., 2003

6. Cost Considerations Costly labor intensive tasks: • Direct observations • Scoring of performances • Sometimes, data analyses Interactive Educational Systems Design, Inc., 2003

7. Cost Considerations Observation costs: • Traveling to observation sites • Identifying or developing observation protocols • Developing training materials • Training observers • Re-checking observer reliability Interactive Educational Systems Design, Inc., 2003

8. Cost Considerations Ways to cut observation costs: • Use local observers • Use graduate student observers • Use publicly available observation protocols and training materials Interactive Educational Systems Design, Inc., 2003

9. Cost Considerations Travel costs related to: • Meetings between researchers and the publisher or developer • Meetings between senior researcher(s) and leaders of participating institutions • On-site data collection Interactive Educational Systems Design, Inc., 2003

10. Cost Considerations Performance scoring costs: • Development of scoring rubrics and performance prompts • Training scorers • Conducting reliability checks Interactive Educational Systems Design, Inc., 2003

11. How Long Will It Take? • Conduct literature review 2-3 months • Design research plan 1 month • Design and test new instruments (if any) 2-3+ months • Pilot test research design, analyze data, and revise (time permitting) 6-8 months • Implement study 4-10 months • Data analysis 1-2 months • Report preparation 1-2 months Interactive Educational Systems Design, Inc., 2003

12. How Long Will It Take? Saving time on the literature review: • Search for a meta-analysis of experimental studies on a topic. • Search for an existing literature review. • Locate overview publications published by nationally recognized professional organizations. • Conduct electronic searches. • Consult a research librarian. Interactive Educational Systems Design, Inc., 2003

13. How Long Will It Take? Study duration depends on: • The nature of the product • The scope of the product or of the “chunks” of the product to be evaluated • How much new data collection is necessary • The nature of the data analysis Interactive Educational Systems Design, Inc., 2003

14. Securing School Participation in Research Projects • Focus on the key district decision-maker Interactive Educational Systems Design, Inc., 2003

15. Securing School Participation in Research Projects Before consenting to participation, most district decision-makers will want to know: • The benefits to participation • The costs • That no harm will come to participants Interactive Educational Systems Design, Inc., 2003

16. Securing School Participation in Research Projects It may be necessary to: • Obtain informed consent from students, parents, and/or teachers • Receive approval from the District’s Internal Review Board for the Protection of Human Subjects Interactive Educational Systems Design, Inc., 2003

17. Developing Research White Papers and Portfolios Why Publishers Need a White Paper or Portfolio • District administrators need to document that their instructional materials purchase decisions are based on research. • Publishers need to make the best case that their product designs are supported by SBR. Interactive Educational Systems Design, Inc., 2003

18. Developing Research White Papers and Portfolios Research White Paper Hybrid research and marketing document • Executive summary of a product’s research base • Description of related product features • Direct research on the product’s effectiveness (if available) Interactive Educational Systems Design, Inc., 2003

19. Developing Research White Papers and Portfolios Research Portfolio Detailed abstracts of the research cited in the white paper, including: • Supporting research • Direct research of the product’s effectiveness Interactive Educational Systems Design, Inc., 2003

20. Developing Research White Papers and Portfolios Researching the Product Design Two alternative scenarios: • Products currently being designed • Existing products Interactive Educational Systems Design, Inc., 2003

21. Developing Research White Papers and Portfolios Researching the Product Design For products currently being designed: • Build a review of the research literature into the design process. • Update the review just prior to preparing the white paper or portfolio. Interactive Educational Systems Design, Inc., 2003

22. Developing Research White Papers and Portfolios Researching the Product Design For existing products: • Draw on the history of the product design. • If there is no historical record, conduct a “reverse analysis” of the product design. Interactive Educational Systems Design, Inc., 2003

23. Developing Research White Papers and Portfolios Researching the Product Design When completing a literature review, distinguish between: • Theories or expert opinion • Scientifically based research Interactive Educational Systems Design, Inc., 2003

24. Developing Research White Papers and Portfolios What to Cover in the White Paper • Direct SBR evidence of product effectiveness • Content coverage Interactive Educational Systems Design, Inc., 2003

25. Developing Research White Papers and Portfolios What to Cover in the White Paper • Instructional design • Reference list Interactive Educational Systems Design, Inc., 2003

26. Developing Research White Papers and Portfolios Expanding to Include a Research Portfolio • More detailed abstracts of each study featured in the white paper • Can be distributed in print and online • Will help schools and districts make the case that their purchase decisions include consideration of SBR Interactive Educational Systems Design, Inc., 2003

27. Developing Research White Papers and Portfolios Expanding to Include a Research Portfolio Each abstract should include: • Description of the student population • Number of students, classes, and schools • Explanation of how the sample was selected Interactive Educational Systems Design, Inc., 2003

28. Developing Research White Papers and Portfolios Expanding to Include a Research Portfolio Each abstract should include: • Description of the experimental treatment • Description of the comparison treatment • Description of the assessment instrument(s) Interactive Educational Systems Design, Inc., 2003

29. Developing Research White Papers and Portfolios Expanding to Include a Research Portfolio Each abstract should include: • Statistical analyses performed • Summary of results of analyses • Author’s conclusions • Limitations of the study • Links to the target product Interactive Educational Systems Design, Inc., 2003

30. Developing Research White Papers and Portfolios Size of Documents • Research White Paper: • Supplemental product: 5-10 pages • Core curriculum product: 10-20 pages • Research Portfolio: 1-2 pages per abstract Interactive Educational Systems Design, Inc., 2003

31. Developing Research White Papers and Portfolios Who Should Help Prepare the White Paper? • In-house research expert or outside research consultancy • Product design staff • Marketing/communication staff Interactive Educational Systems Design, Inc., 2003

32. Developing Research White Papers and Portfolios Cost Factors Cost of three major phases: • Research review • White paper development • Portfolio abstract development Interactive Educational Systems Design, Inc., 2003

33. Developing Research White Papers and Portfolios Cost Factors Research review tasks: • Search literature • Develop criteria for including research in the white paper • Complete preliminary review of research documents • Write internal abstracts (not for publication) Interactive Educational Systems Design, Inc., 2003

34. Developing Research White Papers and Portfolios Cost Factors Research review tasks: • Review product information (if the writer is an external contractor) • Prepare outline of white paper • Make final decisions about white paper development Interactive Educational Systems Design, Inc., 2003

35. Developing Research White Papers and Portfolios Cost Factors White paper development tasks: • Write summaries of individual research studies • Write explanations of how product content, features, and functions relate to the research • Write the introduction and conclusion • Prepare the reference list • Edit the paper Interactive Educational Systems Design, Inc., 2003

36. Developing Research White Papers and Portfolios Cost Factors Portfolio of abstracts development tasks: • Develop template of information categories (data fields) • Write abstracts for publication • Edit the abstracts Interactive Educational Systems Design, Inc., 2003

37. Developing Research White Papers and Portfolios How Long Will It Take? • Research review 1-3 months • White paper development 2-4 weeks • Portfolio of abstracts development 1-3 weeks Interactive Educational Systems Design, Inc., 2003