A New Methodology for Testing the Effect of Using Word Processors on NAEP Achievement

1. A New Methodologyfor Testing the Effect of Using Word Processors on NAEP Achievement Amos Glenn July 13, 2013

2. Why a new methodology? • 2012, Spring: A complete methodology is prepared by the author to answer research questions using the “restricted use” data the NCES usually makes available to researchers for secondary analyses. • 2012, Fall: The NCES published a report on the results of the 2011 NAEP Writing assessment and the “public use” data is made available though the online Data Explorer tool at the NCES website • 2012, Winter: The usual time when “restricted use” data is made available to researchers passes with little response from NCES to author’s inquiries • 2013, Summer: Through personal connections, the unofficial reply from the NCES is that the 2011 NAEP Writing assessment “restricted use” data will probably never be made available to researchers • 2013, Fall: A new methodology is developed to work within the constraints of “public use” data and Data Explorer tool to answer the research questions

3. Overview of New Methodology • Variables of interest are selected as treatment • Confounding variables are selected for control • Effects of Variables of Interest are Measured • The results of these measurements are compared to draw conclusions concerning the research questions.

4. Overview of New Methodology Variables of Interest Ask students to use computer to complete writing started by hand  details Ask students to use computer to draft and revise writing  details Ask students to use word processing to check spelling  details Use computer for writing for school assignments details Use computer from the beginning to write paper details Use computer to complete paper  details Use computer to make changes to paper  details Used paper and pencil to organize writing for the first writing task  details Used paper and pencil to organize writing for the second writing task  details Used the computer to organize writing for the first writing task  details Used the computer to organize writing for the second writing task  details Availability of computers for writing instruction details • Variables of interest are selected • 12 variables of interest are selected to represent different aspects of the use of word processors in classrooms (treatment) • Variables are selected using the NAEP Data Explorer, the online tool used at the NCES website to collect and analyze NAEP data. • The effects of these variables on students’ average scale scores on the NAEP will be measured to address the research questions

5. Overview of New Methodology • Confounding variables are selected • 5 additional variables likely to confound the measurement of the effects of the treatment (12 variables of interest) are selected from those available through Data Explorer. • Selection is grounded in literature review. • These variables will be controlled for when measuring effects of the variables of interest. Confounding Variables Gender National School Lunch Program eligibility Parental education level Race/ethnicity Student disability or English Language Learner status School location

6. Overview of New Methodology Combinations of Confounding Variables into 15 Subpopulations • Effects of Variables of Interest are Measured • The Data Explorer tool is limited to using only three variables in any analysis. • To overcome this limitation and still control for all 5 confounding variables, the effect of each of the 12 variables of interest in the treatment is measured in 15 subpopulations, with each subpopulation representing one of the 15 possible combinations of the 5 confounding variables (see table on right)

7. Overview of New Methodology • The results of these 180 (12 x 15) measurements are compared to heuristically draw conclusions concerning the research questions.

8. Example of Measuring 1 variable of interest’s (treatment) effect while controlling for 2 confounding variables OVERVIEW OF STEPS • Using the Data Explorer tool, select the criteria of the data to be explored • Select the variables necessary for one measurement • Add a new report and collapse categories • Arrange the control variables into rows and treatment variables into columns • Create a second new report arranging only the control variables to the columns, and select the “percentages” statistic to be reported. • Build and export the two created reports • Calculate the estimate of treatment and test statistic. Record the results in the table of treatment effects table

9. Example of Measuring 1 variable of interest’s (treatment) effect while controlling for 2 confounding variables • Using the Data Explorer tool, select the criteria of the data to be explored • Subject: Writing • Grade: 8 • Measure: Writing Scale • Jurisdiction: National

10. Example of Measuring 1 variable of interest’s (treatment) effect while controlling for 2 confounding variables • Select the variables necessary for one measurement • Gender • Parent’s education level • Use computer from beginning to write paper

11. Example of Measuring 1 variable of interest’s (treatment) effect while controlling for 2 confounding variables • Add a new report and collapse categories • collapse the four categories of “use computer from beginning to write paper” into two new categories: “never-sometimes” and “often-always”)

12. Example of Measuring 1 variable of interest’s (treatment) effect while controlling for 2 confounding variables • Arrange the control variables into rows and treatment variables into columns Arrange the “gender” and “parent education level” variables into rows and the “use computer from beginning to write paper” variable in columns, allowing two mean scores (and standard errors of mean scores) to be generated for every possible combination of genders and parent education levels: the first average score for those in the “never-sometimes” group and the second average score for those in the “often-always” group.

13. Example of Measuring one treatment effect while controlling for two confounding variables • Create a second new report arranging only the control variables to the columns, and select the “percentages” statistic to be reported. This report will contain what percent of the whole group each of the combinations of “gender” and “parent education level” represents. This allows the relative weight of the mean score to be included in the measurement of effect.

14. Example of Measuring 1 variable of interest’s (treatment) effect while controlling for 2 confounding variables • Build and export the two created reports

15. Example of Measuring 1 variable of interest’s (treatment) effect while controlling for 2 confounding variables • Calculate the estimate of treatment and test statistic. Record the results in the table of treatment effects table SUBSTEPS FOR CALCULATIONS Copy data from exported tables Estimate the effect for each subpopulation Estimate the effect for the whole population Estimate the variance of the effect of treatment Calculate the test statistic and p-value

16. Copy data from exported tables • Calculate the estimate of treatment and test statistic. SUBSTEPS FOR CALCULATION

17. Copy data from exported tables • Calculate the estimate of treatment and test statistic. SUBSTEPS FOR CALCULATION

18. Copy data from exported tables • Calculate the estimate of treatment and test statistic. SUBSTEPS FOR CALCULATION

19. Estimate the effect for each subpopulation • subtract the Average Scale Score of the Control Group from the Average Scale Score of the Treatment Group • Calculate the estimate of treatment and test statistic. SUBSTEPS FOR CALCULATION

20. Estimate the effect for the whole population • Multiply the “Estimation of the subpopulation effect” by the “percent of the sample” represented by that subpopulation (e.g., 1.45 * 0.03 = 0.04) to apply the relative weight of that subpopulation • Sum the products to estimate the effect for the whole population (now controlled for gender and parent’s education) • Calculate the estimate of treatment and test statistic. SUBSTEPS FOR CALCULATION

21. Estimate the variance of the effect of treatment • The variance of each group’s average scale score is the square of the Standard Error of that’s group’s average scale score • Calculate the estimate of treatment and test statistic. SUBSTEPS FOR CALCULATION

22. Estimate the variance of the effect of treatment • The variance of the difference of the average scale score between groups is calculated by summing the variance of each group’s average scale score. No covariance coefficient is subtracted because the “worst case” of zero covariance is assumed. This assumption is made because the Data Explorer tool will not allow covariance to be calculated and this assumption raises confidence in any rejection of the null hypothesis. • Calculate the estimate of treatment and test statistic. SUBSTEPS FOR CALCULATION

23. Estimate the variance of the effect of treatment • Multiply the “variance of the differences of the average scale score” for each subgroup by the the square of the “percent of the sample” represented by that subpopulation to apply the relative weight of that subpopulation. • Sum those products to estimate the variance of the estimate of the effect in the population • Calculate the estimate of treatment and test statistic. SUBSTEPS FOR CALCULATION

24. Calculate the test statistic and p-value • The test statistic is the ratio of effect to variance of effect, and is calculated by dividing the estimate of effect by the variance of the estimated effect • The p-value for the test statistic is found by comparing the statistic to a normal distribution. This is done because the sample size of the NAEP is large enough to allow the variance of the average score of each subpopulation to be treated as if known (instead of estimated) • Calculate the estimate of treatment and test statistic. SUBSTEPS FOR CALCULATION

25. Calculate the estimate of treatment and test statistic. Record the results in the table of treatment effects table

26. Addressing Research Questions Once the Measurement Table (below) is completed, no further statistical testing is advisable (both for Type I errors and simple lack of meaningful tests). The author must look for patterns of effects, and then draw and defend his own conclusions.

27. What’s Next? All the hurdles preventing the completion of the dissertation are now surmounted and progress should be fairly unimpeded from now on. Remaining writing steps include: • Minor changes to Intro and Lit Review chapters. • Extensive changes to the Methodology Chapter, including adding a section on how NCES prepares NAEP data for the Data Explorer online tool • Chapter 4 will be tediously collecting online data, performing calculations, and finding ways to clearly illustrating the results. • Chapter 5 will need to be carefully written to avoid drawing erroneous conclusions

28. Final thoughts… • Waiting a year for the “restricted use” data was a mistake, but not one that could be easily avoided • Time is running out to complete the ILEAD program, so there is an even greater need for speed • It’s time for me to assemble a committee and begin the formal process of completing the dissertation and graduating.