9. Weighting and Weighted Standard Errors

1. 9. Weighting and Weighted Standard Errors

2. Prerequisites • Recommended modules to complete before viewing this module • 1. Introduction to the NLTS2 Training Modules • 2. NLTS2 Study Overview • 3. NLTS2 Study Design and Sampling • NLTS2 Data Sources, either • 4. Parent and Youth Surveys or • 5. School Surveys, Student Assessments, and Transcripts • 6. Implications for Analysis and either • 7. Parent and Youth Surveys or • 8. School Surveys, Student Assessments, and Transcripts

3. Overview • Purpose • Sampling weights overview • Creating NLTS2 sampling weights • Sampling weight example • Which sampling weight to use • Obtaining correct standard errors by correcting for design effects • Type 1 error • Analysis recommendations • Closing • Important information

4. Purpose • Learn how to obtain valid point estimates by using sampling weights. • Learn how to obtain valid standard errors by adjusting for “design effects.”

5. Sampling weights overview • Purpose of sampling weights • To produce point estimates that are representative the universe (i.e., national population of students with disabilities in age group; each disability category).

6. Sampling weights overview • Why do we need to weight? • When a population includes some low-incidence groups of interest, those groups typically are oversampled. • Oversampling ensures that the sample includes some members of those groups. But unless weighted, that sample does not represent the total population results. • Demographic groups may have differential response rates.

7. Sampling weights overview If all students in a disability group had been equally likely to be selected,then to make results represent the full population, weight each observation by the number of individuals in the population that it represents. Example using synthetic data for illustrative purposes:

8. Creating NLTS2 sampling weights • Example • Local education agency (LEA) had 100 students with learning disability. • Our sample had 10 students with learning disability, drawn with equal probabilities. • We obtained responses from 5 students with learning disability. • Students’ within-LEA weight would be 100/5= 20. • The universe of LEAs in the cell served 400,000 students. • Sampled LEAs in a cell, drawn with equal probabilities, served 2,000 students. • Each student in the sampled LEAs represents 400,000/2,000= 200 students. • Students’ total sampling weight would be 20 x 200 = 4,000.

9. Creating NLTS2 sampling weights • State schools were treated as a sampling cell. All were sampled; not all responded. • For each disability category, a weight was calculated by multiplying the number of students with that disability on the rosters of the responding schools by the inverse of the proportion of state schools that submitted rosters.

10. Creating NLTS2 sampling weights • Adjustments • So that weighted sample matches the number of students in each disability category, as reported to OSEP by the states for the 1999–2000 school year. • So that the weighted sample matches known characteristics of the population, such as age group and race/ethnicity.

11. Things to note about NLTS2 samplingweights • The youth is the unit of analysis. • Research questions must pertain to youth only. • With the weights provided by NLTS2, data cannot be used to represent the universe of teachers, classrooms, schools, districts, or states. • There is clustering as a result of the sample design. • Clustering has implications for standard errors. This is covered a little later. • Results using NLTS2 data must always be weighted for reporting or publishing. • Weights differ for each wave and each instrument.

12. Sampling weight example Synthetic data for illustrative purposes.

13. Sampling weights example Differences in weights across waves and data collection instruments Synthetic data for illustrative purposes.

14. Which sampling weight to use? “Help! I have more than one weight to choose from; which one do I use?” • When combining data from multiple sources (instruments or waves) in analyses that require data from all of those sources, a general rule is to use the weights from the source for which the sum of the weights of the individuals is largest. • When combining data from sources with a lot of nonoverlapping data, proceed with caution and consult a statistician.

15. Correcting standard errors for design effects • Reason: In a multistage sample, multiple observations are drawn a given location. Observations may be correlated. • Sample weights will produce correct point estimates but may produce incorrect standard errors. • Statistical adjustments for clustering “design effects” produce correct standard errors.

16. Correcting standard errors for design effects • Use • Taylor linearization • Replicate weights • For NLTS2, SRI’s approximation algorithm.

17. Correcting standard errors for design effects:Taylor linearization approach • With Taylor linearization, you specify the strata and the cluster variables. • Stratum variable has 64 values (geographic region, size, and wealth) • Cluster (first-level PSU) variable: LEA • A problem arises: when there is only one observation per cluster. • Solution: Collapse these clusters within their stratum. Generally, we recommend using replicate weights.

18. Correcting standard errors for design effects:Replicate weights • For any data collection activity, NLTS2 provides 32 replicate weights. • Each replicate is calculated as follows: • Half the LEA sample is used. • Respondents in selected LEAs are weighted up to the universe. • Differences across replicates reflect true variability in the full sample.

19. Correcting standard errors for design effects:SRI’s simple approximation SE = Standard error M = Mean of the sampling weights V = Variance of the sampling weights Adjusted SE = SE *

20. Correcting standard errors for design effects: • What does an adjusted standard error buy you? • Accurate confidence intervals around estimates • Greater confidence that differences observed are truly differences in the population

21. Weighted standard errors: Example means and standard errors Randomly selected subset of the NLTS2 data used in all examples

22. Weighted standard errors: Example estimates and p values for gender difference Randomly selected subset of the NLTS2 data used in all examples

23. Examples of statistical packages thatcalculate weighted standard errors • SAS • STATA • WESVAR Taylor linearization or replicate weights • SUDAAN • SPSS – Taylor linearization only

24. Weighted standard errors • Examples of statistical packages that calculate weighted standard errors • NLTS2 training modules use SAS and SPSS for analysis examples. • SPSS – General statistical package • Requires SPSS Complex Samples Module for calculating accurate standard errors • GUI interface; Taylor linearization • SAS – General statistical package • Requires the SAS Statistics Module for calculating accurate standard errors in PROC SURVEY procedures • Taylor linearization or replicate weights

25. Type 1 error • Applies to NLTS2 as to any other study • Temptation to perform many tests drastically increases the probability of spurious results • Use caution

26. Analysis recommendations Two analysis recommendations • Start simple! • Before you start more complicated analyses, • Understand who’s in and who’s out of the analyses. • Understand the distributions. • Understand the bivariate relationships.

27. Closing • Topics discussed in this module • Sampling weights overview • Creating NLTS2 sampling weights • Sampling weight example • Which sampling weight to use • Obtaining correct standard errors by correcting for design effects • Type 1 error • Analysis recommendations

28. Closing • Next module: • 10. NLTS2 Documentation Overview

29. Important information • NLTS2 website contains reports, data tables, and other project-related information http://nlts2.org/ • Information about obtaining the NLTS2 database and documentation can be found on the NCES website http://nces.ed.gov/statprog/rudman/ • General information about restricted data licenses can be found on the NCES websitehttp://nces.ed.gov/statprog/instruct.asp • E-mail address: nlts2@sri.com