Nationally Representative Sample Surveys: What Can They Tell Us About Remediation? What Should They Do Next?

1. Nationally Representative Sample Surveys: What Can They Tell Us About Remediation? What Should They Do Next? Tom Weko National center for Education Statistics, Postsecondary, Adult, and Career Education 30 June 2010

2. What Nationally Representative Sample Surveys Cannot Do… • Research that uses evidence based upon RCT or quasi-experimental methods can (potentially) provide strong causal inferences about what works. • Nationally representative sample surveys (NRSS) are not well-suited to strong causal inferences about effects. • Nonetheless, there are some important contributions to be made by NRSS, which I’ll focus on today. • They are: • (Obviously), nationally representative data; • That contain student self-reports about things that are analytically important, but cannot capture through administrative data.

3. But, They’re Nationally Representative…. • Much of the work on effects of remediation is done at an institutional or state level. • NRSS are the only means by which to address questions about the incidence of remedial education among a nationally representative sample of institutions and students. • They will remain the only nationally representative picture for the foreseeable future, since postsecondary SUR systems: • Are still in development (29 SURS in 25 states have course-level remediation data) • Are not inclusive of all institutions (nonprofit, for profit); • Do not follow common data standards; • Are not interoperable.

4. Nationally representative…. NRSS data about the incidence of remedial education are captured by one (or, two) means. • Self-reported in the student interview. Planned for NPSAS:2012…. • [1st and 2nd year students] Since you completed high school, have you taken remedial or developmental courses to improve your basic skills, such as in mathematics, reading, writing, or studying? • Did you take any remedial or developmental courses during the 2011-2012 school year? • How many remedial or developmental courses did you take in the following subjects during the 2011-2012 school year in Mathematics? Reading? Writing? • Should we attempt to distinguish between course attempts and completes in the interview? Include study skills?

5. Nationally representative…. • Transcripts. NCES secondary longitudinal studies (NELS:88/2000) have collected and analyzed transcripts to examine remedial course taking (NCES 2003-394). The 2008 Postsecondary Education Transcript Study (PETS) has collected and coded 25,190 transcripts for the 2003/9 BPS study and 16,080 for the 2008/09 B&B study. Principal resource is BPS, not B&B, since latter is completion cohort, and has collected transcripts only from BA awarding institution, not institution of origin if other than BA institution. • Public use data based upon these transcript data will be available in 2011. Released through PowerStats (http://nces.ed.gov/datalab/) • Restricted data based upon these transcripts will also be available in 2011.

6. NRSS contain analytically valuable student reported data NRSS contain data elicited in the student interview that are missing in administrative data. • Self-reported measures income and parental education that can be used to characterize SES, so we know who is taking these courses. • Student information, expectations, and evaluations about remedial education. Such as: Were they expecting this to happen? Before enrolling in postsecondary education did they seem themselves as having weaknesses in the subject(s) for which they’re now taking remedial coursework? • Other student information and expectations that may be associated with remediation, such as student expectations about the probability of transfer or graduation, and expectations about wages.

7. What should we do next in NRSS? • Remedial course taking measured through interviews or transcripts has an important limitation: institutions use different assessment instruments and cut scores to assign students to the treatment, so we’re not capturing a standardized measure of ability or persistence risk. • Ideally, survey data collections would collect whichassessments are used, actual assessment scores, and cut points for assignment to remediation from sampled institutions. • Can this be done? • Where on campus do these data reside? • How long are they maintained? • How many institutions are using local assessments, as opposed to the major industry assessments? • Is it possible to maintain institutional participation rates while collecting cut points?

8. What Should We Do Next? • The National Assessment Governing Board is seeking OMB approval for “Evaluating Student Need for Developmental or Remedial Courses at Postsecondary Education Institutions” http://www.reginfo.gov/public/do/PRAViewICR?ref_nbr=201006-3098-001 120 pilot institutions, July 2010, 1,670 full scale 2 and 4 year institutions, November 2010 • NPSAS:2012 field test planning to ask institutions to provide information on the type of placement test administered, and individual student scores. • The payoff? NRSS data would contain “standardized measures of ability” and an opportunity to look at what happens when students with like assessment scores are exposed to different information about their abilities, i.e. assigned to remediation or not.