NHES 2012 Pre-Proposal Conference January 19, 2011

1. NHES 2012 Pre-Proposal Conference January 19, 2011

2. Overview • History and purpose of NHES • Redesign • NHES 2012 • Overview of data collection activities • Sampling, questionnaires, data collection procedures, telephone non response follow-up and data file creation • Overview of ED / NCES procedures • Deliverables and review procedures • Security plan and data protection • Question and answer

3. History and Purpose • Designed to study education research issues that cannot be addressed efficiently in institutional surveys • Allows NCES to collect data on a wide range of topics directly from households, including nonparental child care, parent involvement in education, school readiness, and adult education • Designed to allow for the analysis of trends in a number of important topics over time

4. History Continued • Nationally representative samples of children from birth through grade 12, as well as adults • Each administration has a screener survey and 2 or 3 extended, topical surveys • Had been conducted by telephone using list-assisted random digit dial (RDD) design • Interviews conducted in English or Spanish • Data collected from January through May because of school related items

5. Redesign • In 2008, a contract was awarded to redesign the NHES • There were two phases: • A small feasibility test in 2009 (pilot test) • A large-scale field test starting in January 2011 (field test) • The final design will be guided by results from the 2011 field test • The SOW will include options to account for the possible final design

6. NHES 2012 Overview • Data collection begins January 2012 and ends May 2012 • Two-phase (screener and topical) mixed-mode (mail and telephone) data collection

7. Data Collection • Households will be screened by mail for eligible children • Three topical surveys will be fielded by mail: • Early Childhood Program Participation (ECPP) • Targets children from birth through age 6, not in kindergarten • Parent and Family Involvement in Education (PFI) • Targets children age 6 – 20 in kindergarten through 12th grade (or equivalent); • includes a related but distinct homeschool questionnaire • Spanish versions of all instruments will be available (a bilingual screener may also be offered) • Surveys have already been designed but may need slight modifications and updates

8. NHES 2012 • Address Based Sample (ABS) will be utilized • Sampling of addresses from a near universal listing of residential mail delivery locations • Frame should be based on most recent USPS- DSF data • Contractor will need to provide a plan that details how they will handle frame issues such as throwbacks, vacant units, P.O. boxes, etc. during sampling, and PMRs and other issues during data collection • A phone match will be required for telephone non response follow-up

9. Sampling • The final sample must be nationally representative of the two target populations:  1)  children 0-6 and not yet in kindergarten and 2) children in grades K-12 • Oversampling will likely be required to ensure adequate Black/African American, Hispanic and other minority group representation for analysis • Contractor will need to develop a plan for within household sampling on a rolling basis

10. Sampling Continued • We estimate starting sample of approximately 198,000 households • Contractor can suggest alternate sizes • Carefully consider the design aspects of the survey when projecting response rates and eligibility rates • Sample size is driven in part by precision requirements for estimates

11. Screener Data Collection • Initial screener mailing will contain a $2 or $5 cash incentive depending on results from 2011 field test. • Security procedures will need to be in place for handling cash incentives (mailout and return) • A magnet may also be included • A prenotice letter prior to the initial screener mailing may also be used

12. Screener

13. Non Response Follow-up: Screener • Up to three screener questionnaires will be mailed to households • A possible prenotice letter • Reminder postcard • Third questionnaire mailing will use FedEx or Priority Mail • All packages include: cover letter, questionnaire, and prepaid return envelope • Telephone non response follow up may be utilized depending on results from the 2011 Field Test

14. Non Response Follow-up: Topical • Up to three topical questionnaires will be mailed to households • First mailing may be sent by FedEx or Priority Mail or include a cash incentive if using first class mail • An alternate envelope (color and wording) from the screener envelope may be required • All packages include cover letter, questionnaire, and prepaid return envelope • Option for telephone non response follow-up

15. Information to identify child will be printed on form We would also like to customize items

16. School Information Will be Printed from CCD and PSS files on PFI Contractor will need to identify schools in the household’s neighborhood from the CCD and PSS then print the names of these schools on this item. Respondent will mark which school their child attends. Contractor will later merge data about the selected school onto the restricted use data file from the CCD or PSS.

17. Other Considerations • Across all three screener waves 472,000 questionnaires could be mailed • Across all three topical waves 86,000 questionnaires could be mailed • Contractor should demonstrate experience managing mailing operations of this size and complexity • Quality control is critical • for privacy/confidentiality reasons questionnaires can not be sent to incorrect household (would be a reportable privacy breach)

18. Options • Screener switch • Testing two versions of screener (screenout and engaging) in 2011. Looking at impact of ‘switching’ version at 1st non response follow up. • Testing impact of prenotice letter for first screener mailing • Testing a variety of topical contact strategies • Differing incentive levels, mailing envelope, special handling

19. Telephone Non Response Follow-up • Telephone underperformed compared to mail in 2009 Pilot • Testing as a non response follow-up mode in 2011 • If option is exercised, looking for most cost effective implementation • Screener and topical are separate events • Control system needs to be able to move case back and forth between modes

20. Telephone Non Response Follow Up Continued • Interviews conducted in English and Spanish • All cases will need to be treated as refusal conversion • Information that will be helpful: • Number of experienced interviewers available for NHES • Overall phone response rates from past national studies you have conducted

21. Telephone Considerations • Approximately 32,000 screener non response cases • Roughly five week period to work screener cases before switching to topical • Approximately 7,000 topical non response cases (PFI and ECPP) • In past NHES ~5% cases completed in Spanish • Topical instrument will need to display data from screener and have a school look up table to match mail forms • Contractor should demonstrate experience managing a telephone operation of this scale and complexity within the NHES schedule

22. Telephone Number Look up • The Contractor should plan to look up telephone numbers for mail non responding cases where there is no match on the sample frame or the match is incorrect. (Sometimes called tracing or research) • Vendor databases used for searching need to meet NCES guidelines • E.g., vendor’s database can not be updated using NCES data and all information must be securely deleted after search is complete • A data protection contract must be in place between contractor and vendor doing the matching

23. Data Collection • Screeners must be QC’d and entered quickly to prepare for topical mailing • If names are collected, they need to be reviewed to ensure only valid names are printed on topical (ideally through some combination of manual / automated review). • Invalid names consist of curse words, blatantly fake names, etc. • Topicals will be mailed on flow basis no more than three weeks from receipt of screener • Less than 21 days to process screener, sample child, prepare topical print file, print (or label) topical questionnaire and cover letter, and mail

24. Metadata • Contractor will need to collect and report metadata on all aspects of study • Data needed to evaluate effectiveness of different collection stages and modes, efficiency of case management • E.g., date questionnaire mailed, date received at contractor, questionnaire wave, date keyed, respondent call logs, postmaster returns, case status including call attempts and outcome codes. • File preparation: • Flag variables edited or imputed, rate at which autocoding did not work and number of missed skip patterns

25. Data Entry • Contractor will be responsible for validating the accuracy of the data collected from all modes of data collection • Whatever method of data entry is selected by the contractor should already be validated for accuracy on large scale questionnaires with text, numeric and closed response options and have built in QC measures

26. System Design • In proposal contractor should emphasize their approach and experience handling the following aspects of data collection: • Checking in and processing large quantities of questionnaires that are returned in waves • Questionnaire design and design software capabilities • Data entry approach and ability to process these quantities of questionnaires within the specified time frames • Ability to track and control individual forms from printing stage through archiving

27. Building and managing case control systems for two phase multi-mode studies • The short data collection time frame and complex design necessitate an automated case management system • Project Manager should have daily access to sample performance in case course corrections are necessary

28. Case control systems for multi-mode studies continued • Requests for additional paper questionnaire: • From respondents during phone non response follow-up: • Contractor needs to generate a new mailing package, mail the questionnaire within a couple of days and place a two week hold on call attempts to the household to allow time for response. • From the government: • The government may request an additional mailing to households with certain phone status codes (e.g., soft refusal on the phone or ring no-answer).

29. Data File Creation • End product will be separate ECPP and PFI datafiles (restricted and public versions). Some screener data will be carried over to the datafile • Contractor will draft and implement specifications to edit and impute data • Should have experience editing mail survey data

30. Data File Creation - Editing • 100% imputation of missing and out of range values • NHES has used ‘hot deck’ procedures in the past • Should have experience developing and implementing hot deck imputation specifications • No unreadable fields or unallowable characters • Valid skips are not imputed and can be distinguished from valid zeros and nonzero data • Files are checked for internal consistency and basic tables are run and reviewed • Duplicate responses are reconciled

31. Data File Creation- Composite & Classification Variables • Contractor will create 20-30 composite or classification variables • SES using income measures, parent’s education, count of adults in HH, etc. • Some variables pull from external sources (ACS, Census, Common Core of Data) • Variables should be consistent with past NHES • All code documented and included in manual

32. Data File Creation • Prepare for Disclosure Review Board (DRB) • Prepare a Disclosure Analysis Plan (DAP) • A plan that proposes how the data will be protected from disclosure. This may include topcoding and perturbing data as well as other disclosure avoidance techniques • Weighting to ACS and Census • Create person level weights, rake to population totals • Standard Errors • Complex sample design, provide replicate weights and base weights • Previous NHES studies use Jacknife methods and PSU / Strata variables for Taylor Series variance estimation • Design effect values must be included in documentation • NCES seeks to release files within 8 months of data collection

33. Data File Creation • SAS, SPSS and Stata versions • Files prepared for upload to EDAT (NCES online data access tool) • Codebooks that include a listing of each variable, its values, labels and unweighted and weighted frequencies • User’s manuals (see examples on our website) • PFI, ECPP, overall and restricted use manuals • Cover all aspects of the survey design, sampling, data collection, data anomalies, data file structure and layout, and weighting

34. Reports and Review • NCES thoroughly documents datasets • Fully documented sampling plan, weighting plan, edit specifications, quality control procedures • Conduct bias analysis • Using frame and external data and possibly mode / wave data from the file • Accurately document response rates and case outcome codes

35. Definition of “Draft” • Complete product of high quality • Should be fully formatted and free of grammatical and typographical errors • The government expects that only minor revisions should be needed for drafts to become final deliverables

36. Reports • All reports must strictly adhere to NCES guidelines for style and content • A draft will not be acceptable if it does not meet these guidelines

37. Review Timeline for Publications • COR review (2 weeks) • Management review (3 weeks) • Senior management review* (6 weeks) • Technical review* (6 weeks) • Commissioner review (4 weeks) • IES Review (6 weeks) *concurrent

38. Security Considerations • Contractor needs to have facilities to securely store over 250,000 questionnaires • Sample information, as well as questionnaires, may be considered Personally Identifiable Information (PII) • Attempts should be made to minimize the number of subcontractors needing access to this information • All subcontractor employees and systems that will come in contact with NCES PII will need to meet the same requirements as the prime contractor, including background investigations and information systems certification and accreditation

39. Additional Security Considerations • If using an outside vendor for the sample frame, steps will need to be taken to minimize disclosure risk (e.g., drawing additional cases so vendor will not know who is in final sample)

40. Data Security Plan • Identify all processes where PII is collected, handled, and transferred • Explain how PII will be protected during all phases of collection, handling, and transfer within your staff and information systems, as well as those of any subcontractors

41. Information System Security • Contractor and subcontractor will need to go through Certification and Accreditation (C&A) process for their computer systems and receive an Authority to Operate (ATO) before data collection can begin • Extensive documentation and on site visits are required for this process • Contractor should allow 3 months from submission of complete paperwork to receive ATO

42. Contractor Should be Familiar With and Able to Meet the Requirements of: • IES Confidentiality Statute (20 U.S.C. 9573) • Confidential Information Protection and Statistical Efficiency Act (CIPSEA) of 2002 (Public Law 107-347, Title V) • Federal Information Security Management Act (FISMA) of 2002 (Public Law 107-347, Title III) • Management of Federal Automated Information Resources (OMB Circular A-130; Appendix III) • OMB Memorandum: M-06-15, M-06-16 and M-06-19 • National Institute of Standards and Technology (NIST) standards (NIST publications 800-37, Rev. 1; 800-53 Rev 3; 800-60) • FIPS Publications 199 and 200

43. C & A Process and Timeline for New System • Preliminary C&A Activities • e.g., Project Kick-Off Meeting, Data Sensitivity Worksheet, System Categorization (FIPS 199) • Phase I – Creation of Security Documents • System Security Plan (SSP), Contingency Plan (CP), Configuration Management Plan (CMP) • Phase II – Conduct Security Assessment (3 to 4 weeks) • e.g., Risk Assessment Report, ST&E, POA&M, Executive Summary, C&A Memo • Conduct Final Security Assessment Review (FSAR) (4 to 6 weeks) • Report detailing the current security posture of the system • Provides certification recommendations for the system • Findings from ST&E need to be addressed by System Owner. IV&V will work with System Owner to either 1) remediate, 2) accept risk, or 3) create a corrective action plan (CAP) for each finding, including completion dates • Accreditation of the System (1 week) • Preparation and signing of the Authorization to Operate (ATO) Letter • C&A Start-to-Finish Time (12 to 17 weeks) • Re-Certification of previously C&A’d systems may take less time, depending on current state of documentation

44. Timing Considerations • Many complex documents will need to be created concurrently after contract award to meet schedule dates: • C&A documents, Data Security Plan, cover letters and questionnaires, sampling plan, project schedule, and OMB package are all needed within approximately the same time frame • NCES seeks to release data within 8 months of data collection

45. Questions?

46. Contact Information • Sharon Masciana • Contract Specialist • 550 12th Street, SW, Room 7129 • Washington, DC 20202-4240 • Office - 202-245-6132 • Fax - 202-245-6296 • Sharon.Masciana@ed.gov