1 / 32

Casey Langer Tesfaye and Susan White

Can a survey of U.S. High Schools be Replaced or Reduced through Web Searches? The Successes and Complications of an Experimental Strategy. Casey Langer Tesfaye and Susan White. Nationwide Survey of High School Physics Multiphase survey Sample: public & private HS via NCES

donnan
Download Presentation

Casey Langer Tesfaye and Susan White

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Can a survey of U.S. High Schools be Replaced or Reduced through Web Searches? The Successes and Complications of an Experimental Strategy Casey Langer Tesfaye and Susan White

  2. Nationwide Survey of High School Physics • Multiphase survey • Sample: public & private HS via NCES • Refreshed quadrennially, Redrawn periodically • Phase 1: schools, Phase 2: teachers Background

  3. Does your school offer Physics? Names & e-mail addresses of physics teachers Phase 1: Schools

  4. Multimode survey • Mode: paper, web, phone Background

  5. Paper survey is really expensive Problem 1

  6. Schools are oversurveyed • One IRB is not enough • More and more schools have custom research applications or processes Problem 2

  7. Address lists out of date • Some schools close or move • Some schools difficult to reach by phone Problem 3

  8. Eliminate paper survey altogether Reduce contact with schools Filter out closed schools faster Goal

  9. Begin with web searches Call remaining schools Proposed solution

  10. 3 minute limit Zoned form Strategy

  11. Responses • Response rate • Data quality • Research process requests • $ savings Evaluating the effort

  12. Overall response rate lower, but still over 90% Outcome

  13. Outcome

  14. Difficult to measure Many issues to consider Data Quality

  15. Googling is a skill • Google evolves • Search strategies and techniques vary • Evolve over time There is a tradeoff between encouraging creative search strategies and evaluating the quality of the information yielded Training and Streamlining

  16. Sources varied • Search results & search pages (e.g. Physics Olympic teams, activities rosters) • School websites, class websites, department pages, grade-level pages, parental resources • Great schools websites • LinkedIn and other social networking sites Data Sources varied

  17. Many schools share the same name • Double check school location • Many cities share the same name • Use full/er address to verify • Many teachers share the same name • Listings with last names only are common Quality Control

  18. Very different configurations of information for each school • Some easier to navigate, some had more or less information or differently targeted information (e.g. sports teams) • Different definitions (e.g. Upper School) Information varied

  19. Search results vary • Date of updates varies • Site content varies • Websites go up & down, sometimes cached, sometimes redesigned Information varied

  20. Goal of form was staged • Stage 1: 1. Is physics information available? 2. Is physics teacher information available? What makes a complete entry?

  21. Goal of form was staged • Stage 1: 1. Is physics information available? 2. Is physics teacher information available? • Stage 2: 3. Then is there any helpful contact information? - What constitutes helpful contact information?  New classification: helpful, but not complete What makes a complete entry?

  22. Every mode of survey has data quality issues • Web searches yielded more inaccurate information that in the past • Inaccurate information leaks into teacher survey • E.g. “I don’t teach physics” “I teach physics, but I have no students this year” • More of a need for a streamlined system for corrections Data Quality

  23. Web search, email & phone call cheaper than Paper survey, email & phone call Web search & phone calls roughly equal (but web had the added benefit of filtering out closed schools faster and reducing contact with schools) Cost

  24. Web searches save $ • Web searches lead to less research process requests and potential rejections But… • Web searches cannot be the only mode • A system that uses web searches: • Must have streamlined procedures for updating incorrect information In Sum…

  25. More information online every day Google continually improving and streamlining search results The web continually evolves

  26. The web continually evolves

  27. Training cannot be static • Develop search and quality standards that can evolve over time • Develop a system that encourages web search staff to discuss their strategies • Use these interactions to understand what information you are collecting and where you are collecting it from The web continually evolves

  28. We will continue the web searches • But we will strengthen and streamline the feedback loop In the future

  29. Questions, comments? Casey Langer Tesfaye clanger@aip.org Thanks!

More Related