1 / 29

Assessing Undergraduate Sustainability Knowledge Campus Wide :

Environmental and Social Sustainability Lab. Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath - UMD. Assessing Undergraduate Sustainability Knowledge Campus Wide :. From design to implementation and analysis. Overview.

raisie
Download Presentation

Assessing Undergraduate Sustainability Knowledge Campus Wide :

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Environmental and Social Sustainability Lab Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath - UMD Assessing Undergraduate Sustainability Knowledge Campus Wide: From design to implementation and analysis

  2. Overview • How we developed our Assessment of Sustainability Knowledge (ASK) • Why an ASK is important & how it can help • An aside on Knowledge and Literacy • Conducting an assessment • Thinking long term…

  3. Developing an assessment • Built upon the “triple bottom line”, the “three legged stool”, the “3 p’s” • Environmental (planet) • Economic (prosperity) • Social (people)

  4. Developing an assessment • Replicated questions used in the past • Coyle, 2005. “Environmental Literacy in America.” • Solicited topics and questions from experts • Held expert focus groups • Pilot tested among professors, graduate, and undergraduate students • Narrowed down to 30 questions

  5. Developing an assessment • Distributed those 30 to OSU students • Used IRT to throw out 14 • Added UMD’s 16 • Distributed those to OSU and UMD students • Used IRT to throw out 2 • Current ASK has 28 items: • ess.osu.edu

  6. What is the most common cause of pollution of streams and rivers? Surface water running off yards, city streets, paved, lots, and farm fields Which of the following is the most commonly used definition of sustainable development? Meeting the needs of the present without compromising the ability of future generations to meet their own needs Conceptualizing sustainability knowledge Environmental Social Sustainability Many economists argue that electricity prices in the U.S. are too low because… They do not reflect the costs of pollution from generating the electricity Economic

  7. How is this helpful?

  8. Need for measuring knowledge • University goals- • More along the lines of: • “Become carbon neutral by 2050” • Less common: • “Create sustainably minded citizens of tomorrow” • Can help track improvement over time

  9. Need for measuring knowledge • Serves as an evaluation for specific academic efforts: • Interdisciplinary programs • Sustainability majors/minors • STARS Credit: • ER 6 (STARS 2.0) – 3 points available

  10. “Knowledge” vs. “Literacy” • Knowledge • Can be objectively measured • Can be used to evaluate academic programs • Literacy • = Knowledge • Knowledge +values, attitudes, and behaviors • Can be used to evaluate outreach efforts, sustainability campaigns

  11. Measuring Behaviors, Values, and Attitudes • Make sure you are asking the right questions • Collaborate with sustainability departments to target specific behaviors (e.g., leaving lights on) • Include questions on behavioral barriers • Other behaviors that may get at the same concept

  12. Measuring Behaviors, Values, and Attitudes • Collaborate with academic departments to develop a good survey design • No need to reinvent the wheel • Each survey could be a Master’s thesis

  13. Conducting an Assessment

  14. Conducting an Assessment • Find your partners! • IRB approval • Required for publication • Exempt status • Registrar approval* • Student’s emails, majors, and demographics • Survey software* • We use Qualtrics, but there are others *If a large scale assessment is planned

  15. Maximizing Response Rates • Maximizing response rates is important to reduce uncertainty about how well your completed sample matches the population of interest. • Dillman(2008) and others have long studied how to maximize response rates for surveys that were telephone, mailed, or in-person. • There is growing research on electronic survey response rates.

  16. Research Methods • We compare response rates from five different survey implementations: • 1: 2012 spring OSU (n=10,000) • 2: 2013 spring OSU sample A (n=10,000) • 3: 2013 spring OSU sample B (n=10,000) • 4: 2013 spring OSU School of ENR (n=538) • 5: 2013 spring UMD (n=10,000) • We tried different treatments and tracked survey responses with survey software (Survey Monkey and Qualtrics)

  17. Key Variables Affecting Response Rates among College Students • Timing • When to send the invitation and reminders • Incentives • Email text • Who it is from • Questionnaire format • Long list of questions vs. more page clicks

  18. Timing Matters • Time of Semester: • Last week of semester and into finals week • Middle of semester • Time of Day: • 6:00 am • 6:00 pm • Reminders are critical • big spike in completed surveys after each reminder, with a fast decay

  19. Timing Matters • Overall, you want students sitting at their computers… but wanting to be distracted

  20. Incentives • Some disagreement in research on best way to provide incentives • ahead of time (Dillman 2008) • randomly select winner of 1 big prize • give more/all respondents smaller prizes • May impact the validity of the data

  21. SurveyResponses UMD n=1,556 Day OSU n=2,621 Day

  22. Email text • How the invitation emails are phrased affects response rates. • Besides making the invitation personal, clear, and as short as possible, prior research has found that who the invitation comes from matters. • An appeal from a trusted authority increases response rates.

  23. Email text: Appeal from Authority • We compared an appeal from authority versus an appeal from a peer (student). • Two surveys, A and B, had an appeal from a higher authority (University VP) for first 3 contacts. • For the 4th contact kept the higher authority for A, but switched B to have to an appeal from a grad student

  24. Email text: Appeal from Authority • Results: = additional 538 = additional 348 over the next 65 hours

  25. Questionnaire Format • Trade-off: • Long list of questions to scroll down • Shorter lists with a page click to get to the next page • We analyzed respondent drop-outs spots • Highest spots were just after clicking to the next page • We recommend finding a balance…

  26. Non-Respondent Bias • Even after efforts to maximize response rates, are the non-respondents different from the respondents? • We conducted a non-respondent short survey (5 questions + some demographics). • Results indicate that non-respondents are slightly but significantly less knowledgeable about sustainability topics, but no difference in GPA or pro-environmental behaviors.

  27. Planning ahead… • For longitudinal studies: • Write a multiple year IRB • Let registrar know this is a yearly survey • Finding partners • Sustainability office • Academic departments with survey expertise (interdisciplinary social science, communications, environmental studies, sociology, political science, psychology)

  28. Acknowledgements Funded by: • Office of Energy Services and Sustainability http://sustainability.osu.edu/ • OSU’s School of Environment & Natural Resources • http://senr.osu.edu/

  29. Thank You! Questions? • Environment and Social Sustainability Lab • ess.osu.edu • Contains: • This presentation • The 28 question ASK • Forthcoming article • Email • Zwickle.1@osu.edu • Koontz.31@osu.edu

More Related