Developing an Assessment of Sustainability Knowledge (ASK) for undergraduate students

1. Environmental and Social Sustainability Lab Adam Zwickle Tomas Koontz Andrew Bodine Kristina Slagle Developing an Assessment of Sustainability Knowledge (ASK) for undergraduate students

2. Overview • Need for a systematic way of measuring knowledgeof sustainability concepts • Developing an assessment • Results • Where we go from here

3. Need for measuring knowledge • Lacking in the scientific community • “Sustainability” has been defined and measured a myriad of ways • Often focused merely on the environmental aspect • Commonly tangled with values, attitudes, and beliefs • Needed to operationalize abstract goals…

4. Need for measuring knowledge • University goals- • More along the lines of: • “Become carbon neutral by 2050” • Less common: • “Create sustainably minded citizens of tomorrow”

5. Developing an assessment • Built upon the “triple bottom line”, the “three legged stool”, the “3 p’s” • Environmental (planet) • Economic (prosperity) • Social (people) • Basic knowledge from each knowledge domain? ~OR~ • Knowledge bridging the different spheres?

6. Developing an assessment • Replicated questions used in the past • Coyle, 2005. “Environmental Literacy in America.” • Solicited topics and questions from experts • Held expert focus groups • Pilot tested among professors, graduate, and undergraduate students • Narrowed down to 30 questions

7. Conceptualizing sustainability knowledge Environmental Social Sustainability Economic

8. Phase 1 • Administered 30 question assessment • Through an online survey • Sent to over 10,000 currently enrolled undergraduates • 14.3% response rate

9. Phase 1 - Analysis • Utilized Item Response Theory (IRT) To remove 14 questions • IRT- • Developed in the educational testing field • Used for standardized tests (GRE)

10. An example of IRT… Correct “Over the past 3 decades, what has happened to the difference between the wealth of the richest and poorest Americans?” Information 50/50 threshold Incorrect

11. Questions - Environmental • What is the most common cause of pollution of streams and rivers? • Ozone forms a protective layer in the earth's upper atmosphere. What does ozone protect us from? • What is the name of the primary federal agency that oversees environmental regulation? • What is the primary benefit of wetlands? • Which of the following is an example of sustainable forest management? • In the U.S., what do we currently do with the nuclear waste generated by nuclear power plants?

12. Questions - Social • Which of the following is the most commonly used definition of sustainable development? • The wealthiest 20% of people in the U.S. own approximately what percent of the nation’s privately held wealth? • Over the past 3 decades, what has happened to the difference between the wealth of the richest and poorest Americans? • Higher levels of education generally lead to... • Which of the following populations has the highest rate of growth?

13. Questions - Economic • Many economists argue that electricity prices in the U.S. are too low because… • Which of the following countries has now passed the U.S. as the biggest emitter of the greenhouse gas carbon dioxide? • Which of the following is a leading cause of the depletion of fish stocks in the Atlantic Ocean? • Which of the following is the most commonly used definition of economic sustainability? • Which of the following is the primary reason that gasoline prices have risen over the last several decades in the U.S.?

14. Results • Environmental 6 questions Mean: 4.39/6 SD: 1.48 73% • Social 5 questions Mean: 3.55/5 SD: 1.23 71% • Economic 5 questions Mean: 3.03/5 SD: 1.27 61% • Total 16 questions Mean: 11.08/16 SD: 3.21 69%

15. Example Analysis ANOVA F(3,1330)=8.09, p<.001

16. Example Analysis • Compare sub score by major • Economic score • Economics: 3.82 • Aerospace engineering: 4.00 • Animal science: 3.15 • Finance: 3.02 • English: 2.80 • Accounting: 2.63

17. Phase 2 • Combined our shortened set with a separate assessment developed at the University of Maryland • Administered survey at both campuses

18. Phase 2 - Analysis • IRT to compare information richness of questions • across campuses • using local descriptives (“Fishermen on Lake Erie” vs. “Fishermen in the Chesapeake Bay”) • different ways of asking about the same topic • Structural equation modeling • Compare question types (multiple answer?) • Knowledge structures (3 or 1 domain?)

19. Structural equation modeling Environmental Social Sustainability Economic

20. Survey Implementation • Best practices to maximize response rates: • Invitation script (from authority vs. student) • Page length vs. number of pages • Use of incentives • Time of academic calendar • Time of day, day of week • Reminders

21. Moving Forward • Adjust the instrument: • Some answers change over time • Develop test bank of validated questions – try this on your campus and send us questions • Be clear about limitations: • Multiple choice format • Measures knowledge only • Publish • Phase 1 – International Journal of Sustainability in Higher Education, 2014

22. Moving Forward • Goals: • Evaluate sustainability themed majors, programs, and courses • Serve as an indicator for comparison within institutions • Encourage friendly competition • Hopes: • Help solidify the concept of “sustainability” • Provide some credence for more abstract educational goals

23. Acknowledgements Funded by: • The Ohio State Office of Sustainability • http://sustainability.osu.edu/ • OSU’s School of Environment & Natural Resources • http://senr.osu.edu/

24. Thank You! Questions? • Environment and Social Sustainability Lab • www.ess.osu.edu • Contains: • This presentation • The 16 phase 1 questions with multiple choice answers • Email • Zwickle.1@osu.edu • Koontz.31@osu.edu

25. Appendix • Phase 1 -16 Question Sustainability Literacy Assessment

26. Questions - Environmental • What is the most common cause of pollution of streams and rivers? • Dumping of garbage by cities • Surface water running off yards, city streets, paved lots, and farm fields • Litter near streams and rivers • Waste dumped by factories • Don'tknow • Ozone forms a protective layer in the earth's upper atmosphere. What does ozone protect us from? • Acid rain • Climate change • Sudden changes in temperature • Harmful UV rays • Don'tknow

27. Questions - Environmental • What is the name of the primary federal agency that oversees environmental regulation? • Environmental Protection Agency (the EPA) • Department of Health, Environment, and Safety (the DHES) • National Environmental Agency (the NEA) • Federal Pollution Control Agency (the FPCA) • Don'tknow • What is the primary benefit of wetlands? • Promote flooding • Clean the water before it enters lakes, streams, rivers, or oceans • Keep the number of undesirable plants and animals low • Provide good sites for landfills • Don'tknow

28. Questions - Environmental • Which of the following is an example of sustainable forest management? • Setting aside forests to be off limits to the public • Never harvesting more than what the forest produces in new  growth • Producing lumber for nearby communities to build affordable housing • Putting the local communities in charge of forest resources • Don’t know • In the U.S., what do we currently do with the nuclear waste generated by nuclear power plants? • Use it as nuclear fuel • Sell it to other countries • Dump it in landfills • Store and monitor the waste • Don't know

29. Questions - Social • Which of the following is the most commonly used definition of sustainable development? • Creating a government welfare system that ensures universal  access to education, health­care, and social services • Meeting the needs of the present without compromising the ability of future generations to meet their own needs • Setting aside resources for preservation, never to be used • Building a neighborhood that is both socio-demographically  and economically diverse • Don’tknow • The wealthiest 20% of people in the U.S. own approximately what percent of the nation’s privately held wealth? • 20% • 35% • 50% • 85% • Don'tknow

30. Questions - Social • Over the past 3 decades, what has happened to the difference between the wealth of the richest and poorest Americans? • The difference has increased • The difference has stayed about the same • The difference has decreased • Don’tknow • Higher levels of education generally lead to... • Lower levels of voter turnout • Greater annual earnings • Larger family size • Higher self esteem • Don’t know

31. Questions - Social • Which of the following populations has the highest rate of growth? • North America • Europe • China • Africa • Don’t know

32. Questions - Economic • Many economists argue that electricity prices in the U.S. are too low because… • They do not reflect the costs of pollution from generating the  electricity • Too many suppliers go out of business • Electric companies have a monopoly in their service area • Consumers spend only a small part of their income on energy • Don’tknow • Which of the following countries has now passed the U.S. as the biggest emitter of the greenhouse gas carbon dioxide? • China • Sweden • Brazil • Japan • Don’t know

33. Questions - Economic • Which of the following is a leading cause of the depletion of fish stocks in the Atlantic Ocean? • Fishermen seeking to maximize their catch • Reduced fish fertility due to genetic hybridization • Ocean pollution • Global climate change • Don’tknow • Which of the following is the most commonly used definition of economic sustainability? • Maximizing the share price of a company's stock • Long term profitability • When costs equal revenue • Continually expanding market share • Don’t know

34. Questions - Economic • Which of the following is the primary reason that gasoline prices have risen over the last several decades in the U.S.? • Growing percentage of gas stations owned by large  corporations • Increasing oil discoveries overseas • Higher rates of state and federal gasoline tax • Increasing global demand for oil • Don’t know