A Standard Survey for Undergraduate Non-Science Majors: The Astronomy Diagnostic Test

1. A Standard Survey for Undergraduate Non-Science Majors: The Astronomy Diagnostic Test Beth Hufnagel University of Maryland, USA hufnagel@astro.umd.edu 7 August 1999 AAPT Summer MeetingSan Antonio, Texas

2. Faculty E. F. (Joe) Redish David Hammer John Layman (emeritus) Grad Students Aziza Baccouche Lei Bao Hadley Lawler Rebecca Lippmann Mel Sabella Postdocs Richard Steinberg§ Michael Wittmann Andy Elby* Apriel Hodari* Beth Hufnagel*(§ NAE Spencer Fellow *PFSMETE Fellow)

3. OverviewAstronomy Education Research Students knowledge andbeliefs about … The Universe“Conceptions” Knowing“Epistemology” Themselves“Affect” Development and assessment of curricula and teaching techniques

4. Measuring Concept Learning: Why A Standardized Survey? • Document the effectiveness of teaching. • Understand student preparedness. • Guide selection of curriculum/teaching approach. • Measure the effectiveness of different teaching approaches on learning.

5. Collaboration for Astronomy Education Research (CAER) Montana State Jeff Adams Christine Brick Tim Slater Univ. of Maryland Grace Deming Beth Hufnagel Univ. of Nebraska Rebecca Lindell Adrian Univ. of New Mexico Michael Zeilik

6. What is theAstronomy Diagnostic Test? • 33 multiple-choice questions • 21 concept/knowledge questions • 12 student background questions • Validated and Reliable • A wrong answer means the student doesn’t understand. • A right answer usually means the student understands. • Diverse Comparison Database • 22 classes ranging from 6-201 students • 17 institutions of 5 types

7. Pre-Course Results Spring 1999Sample ADT Question8. Where does the Sun’s energy come from? A. The combining of light elements into heavier elements. B. The breaking apart of heavy elements into lighter ones. C. The glow from molten rocks. D. Heat left over from the Big Bang. Did not attempt to answer. Total No. % 420 27 870 56 142 9 122 8 3 0 1557 100

8. 1998/1999 Development of ADT version 2.0 • The choice of ADT topics was based on two prior diagnostics by Phil Sadler and Mike Zeilik. • 2500+ students in 26 institutions of 6 types • Statistics, e.g., Spring 1999Discrimination range 0.32–0.67 (out of –1.00 to +1.00). • The ADT version 1.0 was re-written using standard psychometric rules. • 30 student written responses to open-ended questions. • Validation based on ~60 interviews, • open-ended in Fall 1998, • with distractors in Spring 1999.

9. Research Questions Do the results depend on … • the type of institution - community college, liberal arts, state university, or technical? Yes • the class size? No • gender? Yes – Why?

10. Results from Spring 1999Pre-Course Scores by Gender Gender matters. Female Male N 825 683 Mean 28% 38% Std. Error 0.4% 0.6%

11. Summary • The ADT is now available in PDF format from http://solar.physics.montana.edu/aae/adt/. • Encourage conceptual understanding as a goal for astronomy courses. • More quantitative measurement of the effectiveness of curricula/learning tools. • CAER will develop a web-based, ADT database with independent concept clusters.

12. Favorite Quote from StudentInterview “I seem to know, or think I know, a lot of things. I just don’t know … the insides of them.” - Debbie(Undergraduate astronomy student during the last week of the course.)

13. Thanks to these generous participants:

14. Phil Appleton, Iowa State University • Chris Depree, Agnes Scott College • Van Dixon, Vista Community College • Andrea Dobson, Whitman College • Jack Dostal, Hawkeye Community College • Doug Hamilton, University of Maryland at College Park • Mary Kay Hemenway, University of Texas at Austin • Thomas J. Kassebaum, Denison University • Janet Landato, W.R. Harper College • Bernie McNamara, New Mexico State University • Heather Morrison, Case Western Reserve University • Jeff Saul, North Carolina State University • Steve Shawl, University of Kansas • Harry Shipman, University of Delaware • Patrick Shopbell, Prince George’s Community College • Greg Stacy, Southern University • Bill Warren and Rolfe Chandler, Lord Fairfax Community College

15. Results from Spring 1999Pre-CourseScores by Class Size The average pre-course score does not depend on class size.

16. Results Spring 1999 Pre-course Scores by Institution/Gender #Classes #Students Women% Men% State Universities 11 635/499 28(1.0) 37(1.5) Liberal Arts 5 110/124 26(1.1) 38(5.1) Community Coll. 4 52/43 28(1.7) 39(2.6) Technical 1 6/17 29(14) 59(21) Women’s 1 22/0 30(8) n/a (standard error in parentheses) In all 22 classes, the average pre-course score for men is higher than the average score for women.

17. Implications for Instruction: Explicitly Evoke Useful Ideas Students come to an introductory astronomy class with a great deal of knowledge. A professor can help her students use their existing correct and relevant ideas to construct theories closer to those held by astronomers. For example, all of the students knew that the Sun is hot, and that fluids (like coffee) will cool if no heat source is present. Use a demonstration to encourage them to apply this knowledge to stars so that they will conclude stars need an internal heat source.

18. Please Add Name for Preprints San Antonio AAPT Summer Meeting