Researching Student Learning in the Geosciences

1. Researching Student Learningin the Geosciences Steve Semken This session is intended for those who have a working understanding of how learning research is applied to teaching. It will include a practical survey of quantitative and qualitative methods that can be used to collect and analyze data on student learning in different settings. July 2010

2. Why conduct research on your students’ learning? • It makes you a better teacher. • Feedback on how well your students learn is an indispensable part of reflective, professional teaching. • Sharing your findings with colleagues could also benefit them. • It could benefit you professionally. • Your findings may be publishable in any of a number of refereed journals: Journal of Geoscience Education, Journal of College Science Teaching, Science Education, Journal of Research in Science Teaching, International Journal of Science Education…even Science! • Your work could seed or enhance a grant proposal: NSF, NASA, NOAA, DOE science proposals typically require a Broader Impacts or Education & Outreach component. These agencies also offer many funding programs that specifically target science education (both research and practice). • It could even further the contributions of geoscience to the scholarship of teaching and learning….

3. Before we begin, consider these points…. • Human beings are what we study in teaching and learning research. • The theories that inform this research and the tools we use to carry it out come from the social and behavioral sciences. • Many of these ideas and methods may seem foreign to researchers who have been trained solely in the natural sciences. • Research involving students is thus human subjects research. • The well-being and protection of students should always be our foremost concerns (and are legal and ethical obligations…more on this later). • Shared interest in improving the quality and effectiveness of science education has led to unprecedented collegiality and collaboration among scholars of science and scholars of education! Bush, S. D., Pelaez, N. J., Rudd, J. A., Stevens, M. T., Tanner, K. D., & Williams, K. S. (2008). Science faculty with education specialties. Science, 322, 1795-1796.

4. How do I conduct research on student learning? • How do I devise my research project? • What kinds of data should I collect? • What methods and tools will I use to collect the data? • How will I analyze the data and interpret the results? • What permission will I need to do this? • How do I disseminate my findings? • What else should I consider in doing learning research? These discussions will be followed by a few illustrative examples of recent studies of geoscience learning.

5. How do I devise my research project? • Learning research should be guided by a question to answer, an observation to explain, a hypothesis to test. …How do I come up with one? • What problems have you or your colleagues encountered in teaching? • Have you found relevant prior work in the literature that you could refine or extend? • Has a funding agency issued a Request for Proposals with a problem they’d like to see addressed? • What interests you about student learning? • …Is my proposed project feasible? • What is the setting (a course, a section, an exercise, a field trip, etc.)? • How many students will be involved? What level of cooperation will be needed? • How much time will it require? (Not just for data collection, but analysis, writing, etc.) • Are there any potential ethical conflicts? • …What similar work has been done? • Review the geoscience-ed literature to avoid reinventing any wheels, to identify potential pitfalls, and to help you select the right methods and tools for your project. Trochim, W., & Donnelly, J. P. (2007). The research methods knowledge base (3rd ed.). Mason, OH: Atomic Dog Publishing. Online version at http://socialresearchmethods.net

6. 2. What kinds of data should I collect? • How do we define and characterize our students’ learning? • The three domains of learning Cognitive “thinking” Recall or recognition of knowledge; develop-ment of intellectual abilities and skills Affective “feeling” Changes in interest, attitudes, values, and motivation Psychomotor “manipulating” Honing of motor skills, manipulative abilities, athleticism Does a student really care about learning geology? Is a student comfortable working in the field? Does a student value knowledge relevant to environmental sustainability? What motivates a student to study hard and do well on an exam? How do physical ability (or disability) impact field learning? Do reflexive actions play a role in field geoscience? Can a student recall the definition of an igneous rock? Can a student summarize the theory of plate tectonics? How does a student account for differences in water quality? Can a student critically review a published research article? How well can a student map the geology of an unfamiliar area? Terra celebra Terra nova Terra incognita Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: Cognitive domain. New York: David McKay. Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1964). Taxonomy of educational objectives: Affective domain. New York: Longman.

7. 3. What methods and tools will I use to collect the data? Quantitative methods Data captured numerically and analyzed by mathematical and statistical methods Qualitative methods Descriptive data captured verbally, textually, or by means of observations; interpreted and coded “Mixed-methods” research combines both quantitative and qualitative methods Strongly Agree 1 Agree 2 Neutral 3 Disagree 4 Strongly Disagree 5 The Likert scale Quizzes, exams ConcepTests Content assessments Concept inventories Attitude surveys Course evaluations (quantitative) Interviews Focus groups Observation Questionnaires Student feedback (e.g., minute papers) Artifacts (e.g., assignments) Course evaluations (descriptive) To be most useful and effective, instruments must be both valid: a test tests what you expect it to test and reliable: a test works the same way every time it is used Validity and reliability are established by expert review and statistical tests.

8. 3. What methods and tools will I use to collect the data? Effects of course on subsequent coursework or career path Pre-course attitudes about subject Changes in content knowledge over course Time Student tracking Retention of content knowledge Preconceptions of content knowledge Changes in attitudes over course Course or activity ends Comparison of different courses or methods Ability to perform a task Duration Summative assessment Post-test Formative assessment Test peer learning Course or activity starts In-course Feedback on learning during a lesson Pre-test Front-end assessment • When do I collect the data? That depends on what you are looking for! • For example: Behaviors in class or field

9. 3. What methods and tools will I use to collect the data? • If you are just starting out, don’t start from scratch! • First try working with existing, well-used, valid and reliable methods and tools, which you can find in sources and references such as these: • Assessment Primer at SERC Starting Point: serc.carleton.edu/introgeo/assessment • Student Learning at SERC Cutting Edge: serc.carleton.edu/NAGTWorkshops/assess • Online Evaluation Resource Library: oerl.sri.com • Research Methods Knowledge Base: socialresearchmethods.net • Journal of Geoscience Education articles online: nagt.org/nagt/jge (Join NAGT and subscribe!) • Journal of College Science Teaching articles online: nsta.org/college (Join NSTA and subscribe!) • Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Thousand Oaks, CA: Sage Publications. • Angelo, T. A., & Cross, K. P. (1993). Classroom assessment techniques: A handbook for college teachers. San Francisco: Jossey-Bass. • Feig, A. P., & Stokes, A. (Eds.) (In press, expected fall 2010). Qualitative research in geoscience education: GSA Special Paper. Boulder, CO: Geological Society of America. • Whitmeyer, S. J., Mogk, D. W., & Pyle, E. J. (Eds.). (2009). Field geology education: Historical perspectives and modern approaches: GSA Special Paper 461. Boulder, CO: Geological Society of America. • Manduca, C. A., & Mogk, D. W. (Eds.). (2006). Earth and mind: How geologists think and learn about the Earth: GSA Special Paper 413. Boulder, CO: Geological Society of America.

10. 4. How will I analyze the data and interpret the results? • Quantitative Analysis • If you have designed your project well and collected data carefully, analysis should be straightforward. • Data preparation: includes data logging; checking data for accuracy; entering data into the computer; mathematically transforming data as needed; developing appropriate database. • Descriptive statistics: used to describe the basic features of the data in a study. Typically includes univariate analysis (distribution, mean, median, mode, std deviation), and correlation between and among variables (bivariate or multivariate). • Knowing how “well-behaved” your data are • makes the next step possible: • Inferential statistics: statistical methods • applied to investigate questions, models, and hypotheses. • Enables you to make inferences from your sample data. • Various tests for significance: t-test, Analysis of Variance (ANOVA), Analysis of Covariance (ANCOVA), • regression analysis, factor analysis, etc. User-friendly software (e.g., SPSS) is readily available. Statistical significance of differences and gains is the gold standard! Trochim, W., & Donnelly, J. P. (2007). The research methods knowledge base (3rd ed.). Mason, OH: Atomic Dog Publishing. Online version at http://socialresearchmethods.net

11. 4. How will I analyze the data and interpret the results? • Qualitative Analysis • Transcription • Data are typically analyzed in textual form. Audio- and video-recorded observations and interviews must first be transcribed. • Coding • Interpreting the data in order to organize them and identify patterns, themes, and relationships. The qualitative analyst reads the data and demarcates various segments (single words to entire sentences or phrases) according to meanings. Each data segment is labeled with a code that allows it to be categorized for further interpretation. • e.g., Coding of responses by Petrified Forest National Park visitors to the question: How do you think the landscape you see [here] came to look the way it does now?(Bueno Watts, 2007). • (F) Erosion of water and wind. • (M) 200 million years of time. • (F) You can see the rocks falling already still. • (M) I think it was a long process. You can see the layers. • (F) uhh… You can see the water. • (M) [C]learly the amount of sun and the lack of rain had a lot to do with it. • A priori coding: Coded segments are classified under predetermined categories. • Emergent theme coding: Categories “emerge” as the data are coded. • Typically, there is more than one coder, and they compare their interpretations to ensure reliability. Bueno Watts, N. (2007). Visitor preconceptions and meaning-making at Petrified Forest National Park. Master’s thesis, School of Earth and Space Exploration, Arizona State University.

12. 4. How will I analyze the data and interpret the results? • Qualitative Analysis (continued) • Coding • can also be done mechanically, using computer programs (e.g., nVivo) that identify and count words, phrases, or sentences. • Interpretation • When coding is complete, the nature and distribution of coded data are studied and interpreted to uncover meaningful patterns and relationships such as: • student attitudes about a subject before and after a class or an entire course • behaviors observed during a lecture versus behaviors observed during a hands-on lab • student preconceptions, misconceptions, and conceptions about subject matter • Note also that all coded qualitative data can be quantified, and analyzed mathematically and statistically as quantitative data (e.g., Chi, 1997). Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. Journal of the Learning Sciences, 6, 271-315.  Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Thousand Oaks, CA: Sage Publications. Trochim, W., & Donnelly, J. P. (2007). The research methods knowledge base (3rd ed.). Mason, OH: Atomic Dog Publishing. Online version at http://socialresearchmethods.net

13. 5. What permission do I need to do this? hhs.gov/ohrp • Any college or university that receives Federal research funding is legally required to maintain an Institutional Review Board (IRB) to approve and oversee all uses of humans as research subjects. • To work with students you will need IRB approval, even if your study is deemed exempt from subsequent IRB oversight. • The IRB will typically require: • All investigators to have human subjects research • certification (online; e.g., citiprogram.org); • Informed consent from and ensured • confidentiality of students; • No suggestion of coercion (e.g., grades • or recommendations); • The ability for students to opt out of participation • at any time.

14. 6. How do I disseminate my findings? • Presentations and abstracts: • GSA/NAGT Annual Meeting; GSA Sectional Meetings; AGU Fall and Spring meetings; AAPG Annual Meetings all host numerous geoscience-education sessions and activities. • National Association for Research in Science Teaching (NARST.org) and American Educational Research Association (AERA.net) hold interdisciplinary research conferences every Spring. • Peer-reviewed journal articles: • Journal of Geoscience Education (NAGT.org/nagt/JGE), published by NAGT • Journal of College Science Teaching (NSTA.org/college), published by National Science Teachers Association. NSTA also publishes other journals for secondary, middle, and elementary science education. • Journal of Research in Science Teaching (www3.interscience.wiley.com/journal/31817/home), published by NARST. • Science Education (www3.interscience.wiley.com/journal/32122/home); International Journal of Science Education (tandf.co.uk/journals/tf/09500693.html); Electronic Journal of Science Education (EJSE.southwestern.edu); the list goes on. • Peer-reviewed book chapters: • GSA has published two education-focused Special Papers, and a third is in press. • Academic publishers such as Springer publish numerous refereed volumes of contributed papers. • Community online resources: • Science Education Resource Center (SERC.carleton.edu)…this one you should already be very familiar with! • Digital Library for Earth System Education (DLESE.org)

15. 7. What else should I consider in doing learning research? • Make the purpose and process of your research clear to your students (through informed consent, but also by means of explanation). • Give students as much feedback as you can from your findings. • Focus your work on what interests you most. • Find out if there are institutional resources you can draw on. • Consider collaborating with faculty in your College of Education. • Consider collaborating with geoscientists and geoscience educators outside of your institution. • Consider funding graduate or undergraduate research assistants to work with you (in or out of your academic unit). • Be sure you know how your Department, School, or College views this research. • If you are tenure-track, determine if this research (including potential publications and grants) will help your tenure case. Cross, P. K., & Steadman, M. H. (1996). Classroom research: Implementing the scholarship of teaching. San Francisco: Jossey-Bass.

16. Examples of methods and tools applied in research studies 3 (26% , 0%) 4 (19%, 18%) 5 (44%, 75%) 6 (11%, 7%) Initial Responses responses following discussion • Example: Geoscience ConcepTests (Quantitative; Cognitive domain) • ConcepTests are short multiple-choice conceptual questions designed for rapid quantitative assessment of student understanding (Mazur 1997). • Examine the map and answer the question: How many plates are present? When physical models were used in teaching correct responses increased to (56%, 84%) (McConnell et al. 2006) Geoscience ConcepTest bank at SERC Cutting Edge: serc.carleton.edu/introgeo/interactive/conctest.html Mazur, E. (1997). Peer instruction: A user’s manual. Upper Saddle River, NJ: Prentice-Hall. McConnell, D. A., Steer, D. N., Owens, K. D., Knott, J. R., et al. (2006). Using conceptests to assess and improve student conceptual understanding in introductory geoscience courses. Journal of Geoscience Education, 54, 61-68.

17. Examples of methods and tools applied in research studies • Example: Geoscience Concept Inventory (GCI) • (Quantitative; Cognitive domain) • The GCI(Libarkin & Anderson, 2005) is a valid and reliable multiple-choice instrument widely used to assess undergraduate-level geoscience content knowledge, and compare student learning across classes, institutions, regions, etc. • The user designs a 15-question subtest from items in an online test bank. • Geoscience educators are • also invited to create new • items for the test bank: • GCI WebCenter • gci.lite.msu.edu Libarkin, J. C., & Anderson, S. W. (2005). Assessment of learning in entry-level geoscience courses: Results from the Geoscience Concept Inventory. Journal of Geoscience Education, 53, 394-401.

18. Examples of methods and tools applied in research studies • Example: Geoscience Concept Inventory (GCI) • e.g., GCI used to measure geoscience content knowledge of 63 undergraduate participants in an entirely field-based introductory geology course(Elkins & Elkins, 2007). • Students demonstrated statistically • significant improvements in geoscience • content knowledge. • Students showed significantly greater • improvement in knowledge as compared • to students from other introductory geology • courses across the USA. • Findings support the efficacy of geoscience • teaching and learning in the field. Elkins, J. T., & Elkins, N. M. L. (2007). Teaching geology in the field: Significant geoscience concept gains in entirely field-based introductory geology courses. Journal of Geoscience Education, 55, 126-132.

19. Examples of methods and tools applied in research studies • Example: Motivated Strategies for Learning Questionnaire (MSLQ) (Quantitative; Affective and cognitive domains) • The MSLQ(Pintrich et al., 1991) assesses student motivation and strategies for learning. It is an 81-item instrument with six subcategories and no right or wrong answers (students respond using a Likert scale). • It is valid, reliable, and used in many different educational settings. • e.g., GARNET Project(McConnell et al., 2009): • Researchers administered the MSLQ to 340 physical -geology students in diverse colleges and universities. • No major differences in students’ initial motivation and learning strategies among different institutions were found. • Over the course of the semester, students lost test anxiety but also become less academically self-confident, and more likely to seek help from instructors or peers. McConnell, D., Jones, M. H., Budd, D. A., Bykerk-Kauffman, A., Gilbert, L. A., Knight, K., Kraft, K. J., Nyman, M., Stempien, J. A., Vislova T., & Wirth, K. R. (2009). Baseline data on motivation and learning strategies of students in physical geology courses at multiple institutions: GARNET Part 1, Overview [Abstract]. Geological Society of America Abstracts with Programs, 41, 603.

20. Examples of methods and tools applied in research studies • Example: Questionnaire for Semi-Structured Interviews(Qualitative; Affective and cognitive domains) • “Semi-structured” means that while the respondent is prompted with a set of questions, he or she is free to extend the discussion in any direction. • This questionnaire was used to query students • (mostly K-12 teachers) in aplace-based Earth science course • (Williams & Semken, in press). • Items elicited cognitive and affective responses to curriculum, • pedagogy, and student’s sense of the places studied. • Interview transcripts were analyzed to uncover concepts, patterns, • and relationships that were linked into thematic categories. • Positive responses to the place-based approach were reported by • most students: enhanced place attachment and meaning, enhanced • science comprehension, and enhanced teaching. Williams, D., & Semken, S. (in press). Ethnographic methods in analysis of place-based geoscience curriculum and pedagogy. In A. P. Feig & A. Stokes (Eds.), Qualitative research in geoscience education: Geological Society of America Special Paper.

21. Examples of methods and tools applied in research studies • Example: Script for “Think-Aloud” Experiment(Qualitative; Cognitive domain) • “Semi-structured” means that while the respondent is prompted with a set of questions, he or she is free to extend the discussion in any direction. • Off-site study of the Trail of Time(Semken et al., 2009): a 4.5-km walking timeline trail and geological interpretative exhibition at Grand Canyon National Park (Karlstrom et al., 2008). • A scaled simulation was used to investigate whether visitors • could understand and navigate a geologic timeline trail in the Park. • Specific tasks were posed to respondents as they walked. • They were asked to “think aloud” as they performed the tasks. • Findings showed that visitors can make sense of the time scales • and will be able to understand and learn from the actual Trail of Time • at Grand Canyon. • Karlstrom, K., Semken, S., Crossey, L., Perry, D., Gyllenhaal, E. D., Dodick, J., Williams, M., Hellmich-Bryan, J., Crow, R., Bueno Watts, N., • & Ault, C. (2008). Informal geoscience education on a grand scale: the Trail of Time exhibition at Grand Canyon. Journal of Geoscience • Education, 56, 354-361. • Semken, S., Dodick, J., Ben-David, O., Pineda, M., Bueno Watts, N., & Karlstrom, K. (2009). Timeline and time scale cognition • experiments for a geological interpretative exhibit at Grand Canyon. Proceedings of the National Association for Research in • Science Teaching, Garden Grove, California.

22. Examples of methods and tools applied in research studies • Example: Concept Sketches(Qualitative; Cognitive and affective domains) • A concept sketch(Johnson & Reynolds, 2005) is a student- • or instructor-generated sketch intended to illustrate • the important or meaningful aspects of a concept or system. • Student-generated concept sketches reveal level of • understanding of complex concepts, and can be coded • as artifacts of learning. Johnson, J. K., & Reynolds, S. J. (2005). Concept sketches—Using student- and instructor-generated, annotated sketches for learning, teaching, and assessment in geology courses. Journal of Geoscience Education, 53, 85-95.