Gathering SoTL Evidence: Methods for Systematic Inquiry into Student Learning

1. Gathering SoTL Evidence: Methods for Systematic Inquiry into Student Learning Renee A. Meyers Coordinator, UWS SoTL Leadership Site Faculty College, May 2007

2. Introduction • Let’s gather evidence about our class • In groups of 4-5, develop 4-5 survey questions that you would like to know about your classmates as it relates to this class • Structure your questions in such a way that they can be answered with a show of hands • Let’s use your surveys to gather evidence!

3. Introduction • To gather more in-depth evidence, I would like each of you in turn to answer the following interview question (in 1-2 minutes or less): • Do you currently have a research question on teaching/learning that you are hoping to answer? If so, what is it? If not, why are you taking this class? • What question would you most like to have answered by the end of this class?

4. Overview of Session • Orientation to SoTL Research • Methodological Orientations • Types of Questions • Forms of Evidence • Tools for, or Methods of, Gathering Evidence • Working on your SoTL project

5. Overview First Day • Orientation to SoTL Research • Methodological Orientations • Types of Questions • Forms of Evidence

6. Orientation to SoTL Research

7. Orientation to SoTL Research • Interpretive Orientation • Quasi-Objective Orientation • What are research goals of each? • How is human behavior viewed? • What types of methods typically characterize each orientation? • What are strengths/difficulties of each orientation? • Where do you stand? Is there a disciplinary stance?

8. Orientation

9. Orientation to SoTL Methods

10. Orientation to SoTL Methods • Qualitative Methods • Example of a SoTL study using qualitative methods? • Quantitative Methods • Example of a SoTL study using quantitative methods? • Example articles are available in your packets

11. Qualitative Methods • Qualitative methods are used to “understand” more deeply. Typically results do not seek to generalize widely. • Some of the most frequently used types of qualitative methods in SoTL research are: • focus groups/interviews • text or discourse analysis • protocol analysis • observational work

12. Thinking it through . . . • Suppose you wondered how students use instructor feedback to write the second draft of a paper. How might you design a qualitative investigation to address this question?

13. Quantitative Methods • In contrast, quantitative research typically seeks to generalize to a large population. It is “hypothetically” more objective and less interpretive. • Some of the most frequently utilized types of quantitative methods in SoTL research include: • Surveys • Experimental Design • Content analysis

14. Thinking it through . . . • Suppose you wondered how students use instructor feedback from the first draft of their papers to write the second draft. How might you design a quantitative investigation to address this question?

15. Qualitative or Quantitative Methods • Why does a researcher use quantitative methods? • Why does a researcher use qualitative methods? • How does one decide which form of method to use?

16. Method Needs to Fit Your Question • Some quantitative researchers argue strongly that qualitative research is prone to biased interpretation and therefore is inherently flawed • In response, qualitative researchers argue that qualitative methods often yield insights that quantitative research can’t expose. They also argue that quantitative research is not nearly as unbiased as its advocates suggest. • In reality, both qualitative and quantitative research have a beneficial role to play in SoTL research.

17. Types of SoTL Questions

18. Types of Questions • Exploratory questions (What if) • Learning to Lecture article in packet • Descriptive questions (What is) • Lessons From the Best article in your packet • Confirmatory questions (What works) • Practice Versus Review Exams article in your packet • Time Orientation • Cross-sectional research • Longitudinal research

19. Caution • The boundaries between various types of questions are often blurred and not very precise. • For instance, a researcher might say that she is conducting “an exploratory, descriptive study.”

20. Exploratory Questions • Typically exploratory research is conducted to gain a very rough understanding of issues, circumstances, situations, etc. • Rarely are definitive conclusions reached in exploratory research. Instead, exploratory research usually helps to define the specific focus of subsequent research. • Quite often qualitative research methods are used in exploratory research (but not always).

21. Example • Can you provide an example of a SoTL research investigation that would be considered exploratory in nature? • What if?

22. Descriptive Type Questions • The basic goal of descriptive research is to describe what is happening in the classroom, with one’s students, with teaching and learning, etc. • Descriptive research can use either quantitative or qualitative methods

23. Example • Can you give an example of a descriptive SoTL study that would use: • Qualitative methods • Quantitative methods • What is?

24. Confirmatory Questions • Confirmatory research attempts to provide explanations for relationships between variables • Usually this involves quantitative information and more rigorous statistical analysis • This type of research is often where SoTL researchers think they should start • However, often exploratory and descriptive work must be done before relationships can be confirmed

25. Example • Can you provide an example of a SoTL investigation that examines a question interested in determining a causal or confirmatory relationship? • What works?

26. Time Orientation in Research

27. Time Orientation and Research Questions • All research takes place within time and space • Researcher needs to decide whether question being asked is best investigated as a one-time phenomenon, or over time (longitudinally) • Cross sectional research • Longitudinal research

28. Cross-sectional Research • This type of research gives you a one-time “snap shot” of the situation. • This allows you to explore, describe, or test a single teaching-learning episode (for example) • However, you can’t answer questions regarding trends (i.e., “Is this the same as 5 years ago or last semester?”)

29. Longitudinal Research • Longitudinal research is conducted over multiple time periods • Instructors are often interested in trends over time or semesters • Researchers must be very careful to ensure that the research methods used from period to period are consistent

30. Example • Suppose you wanted to study whether group members use “explaining” communication in their group discussions. • If you were to do a cross-sectional study, how would you proceed? • If you were to do a longitudinal study, how would you proceed? • What are the advantages/disadvantages of undertaking each of these time orientations in SoTL research?

31. Types of Evidence

32. Types of Evidence • Secondary data • Primary data

33. Secondary Data • Secondary data are data collected for some purpose or need other than the one being currently pursued by a SoTL researcher. • Secondary data can be found within your own university and also within external organizations.

34. Primary Data • Primary data are collected to specifically answer the questions posed in the current SoTL research project. • Primary data are collected by you in your own classroom, or others’ classrooms • More about primary data later; for now a bit about secondary data

35. Benefits of Secondary Data • There are several reasons why a SoTL researcher might rely on the use of secondary data: • It usually takes less time to access secondary data than collect primary data. • Sometimes the quality of secondary data exceeds the quality of the primary data you can collect • Sometimes the data is only available in secondary form

36. Drawbacks to Secondary Data • The disadvantages of secondary data include: • Secondary data may not specifically apply to the questions being addressed • Secondary data may be out of date and no longer relevant

37. Sources of SoTL Secondary Data • National Center for Education Statistics • UWS Office of Policy and Analysis Research • The Chronicle of Higher Education • U.S. Department of Education • NSSE and FSSE data • Campus Institutional Research

38. Using Secondary Data • How might you use secondary data in a SoTL investigation?

39. Credibility of Secondary Data • Whenever you use secondary data, you must consider how credible it is • Questions to ask: • Does the report reveal who sponsored the research? • Does the report reveal who conducted the research? Is this a respected source? • Are the research methods disclosed and explained? (Size of sample? Characteristics of respondents? Margin of error?) • When was the study conducted?

40. Summary • Think about your orientation (and your disciplinary orientation) to doing research • Think about your comfort level with qualitative and quantitative methods • Think about your question • Think about the forms of evidence/data you might want to access or collect