Conducting Classroom Research in Statistics Education: Issues, Challenges and Examples

1. Conducting Classroom Research in Statistics Education: Issues, Challenges and Examples Andrew Zieffler Ph.D. University of Minnesota University of Minnesota Educational Psychology

2. Statistics Education Research: A Diverse Discipline or a Many Headed Hydra? • Interdisciplinary field of inquiry • Not reliant on any one tradition of empirical research methodology • Variety of research questions, methodologies, operational definitions, outcome variables studied, findings

3. Statistics Education Research: Goals • Improving instruction should be the key goal in any educational research (Raudenbush, 2005) • Therefore, the goal of statistics education research should be the improvement of teaching statistics, leading to improved student learning.

4. Statistics Education Research: Goals • The Research Advisory Board of the Consortium for the Advancement of Undergraduate Statistics Education (CAUSE) (http://www.causeweb.org/research/). • Designed so that the results will have direct implications for instruction • Research studies in this area should specifically address classroom implications and the generation of new research questions

5. Statistics Education Research: Improvement? • How can we improve the future statistics education research drawing on what is available now? • Four Suggestions • Based on Review of research on teaching and learning statistics at the college level • Higher quality research questions • Thorough literature reviews • Paying attention to Measurement • Consideration of different methodologies

6. Formulating a Problem: Developing a Quality Research Question • Narrow down the focus of the research question (Garfield, 2006) • Many of the studies reviewed examined broad research questions • For example, • ‘Does technology improve student learning?’ versus • ‘How does a particular technology help students to understand a particular statistical concept?’

7. Formulating a Problem: Developing a Quality Research Question • “The bottom line for judging research is, does it advance the current knowledge in the field in a significant way (Simon, 2004, p. 158)?” (Field can refer to practitioners or researchers) • How will the study contribute/add to the existing literature? • Relate it to the teaching and learning of statistics • Helps us meet the goal of improving instruction

8. Relevant Information: The Importance of a Thorough Literature Review • Role of the literature review - provide a critical review, analysis and synthesis of the literature relevant to the particular topic being studied • How the literature reviewed is relevant to the research question being examined. • Helps contextualize the research within the field (Identifies gaps in previous research, etc.) • Builds on the work of others

9. Relevant Information: The Importance of a Thorough Literature Review • As an interdisciplinary area of research, statistics education researchers need to reflect that in their evaluation of the prior research. • Research appears in journals from many disciplines (Teaching Psychology, JSE, JRME, American Statistician, etc.) • Read and Review the Literature: Be Exhaustive • CAUSE WEB • Statistics Education Journals • Journals in other disciplines • Google Scholar

10. Importance of Measurement: Where Good Intentions Go Wrong • Measurement refers to “the process of quantifying observations [or descriptions] about a quality or attribute of a thing or person (Thorndike & Hagen, 1986, p.5).” • Measurements used are essential to the findings that are produced • Measurements need to be valid and reliable (Pedhazur & Schmelkin, 1991).

11. Importance of Measurement: Where Good Intentions Go Wrong • Descriptions of development of the measurements/evidence of their meaningfulness and appropriateness essential elements in the reporting of research

12. Importance of Measurement: Where Good Intentions Go Wrong • For example, in many of the studies, • Students’ statistical knowledge or reasoning was translated into a degree of quantification by the assignment of a test score to each student. These scores were then generally subjected to some kind of quantitative analysis.

13. Importance of Measurement: Where Good Intentions Go Wrong • Measurements were typically course specific student outcomes, (e.g., final exam grades, course evaluations) • Assessments using instructor constructed items often have less desirable psychometric properties (e.g., Gullickson & Ellwein, 1985; Weller, 2001) • Measurements often have dependence to a particular course • Lack of external validity • Difficult to understand the learning outcomes due to omission of the assessment items that were used by the researcher • Were the students tested on computational and procedural skills, or on higher levels of thinking and reasoning?

14. Importance of Measurement: Where Good Intentions Go Wrong • Recommend the use of research instruments such as CAOS (see ARTIST; https://app.gen.umn.edu/artist/index.html) • Careful development and validation of research created instruments

15. Methodology: The “Gold Standard” is not always the Gold Standard • Imagine comparison of “traditional” course to “reform” course with students randomly assigned to each • Even if it seems experimental, this is NOT the gold standard • Still potential problems • Operational definitions – What is a “traditional” course? • External Validity/Generalizability? • Issues of Fidelity • Individual Differences (teachers, classes, etc.)

16. Methodology: The “Gold Standard” is not always the Gold Standard • For instance, it may be better to compare • Two different sets of activities to develop an student reasoning/understanding of a particular topic (e.g., sampling distribution) • Two different sequences of topics across many sections of the same class. • “Classical experimental method can be problematic in education (Schoenfeld, 2000, p. 645).”

17. Methodology: Analysis • “Good research is a matter not of finding the one best method, but of carefully framing that question most important to the investigator and the field and then identifying a disciplined way in which to inquire into it that will enlighten both the scholar and his or her community (Schulman, 1997, p. 4).”

18. Methodology: Analysis • Methodology needs to be responsive to purposes/contexts of research (Howe & Eisenhart, 1990) • Alternatives to controlled experiments • Classroom-Based Research • Teaching Experiments • Naturalistic Observation • Videotaped Interviews

19. Classroom Research in Statistics Education: Some Advice • Plan, Plan, Plan • Research Question • Study Design/Methodology • Assessment/Measurement • IRB • Pitfalls that may arise

20. Classroom Research in Statistics Education: Some Advice • Form collaborative research groups (ASA, 2007) • Teachers of statistics • Faculty from other disciplines (e.g., psychology or education). • See Garfield and Ben-Zvi (in press) for more arguments and suggestions for this type of research.

21. Classroom Research in Statistics Education: Some Advice • Consult other experts/Collaborate • “Look for collaborators who share your research interests but who may bring different background (even disciplines) and strengths to a new collaboration (Garfield, 2006, p. 8).” • Removes the pressure of having to be an expert in everything

22. References • American Statistical Association. (2007), “Using Statistics Effectively in Mathematics Education Research,” Retrieved Feb. 14, 2007, from ASA Web site: http://www.amstat.org/research_grants/pdfs/SMERReport.pdf. • Garfield, J. B. (2006), “Collaboration in Statistics Education Research: Stories, Reflections, and Lessons Learned,” in Proceedings of the Seventh International Conference on Teaching Statistics, eds. A. Rossman and B. Chance, Salvador, Bahia, Brazil: International Statistical Institute, pp. 1-11. • Garfield, J. and Ben-Zvi, D. (in press), “Developing Students’ Statistical Reasoning: Connecting Research and Teaching Practice,” Emeryville, CA: Key College Press.

23. References • Gullickson, A. R., and Ellwein, M. C. (1985), “Teacher-Made Tests: The Goodness-of-Fit Between Prescription and Practice,” Educational Measurement: Issues and Practice, 4(1),15-18. • Howe, K., & Eisenhart, M. (1990), “Standards for Qualitative (and Quantitative) Research: A Prolegomenon,” Educational Researcher, 19, 2-9. • Pedhazur, E. J., and Schmelkin, L. P. (1991), “Measurement, Design, and Analysis: An Integrated Approach,” Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers.

24. References • Raudenbush, S. W. (2005), “Learning from Attempts to Improve Schooling: The Contribution of Methodological Diversity,” Educational Researcher, 34(5), 25-31. • Schoenfeld, A. H. (2000). “Purposes and Methods of Research in Mathematics Education,” Notices of the AMS, 47(6), 641-649. • Schulman, L. S. (1997). “Disciplines of Inquiry in Education: A New Overview.” in Complementary Methods for Research in Education, ed. R. M. Jaeger, Washington DC: American Educational Research Association, pp. 3-29.

25. References • Simon, M. A. (2004). “Raising Issues of Quality in Mathematics Education Research,” Journal for Research in Mathematics Education, 35(3), 157-163. • Thorndike, R. L., and Hagen, E. (1986), “Cognitive Abilities Test: Examiner's Manual Form 4,” Chicago, IL: Riverside. • Weller, L. D. Jr. (2001), “Building Validity and Reliability into Classroom Tests,” National Association of Secondary School Principals, NASSP Bulletin [Online], February.

26. Contact Information Andrew Zieffler, Ph.D. University of Minnesota zief0002@umn.edu