Measuring confidence to teach statistics to middle & high school grades: The development & validation of

1. Measuring confidence to teach statistics to middle & high school grades:The development & validation of the SETS instruments

2. The Research Team Supported in part by CAUSE (under NSF DUE #0618790) Leigh M. Harrell-Williams M. Alejandra Sorto Virginia Tech Texas State Georgia State University University Rebecca L. Pierce Lawrence M. Lesser Teri J. Murphy Ball State University The University of Texas Northern Kentucky at El Paso University

3. Why measure middle and high school grades pre-service teachers’ Self-Efficacy to Teach Statistics? Teachers who are prepared to teach mathematics are expected to teach statistics as outlined by the PreK-12 Guidelines for Assessment and Instruction in Statistics Education (GAISE) (Franklin et al., 2007) and the Common Core State Standards for Mathematics (CCSSM) (National Governors’ Association, 2010). Self-efficacy is task specific;any instrument that assesses self-efficacy needs to be task specific as well.

5. GAISE PreK-12 Curriculum Framework Process Components Formulate Question Collect Data Analyze Data Interpret Results “Although these three levels may parallel grade levels, they are based on development in statistical literacy, not age. Thus, a middle-school student who has had no prior experience with statistics will need to begin with Level A concepts and activities before moving to Level B.” (p. 13)

6. CCSSM Framework VERSUS Grade-Specific Standards for Each Grade Level Topic-Specific Standards Similar to the 4 processes in the GAISE, there are 8 mathematical practices that are threaded throughout the CCSSM.

7. Development Process for Middle Grades SETS Instrument Spring/Summer 2008

8. Development Process for Middle Grades SETS Instrument

9. The Middle Grades SETS Instrument 26 Likert scale items in this format: Please rate your confidence in teaching middle grades students the skills necessary to complete the following tasks successfully: Scale of 1 to 6 14 Demographic items

10. Selected SETS Items based on GAISE Level-A 12 items Collect data to answer a posed statistical question in contexts of interest to middle school students. Recognize that there will be natural variability between observations for individuals. Select appropriate graphical displays and numerical summariesto compare individuals to each other and an individual to a group. Statistics Standards in the Common Core, Grades 2 - 6 Grades 2 - 5: Represent and interpret data Collecting measurements and creating line plot and bar graphs Grade 6: Develop understanding of statistical variability: • Recognize statistical question • Data has distribution with specific center/variation/shape Summarize and describe distributions • Create boxplots/histograms • Summarize data numerically

11. Selected SETS Items based on GAISE Level-B 15 items Recognize the role of sampling error when making conclusions based on a random sample taken from a population. Recognize that a sample may or may not be representative of a larger population. Recognize sampling variability in summary statistics such as the sample mean and the sample proportion. Use interquartile range, five-number summaries, and boxplots for comparing distributions. Interpret measures of association. Statistics Standards in the Common Core, Grades 7 - 8 Grades 7: Use random sampling to draw inferences about a population. • Inferences can be made from sample about population if sample is representative. • Generate multiple samples to gauge accuracy. Draw informal comparative inferences about two populations. • Use graphs to estimate differences. • Use numerical values to assess differences. Grade 8: Investigate patterns of association in bivariate data. • Construct and interpret scatterplots. • Informally fit, assess and interpret a linear relationship. • Use contingency table to evaluate relationship.

12. 2010 – 2011 Validation Study • Four US public institutions of higher education with significant proportion of students pursuing degrees in education • 309 participants enrolled in either an intro statistics course or a math education course

13. Validation Study - Methods • Confirmatory Factor Analysis • Item Analysis • Rating Scale Analysis • Reliability

14. Validation Study - CFA • Compared unidimensional and two-dimensional factor structure using Multidimensional Random Coefficient Multinomial Logit Model as implemented in Conquest software • Two dimensions: (Friel, Curcio, & Bright, 2001) • Efficacy to Teach “Reading the Data” • Efficacy to Teach “Reading Between the Data” • AIC and BIC confirmed two dimension structure • 0.85 between-dimension correlation

15. Validation Study – Item Analysis

16. Validation Study – Rating Scale Analysis

17. Validation Study - Reliability Reliability of Separation - Analogous to Cronbach’s Alpha • Composite Score: 0.94 Subscales: • “Reading the Data” (Level A): 0.87 • “Reading Between the Data” (Level B): 0.91

18. The High School Grades SETS Instrument • Items completed Spring 2012 • Based on both GAISE (all levels) and two of the four strands for CCSSM for High School Statistics & Probability • Interpreting Categorical & Quantitative Data • Making Inferences & Justifying Conclusions • Data collection • In-service: Summer 2012 • Pre-service: Fall 2012 & Spring 2013 • Analysis during Spring/Summer 2013

19. The High School Grades SETS Instrument 44 Total items: • 26 items from Middle Grades SETS instrument • 18 “new” items based on level C of GAISE and the two strands of the CCSSM Item format: • Please rate your confidence in teaching high school students the skills necessary to complete the following tasks successfully: • Scale of 1 to 6 14 Demographic items

20. Selected SETS High School Items Based on GAISE Level C and CCSSM Evaluate whether a specified model is consistent with data generated from a simulation. Explain the role of randomization in surveys, experiments and observational studies. Estimate a population mean or proportion using data from a sample survey. Evaluate how well the conclusions of a study are supported by the study design and the data collected. CCSSM “Making Inferences & Justifying Conclusions” Strand Understand and evaluate random processes underlying statistical experiments • S-IC.1. Understand statistics as a process for making inferences about population parameters based on a random sample from that population. • S-IC.2. Decide if a specified model is consistent with results from a given data-generating process, e.g., using simulation. Make inferences and justify conclusions from sample surveys, experiments, and observational studies • S-IC.3. Recognize the purposes of and differences among sample surveys, experiments, and observational studies; explain how randomization relates to each. • S-IC.4. Use data from a sample survey to estimate a population mean or proportion; develop a margin of error through the use of simulation models for random sampling. • S-IC.5. Use data from a randomized experiment to compare two treatments; use simulations to decide if differences between parameters are significant. • S-IC.6. Evaluate reports based on data.

21. Potential Uses • Research • Assessment of Teacher Preparation Programs • Analysis of Need for In-Service Professional Development Programs

22. Current Projects Using SETS Instruments Jean Linner Lassiter High School & GaDOE/GCTM Academy “Recognizing that low teacher efficacy can inhibit effective teaching as well as student learning, we hope to use the SETS instrument for high school in a longitudinal study to identify and target those professional learning experiences that increase teacher efficacy for teaching statistics.”

23. Current Projects (continued) Stephanie Casey & Andrew Ross Eastern Michigan University Scores from the SETS for pre-service secondary mathematics teachers enrolled in a statistical methods course were used to assess the pre-service teachers' confidence in teaching statistical concepts in general. Additionally, scores were used to compare the treatment (reform-oriented instruction) section and the control (traditional instruction) section.  The results will be used to inform the instruction of the course and assess whether there was a significant difference between the treatment and control groups with respect to their self-efficacy for teaching statistics.

24. Current Projects (continued) Salome Martínez & Eugenio Chandía Universidad de Chile & Pontificia Universidad Católica de Chile The government of Chile is funding a curriculum project for the preparation of future elementary teachers. The curriculum consists of a series of textbooks for all the content areas, including the Data and Chance strand of the national standards. The SETS instrument will be used to evaluate the impact of the implementation of the textbooks in 10 Chilean institutions of teacher preparation.

25. Information on Using SETS Instruments E-mail Rebecca Pierce at rpierce@bsu.edu to request “Terms of Use” form

26. Thank you for attending our webinar! We’d like to open the floor to discuss the following: • How would you envision using one of the SETS instruments? • What else would you like to know about the SETS instruments?

27. Related Papers • Hilton, S., Kaplan, J., Hooks, T., Harrell, L. M., Fisher, D., & Sorto, M. A. (2008). Collaborative projects in statistics education. In Proceedings of the 2008 Joint Statistical Meetings, Section on Statistical Education (pp. 752-756). Alexandria, VA: American Statistical Association. • Harrell, L. M., Pierce, R. L., Sorto, M. A., Murphy, T. J., Lesser, L. M., & Enders, F. B. (2009). On the importance and measurement of pre-service teachers’ efficacy to teach statistics: Results and lessons learned from the development and testing of a GAISE-based instrument. In Proceedings of the 2009 Joint Statistical Meetings, Section on Statistical Education (pp. 3396-3403). Alexandria, VA: American Statistical Association. • Sorto, M. A., Harrell, L. M., Pierce, R. L., Murphy, T. J., Enders, F. B., & Lesser, L. M., (2010). Experts’ perceptions in linking GAISE guidelines to the self-efficacy to teach statistics instrument. In Proceedings of the 2010 Joint Statistical Meetings, Section on Statistical Education (pp. 4289-4294). Alexandria, VA: American Statistical Association. • Harrell-Williams, L. M., Sorto, M. A., Pierce, R. L., Lesser, L. M., & Murphy, T. J. (Under Review). Validation of Scores from a New Measure of Pre-service Teachers’ Self-efficacy to Teach Statistics in the Middle Grades.

28. REFERENCES • Czerniak, C. M. (1990). A study of self-efficacy, anxiety, and science knowledge in preservice elementary teachers. Paper presented at the National Association for Research in Science Teaching, Atlanta, GA. • Enochs, L.G., Smith, P.L., Huinker, D. (2000). Establishing factorial validity of the mathematics teaching efficacy beliefs instrument. School Science and Mathematics, 100, 194-202. • Finney, S. J., & Schraw, G. (2003). Self-efficacy beliefs in college statistics courses. Contemporary Educational Psychology, 28, 161-186. • Franklin, C., Kader, G., Mewborn, D., Moreno, J., Peck, R., Perry, M., & Scheaffer, R. (2007). Guidelines for Assessment and Instruction in Statistics Education (GAISE) Report: A Pre-K-12 Curriculum Framework. Alexandria, VA: American Statistical Association. (Also available at http://www.amstat.org/education/gaise/) • Friel, S. N., Curcio, F. R., & Bright, G. W. (2001). Making sense of graphs: Critical factors influencing comprehension and instructional implications. Journal for Research in Mathematics Education, 32(2), 124-158.

29. REFERENCES (continued) • Lutzer, D., Rodi, S., Kirkman, E., & Maxwell, J. Statistical Abstract of Undergraduate Programs in the Mathematical Sciences in the United States, Fall 2005, CBMS Survey, American Mathematical Society, Providence, R.I., 2007. • National Governors Association (2010). Common Core State Standards for Mathematics. http://www.corestandards.org/assets/CCSSI_Math%20Standards.pdf • Riggs, I. M., & Enochs, L. G. (1990). Toward the development of an elementary teacher's science teaching efficacy belief instrument. Science Education, 74(6), 625-637. • Watson, J. (2001). Profiling Teachers’ Competence and Confidence to Teach Particular Mathematics Topics: The Case of Chance and Data. Journal of Mathematics Teacher Education, 4, 305–337. • Wenta, R. G. (2000). Efficacy of preservice elementary mathematics teachers. Unpublished doctoral dissertation, Indiana University.