Teaching Basic Skills Mathematics: Outcome Assessment

1. Teaching Basic Skills Mathematics: Outcome Assessment Xi Zhang, Campus Based Researcher, City College Jenny Kimm, Associate Professor, City College San Diego Community College District Presented at the: 2009 Strengthening Student Success Conference San Francisco, CA: October 7, 2008

2. Introduction of the District San Diego Community College District • 2nd largest district in the state • Three 2-year colleges and eleven Continuing Education campuses • Serves approx. 100,000 students each semester

3. Introduction of the San Diego City College • San Diego City College is the first established community college in San Diego. • San Diego City College is a public, two-year community college administered by the San Diego Community College District. • Serving as the educational cornerstone of downtown San Diego, the college offers more than 100 majors, 100 certificate programs and 1,500 classes each semester to 16,000 students.

4. Purpose of the Presentation • The primary purpose of this presentation is to share the research methodology and innovative techniques for data analysis and instrument refinement for SLO assessment rather than to disseminate results of the study. • A second purpose is to support the teaching of Basic Skills Math in community colleges.

5. Purpose of the Study • Demonstrate SLO assessment cycle in the Math Department at San Diego City College • Draw attention to measurement issues in assessing SLOs. • Demonstrate methods of item analysis that has multiple advantages.

6. Research Design • SLO assessment cycle • A repeated measure design: pre and post design

7. SLO Assessment in the Math Department City College Student Learning Outcomes Assessment Cycle 6 Column Form For Developmental Math Program- Math 35, 95, 96 Year: 2008/2009

8. Research Methodology • Rasch Model • Estimate both item difficulty and person ability • Map both parameters on the same scale • Inform instrument refinement • Paired Sample t-test • Statistically significant improvement from pretest to posttest

9. Target Population and Sample Size • Developmental math courses: Pre-Algebra, Beginning Algebra, and Intermediate Algebra. • This totals to over 1000 students taking the developmental math courses • In Fall 2008, we collected pre/post paired data from about 250 students

10. Data Collection: Instrument • Developed one pre test and a similar post test • TestGen testbank software • 8 multiple choice questions • Topics of the questions • A Sample Instrument

11. Data Collection: Test Administration • Pre-test administration • Pre-test grading • Post-test administration • Post-test grading

12. Data Management • For each student, itemized response per question and a total score were entered and paired in Excel • Data then were exported to Winsteps for model fitting • Estimates of item difficulty and student ability were analyzed in SPSS to compare pre test results to the post.

13. Data Analysis • Fit itemized responses rather than the total scores with the Rasch Model to obtain estimates of item difficulty and student ability.

14. Data Analysis • Map both item difficulties and person abilities on the same scale to produce the Item-Person Map. • Compare Pre test and Post test • Paired sample t-test • Anchoring item difficulties

15. Measures • Item difficulty • Student ability

16. Results (Pre-algebra FALL 2008 DATA) • Pre-test • Item difficulty • Student ability

17. Results (Pre-algebra FALL 2008 DATA) • Pre-test • Person-item map

18. Results (Pre-algebra FALL 2008 DATA) • Post-test • Item difficulty • Student ability

19. Results (Pre-algebra FALL 2008 DATA) • Post-test • Person-item map

20. Results (Pre-algebra FALL 2008 DATA) • Pre and post comparison • Anchoring item difficulties to produce a new set of student ability estimates

21. Results (Pre-algebra FALL 2008 DATA) • Pre and post comparison • Anchoring item difficulties to produce a new set of student ability estimates

22. Results (Pre-algebra FALL 2008 DATA) • Pre and post comparison • Paired sample t test

23. Findings • Results revealed that students scored statistically significantly higher in the post test compared to their performance in the pretest. • Content areas that the instructors need to emphasize for teaching. • Information for instrument refinement.

24. INSTRUMENT REFINEMENT

25. Use of Results for Programmatic Improvement • Imbed the post test into the final exam to increase sample size. Also provide online version of pre test to collect data from online developmental classes. • Rewrite or revise test questions based on results of item analysis. • Identify difficult topics and disseminate the information to developmental math instructors for future teaching. • Also disseminate the information to instructors of higher level math course for their preparation and planning.

26. Discussion • Advantages of item analysis • Solve measurement issues • Conduct meaningful comparisons • Bank good test items for constructing future tests • Diagnostic function provides insight of the strength and weakness of student content knowledge.

27. Limitations of the Research • Small sample size • Small number of test items • Item stability • Generalizability of the results

28. Questions?