Two Levels of Course Assessment to Identify Student Learning

1. Two Levels of Course Assessment to Identify Student Learning Carol Lerch, Ph.D lerch@dwc.edu S 52 Friday 11:45 – 12:35

2. Background • Assess new format – College Algebra • Introduced fall 2000 • Previous course failure rate +40% • Looking at students in one specific major • Assess new format – College Algebra • Introduced fall 2000 • Previous course failure rate +40% • Looking at students in one specific major • Course description • Purpose of this course is to provide students with a proficiency in algebraic skills necessary for success in future coursework … • Assess new format – College Algebra • Introduced fall 2000 • Previous course failure rate +40% AMATYC 2007

3. Collect Data • Specifics Fall 2001 – Summer 2007 • Pre-2000 60% pass (anecdotal) • Fall average 85% • Spring average 64% • Summer 80% • Includes Withdraws AMATYC 2007

4. Analysis • Improvement over previous College Algebra course • Fall pass rates acceptable • Continuing to improve • Improvement over previous College Algebra course • Fall pass rates acceptable • Continuing to improve • Spring pass rates much lower • Needs analysis & new plan • Improvement over previous College Algebra course • Fall pass rates acceptable • Continuing to improve • Spring pass rates much lower • Needs analysis & new plan • Summer needs more data • Online first time summer 07 • Improvement over previous College Algebra course AMATYC 2007

5. Changes to Course • Added text related course management system • online homework • Interactive textbook • Instant help • Video, animation • Added section JITTs • Just-in-time-tests • Added text related course management system • online homework • Interactive textbook • Instant help • Video, animation AMATYC 2007

6. Purpose of AssessmentPlan • Pass rates • Pass rates for particular course • Course sequence • Semesters to complete math sequence • Thanks to Jack Rotman • Pass rates • Pass rates for particular course • Course sequence • Semesters to complete math sequence • Thanks to Jack Rotman • Concept assessment • Learning goals/objectives • Pass rates • Pass rates for particular course AMATYC 2007

7. Semesters to Complete • Semesters to pass through the math sequence, passing MA152 for each semester since fall 03: • 82% of those who started in MA098 passed MA152 in 4 semesters (4 courses) • 93% of those who started in MA122 passed MA152 in 3 semesters (3 courses) AMATYC 2007

8. Levels of Assessment • Gathering information about students’ achievement (Marzano, 2001, p. 115) • Level I Assessment • General Course information • Pass rates • Success in subsequent math courses • Semesters to complete sequence AMATYC 2007

9. Levels of Assessment • Level II Assessment • Content specific • Concepts • Learning Map • Summative assessment • Overall course content • Final exam • Item Analysis • Formative assessment • Individual concepts tested AMATYC 2007

10. Cycle of Assessment AMATYC 2007

11. Plan • Online homework required as of spring 2006 • Introduced as option for fall 2005 • Daily quiz • 2-3 questions on concepts in each section • Read text • Come to class prepared • Carrot – replace test grade with quiz average AMATYC 2007

12. Collect Data • Fall pass rate improved overall • 90% • Spring stayed same • 63% AMATYC 2007

13. Analysis • Fall pass rates acceptable • Online homework helped • Anecdotal evidence • Students like all options available with online homework • Have more opportunities to learn • Spring rates need improvement AMATYC 2007

14. Plan Plan • Assesses student learning • Concept assessment • Learning goals/objectives • Develop “Learning Map” • Identify major concepts • Identify teaching objectives • Identify student learning objectives • Means of assessment • Set success standard AMATYC 2007

15. Learning Map AMATYC 2007

16. Learning Map • 1st step • Major topics of course • Linear equations, quadratic equations, inequalities, absolute value, exponential & log equations • Modified as further developed • 2nd step • What do I want them to learn • Teaching objectives • 1st step • Major topics of course • Linear equations, quadratic equations, inequalities, absolute value, exponential & log equations • Modified as further developed AMATYC 2007

17. Learning Map • Knowledge level • Cognitive Process • Action verbs AMATYC 2007

18. Levels of Knowledge • Factual • Basic elements of the discipline • Facts specific to topic • Parts of linear or quadratic equation • Methods to solve quadratic equations • Factual • Basic elements of the discipline • Facts specific to topic • Parts of linear or quadratic equation • Methods to solve quadratic equations • Procedural • Processes of mathematics • Algorithms, techniques, methods • Complete the square • Quadratic formula AMATYC 2007

19. Levels of Knowledge • Conceptual • Interrelationships • Classifications • Principles, generalizations • Pythagorean Theorem • Properties • Metacognitive • Self knowledge • Students need to reflect to learn • Conceptual • Interrelationships • Classifications • Principles, generalizations • Pythagorean Theorem • Properties AMATYC 2007

20. Teaching vs. Learning Objectives • Student learning experience • What they will do • Teaching objective • What I will do • What I want the students to know • Student learning objective • “Students will be able to …” AMATYC 2007

21. Assessment on Map • Each objective has generic question • Indicates how student is assessed AMATYC 2007

22. Learning Map • Gave to students as part of syllabus • Type of questions included • ‘Truth in advertizing’ • Matched final exam to map • Redesigned map • Concepts tested but not part of map • Redesigned final exam • Some concepts not tested but important • Some concepts tested multiple times AMATYC 2007

23. Level II Assessment • Assess concepts learned • Allow overall picture of student progress • Shows strengths • Identifies weaknesses • Areas for further development AMATYC 2007

24. Collect Data • Collected chapter test data • Right/wrong only • Collected final exam item results • 4th semester of data collection • Stats: Percent correct AMATYC 2007

25. Collect Data • Identified concept tested by semester • Shows what is not tested (over-tested) Results of chapter tests for Fall 06, Spring 07, Summer 07 AMATYC 2007

26. Analysis - Tests • More Student Learning Objectives • Needed to add very specific objectives • Dropped some that I thought I wanted but don’t test for • More Student Learning Objectives • Needed to add very specific objectives • Dropped some that I thought I wanted but don’t test for • Success measure – 65% or better correct • Results below need attention • More Student Learning Objectives • Needed to add very specific objectives • Dropped some that I thought I wanted but don’t test for • Success measure – 65% or better correct • Results below need attention • 0 - 10% indicates problem with question as well as topic • More Student Learning Objectives • Needed to add very specific objectives • Dropped some that I thought I wanted but don’t test for • Success measure – 65% or better correct AMATYC 2007

27. Analysis – Final exam • Final used by all sections • Data just my sections • Shows problems with online course on specific concepts AMATYC 2007

28. Asides • Only list what is being assessed • If can’t assess, don’t put as objective • Have a thick skin • Data show changes made in course • Provides longitudinal look at course • Allows comparison of student results by semester or year • Data show where more emphasis needed in teaching AMATYC 2007

29. Before Beginning Assessment • Purpose of assessment • What do you want to know? • Why do assessment? • Action leading from assessment • What do you do with results? AMATYC 2007

30. 5 Basic Questions • Which objectives to assess? • Student learning • What should they know? • How to assess? • Multiple layers • What are results? • Collect/collate data • What is ‘success’? • What to do about results? • Next steps AMATYC 2007

31. References • Angelo, T. A. (1991). Introduction and overview: From classroom assessment to classroom research. Classroom research: Early lessons from success. T. A. Angelo and K. P. Cross. San Francisco, Jossey-Bass: 7-15. • Brockbank, A. and I. McGill (1998). Facilitating reflective learning in higher education. Philadelphia, Society for Research into Higher Education & Open University Press. • Carson, T. (1990). "What kind of knowing is critical action research?" Theory Into PracticeXXIX(3): 167-173. • Cochran-Smith, M. and S. L. Lytle (1993). Inside outside: Teacher research and knowledge. New York, Teachers College. • Elliott, J. (1991). Action research for educational change. Philadelphia, PA, Open University Press. • Leibowitz, M. (1997). Instruction for process learning. Supporting the spirit of learning: When process is content. A. Costa and R. M. Liebmann. Thousand Oaks, CA, Corwin Press, Inc.: 47-54. • Marzano, R. J. (2001). Designing a new taxonomy of educational objectives. Thousand Oaks, CA, Corwin Press. • Mezirow, J. (1990). Fostering critical reflection in adulthood: A guide to transformative and emancipatory learning. San Francisco, Jossey-Bass. • Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA, Harvard University. AMATYC 2007

32. Participation • Gather into groups • Pick one course you teach • List the major concepts • Very few topics • Pick one or two to use • Identify Teaching Objective • Identify Student Objective • Identify knowledge levels • Can be more then one • Identify possible means of assessment • Identify success measures AMATYC 2007

33. Share AMATYC 2007

34. Two Levels of Course Assessment to Identify Student Learning Carol Lerch, Ph.D lerch@dwc.edu S 52 Friday 11:45 – 12:35