Learning Impact Assessment in Online Mathematics & Statistics Classes at Pace University

1. Learning Impact Assessment in Online Mathematics & Statistics Classes at Pace University David Sachs, Nancy Hale, Barbara Farrell, Patricia Giurgescu Pace University The Ninth Sloan-C International Conference on Asynchronous Learning Networks November 16, 2003 –Session 5

2. Data from two mathematics & statistics online courses(“second-generation”): MAT 125 & MAT 234 • MAT 125 –Technical Mathematics –CSIS/NACTEL Program • MAT 234 –Introduction to Probability & Statistical Analysis • Course enrollment --steady from year to year and • Grade statistics --found consistent with those in comparable traditional classes; no loss in the quality of learning outcomes (“no significant difference” phenomenon) • Course materials developed through an iterative process, aimed at exposing students to • a variety of content presentation resources (combining text with dynamic elements and visualization tools), • multiple forms of testing, to track student learning performance and give student feedback • early student interaction, so instructors can identify the intervention and assistance needed by individual students.

3. Students’ frames for managing knowledge in online courses • Online students demand focus, order and structure, particularly in math & stat online courses • “Frames” (Redish 2002) filter students’ knowledge management: • Social (who will I interact with during this course? –instructor, peers) • Material (what course materials will I use (and how)?) • Skills (what will I be doing here? what is expected of me?) • Affect (how will I feel about what I’m doing?)

4. Focus on developing successive layers of mathematical abilities: conceptual understanding,  procedural knowledge, problem solving, waving through five basic content strands, leading to competency in mathematical reasoning, connections and communication.

5. Learning effectiveness targets (measures) • cognitive outcomes • conceptual understanding • procedural fluency • strategic competence (for problem solving) (knowing what, why, how, when and where certain knowledge applies) • communication outcomes • ability to express quantitative information clearly and rigorously, using the most appropriate technological tools, • attitude/productive disposition, civility & integrity • affective and ethical dimension

6. Learning Assessment • Learning outcomes assessment (objective) –quizzes, weekly homework, proctored exams & individual projects. • Learning experiences assessment (students’ perception) –online student satisfaction surveys • Bloom's learning achievement function: S = f(x,y,z) x = cognitive entry characteristics y = quality of instruction, z = affective characteristics (attitude, motivation)

7. Learning modes in online math/stat courses • Supervised learning --learning from examples, provided by instructor – seek to minimize error, i.e., deviation between learner & instructor’s responses – insufficient for learning to act optimally in new problem domains. • Unsupervised learning --student looks for association rules, concept clustering, patterns, without instructor’s direct guidance or "training set“ • performance measures are more difficult to establish and calculate (e.g., can be assessed from student class projects –open ended assignments, with perfection-based grading) • includes incidental learning (as ability to make sense out of related material, e.g. gathered from discussion board interactions) • Reinforcement learning –student is goal-directed and seeks to maximize reward, by interacting with the problem domain • trade-off between minimal investment (exploiting what student can easily acquire in order to obtain reward) vs. further exploration (investing in more knowledge, in order to make better decisions/choices in the future)

8. Math/Stat course design principles for learning effectiveness • Good structure of materials / logical sequencing --so that students can move easily and systematically through content • Actively involve online students through exercises embedded in the lecture notes and classroom tasks for the Discussion Board. • Adaptive instruction --opportunities for peer to peer instruction, to enhance interaction of students and the instructor • Systematic use of embedded assessments and student self-assessment tools. Tracking of student learning --to identify “nodes” of student understanding or misunderstanding (“node” = key point in understanding a particular content area or process – Zygielbaum, 2001). A correct outcome to a node-task leads to subsequent activities, incorrect outcomes lead to remedial tasks and then move to the subsequent activities. • Involve students in solving real-life problems, with real-life data, using technology • Student-centeredness --communication & coaching/support, to help students clarify their thinking process and strengthen problem solving skills.

9. Challenges • distinguish inadequate presentation or faulty assessment items from poor student performance --item response theory & analysis can help • blind-spot of help-based interaction --common assumption is that students, as mature learners, are willing and able to ask for help when needed; but students with weak metacognitive skills are least able to seek assistance. • moving from easily assessable procedural mathematics tasks to assessing higher order skills (complex problem solving and modeling). A popular assessment component is the assignment of a comprehensive class project, reflecting students’ competencies at the end of the course; the instructor gives individual guidance to students throughout the semester, for completing the project, which is then presented to the class in a valid electronic format and may be included in students’ electronic portfolio. The emphasis is on tackling real-life problems, with real-life data and tools, and strengthening the communication skills and technological fluency. The grading is perfection-based (student has to revise and resubmit project, within given deadline, until it passes pre-set quality standards) • grading misclassification error --trade-off between a-risk and b-risk

10. Major shifts in math assessment practice

11. Enduring characteristics of assessment • relevance --how closely the outcomes are related to marketable employment or institution's mission • utility --potential usefulness for individuals • applicability --extent to which the information is relevant for multiple user groups • interpretability --likelihood of understanding by multiple users • credibility --level of trust of different users regarding assessment information on an outcome • fairness --balance of perspective among groups of diff. ability • scope --size and breadth of sample • availability --accessibility, feasibility • measurability --reliability, and validity • cost --appropriateness of expenditures to produce it

12. Recent influences on assessment • online instruction –scaffolding: instructor continually adjusts the level of help in response to the student’s level of performance • cognitive psychology, learner-centric approach: takes into account expanding the zone of proximal development = range of potential each person has for learning the subject, when the learning is facilitated by someone with greater expertise (Vygotsky); target both the level of actual development & the level of potential achievement. • With modern instructional technology, assessment focuses on building frequent and accurate feedback loops directly into the learning process; • Formative assessment  allows students to structure their learning experiences around their individual needs; encourages self-efficiency, self-appraisal, reflection.

13. Slide summary /conclusions • Target classes: Technical Math & Introd. to Statistics & Probability • Online students’ perspective –frames • Desired competency –math ability layers • Learning effectiveness targets • Assessment of learning outcomes (obj.) & experiences (subj.) • Modes of learning –supervised, unsupervised, reinforcement • Design for learning effectiveness in math • Challenges in assessment • Shifts in assessment practice • Characteristics of assessment • Recent influences on assessment DSachs@pace.eduNHale@pace.edu BFarrell@pace.eduPGiurgescu@pace.edu