Measuring Your Success in Course Redesign

1. Measuring Your Success in Course Redesign Sandi Connelly Gosnell School of Life Sciences, RIT, Rochester, NY Sandra.Connelly@rit.edu or RITGenBio@gmail.com

3. The Dirty Words in Education • Success • Efficacy • Assessment • Evaluations

4. Our Topics for Today • How will you measure success? • Will you use historical data (common exams, etc.) to support the efficacy of your assessments (MyLab & Mastering)? • Mastering platform in Sciences similar to MyLab platform in other disciplines • MyLab & Mastering assignments will contribute what percentage to a students final course grade? = assessment • What do you do with the information evaluated?

5. Today’s Plan • My course structure • Implementation of a few simple techniques • Quick ways to measure success • How to use MyLab / Mastering to maximize student learning • How do other data support MyLab / Mastering use? • How much is too much (or too little!) for a grade item? • How to use MyLab / Mastering for Course Assessment

6. My Course • General Biology (year-long sequence) • Typical # of students: 250-340 (taught in two sections) • This year 408 • Typical # of different majors: 55 from all RIT Colleges • ~ 50/50 science/non-science • Freshmen -> grad students • Current grade calculation (2013 – 2014): • 65-70% exams; 15% Mastering Bio HW; 10% Mastering Bio quizzes; 5-10% in class participation

7. Before you measure success…

8. Are Students Prepared to Succeed? • Preparation (lack of!) comes in many forms • Reading / writing comprehension • English as a Second Language (or 3rd, 4th, ….) • Learning disabilities • Fundamental knowledge in sciences • Study skills • Dedication to the course • Dedication to the degree / college • Etc……

9. Preparing Students for Success • Homework & Quiz • Assign immediately on Day 1 • Provide timely feedback on the assignment • “Suggestions” from instructor do hold water! • Require regular assignments • In Class Response Activities • Every day! (“mock quiz” type) • Immediate or timely feedback critical

10. Preparing Students for Success • Be clear about the goals of pre-course activities • E.g. pretests, diagnostic tests, etc. • Benefit to students • Not enrolled in a class in which they cannot succeed • Decreases DWF rates • Increase retention – intangibles • Benefit to you / course • Less time on “catch up”, More time on application • Less stresses associated with DWF, office hours, etc.

11. So you have “prepared” students … Now what?

12. Measuring Success – How? Of what? • Retention (D/W/F) • Common exams? • Shared assessments across sections? • Final exam scores • Final course grades • MyLab / Mastering • Alternative learning initiatives

13. Measuring Success – How? Of what? • What works for … • You • Your students • Your discipline • Your assessment office? • Your supervisor?

14. Measuring Success – How? Of what? • Let’s focus on MyLab / Mastering as a success tool … • We can add in the other things later …

15. MyLab / Mastering = X % • 2008-2009: Not much – trial only; assignments optional • 3 attempts per question; unlimited time • 2009-2010: Not much – trial only; assignments optional • 3 attempts per question; unlimited time • 2010-2011: 5% of grade; ~20% never did any; nearly all students who completed the assignments had a 100% avg • 3 attempts per question; unlimited time • 2011-2012: 15% of grade; all complete at least most; avg scores 88-93% • 2 attempts per question; unlimited time

16. MyLab / Mastering = X % • 2012: 20% of grade HW and 5% on quizzes; all students complete nearly all (> 97% completion rate) • HW = 2 attempts per question; unlimited time • Quiz = 1 attempt per question; 60 min time limit • 2012 Example • Mitosis/Meiosis HW: 96 questions; avg time spent = 44 minutes; avg score = 85.9% • Homework due before lecture • Result? No time in class on process – instead, theory!

17. MyLab / Mastering = X % • 2013 – 2014: 15% of grade HW and 10% on quizzes; all students complete nearly all (> 98% completion rate) • HW = 2 attempts per question; unlimited time; weekly • Quiz = 1 attempt per question; 60 min time limit; weekly • Food for thought … is this % too much? • Will students cheat, copy, etc.?

18. Everything has challenges …

19. What is your biggest challenge? • Time is the untamed obstacle for everyone .. • Take 5 minutes • Sit down! (or better yet – while you are shampooing!) • What information do you really want your students to retain? • WHY? • What do you want your students to leave your class knowing? • WHY?

20. Challenges to Assessment • Things that I have considered and toyed with … • How much do I want the students to do outside of class on MyLab/Mastering? • Limited to “Time Estimated” = 1 hour • Less if possible! I want their attention!

21. Challenges to Assessment • Things that I have considered and toyed with … • What is the value of a long assignment if the students don’t do it, copy it from another student, just use all of the hints, etc.? • For students who are failing exams, but have 100%+ on the homeworks, I look at their time spent in MyLab / Mastering • Discuss issue with them in person and make note in the Institute Reporting

22. Challenges to Assessment • Things that I have considered and toyed with … • Are the students just memorizing answers, or are they applying the information and retaining it? • Have stopped assigning reading / test bank questions • Implement more low stakes quizzes • Forced repeated recall of information in class

23. You have data … Now what?

24. Student Data Assessment • Why assess? • My general interest • Are the students learning what I think they are learning? • How can I improve this course? • University call to assess Gen Ed courses (Accreditation) • Assessment needs? • My interest • Exam and homework scores before and after MyLab / Mastering implementation • University Guidelines: 1 – 4 scale of comprehension

25. Student Data Assessment • What was assessed? • Kept it simple to start! • Exam averages • Add as I have interest / time! • Participation in MyLab / Mastering • Scores on MyLab / Mastering • Time spent on MyLab / Mastering

26. Student Data Assessment • Exam grades (presented herein) • All stored in ExcelTM since 2007 • Mastering (not presented) • Export grades & time spent from Mastering to ExcelTM • Processing, Compilation, and Statistical Analyses • ExcelTM

27. Exam Score Trends Err = St Err Avg n = 244

28. Talk to me data! • Higher exam averages every year – but leveling out • Significant differences only in plants – but trends are obvious • Too many students from too many majors to see huge jumps in scores despite all efforts • Tightening of exam scores • Standard deviations and ranges decreasing • Increasing amount of “average time spent” on MyLab / Mastering assignments

29. Assessment Challenges • Why do exam grades change? • Things I have considered • What changes from year to year other than MyLab / Mastering? • Students? • Accessibility of tutors? • Me? • Order of topics covered? • Lab instructors?

30. Assessment Challenges • Why do exam grades change? • Things that I want to consider • Students • Time spent on studying (other than MyLab / Mastering)? • Major? • College entrance exams? AP Bio? • DWF rates by major and biology “experience” • Me • Lectures improved? • Lecture structure change with online assignments?

31. “Napkin Notes” • General observations: • Increased time spent on assignments after 1st exam of term • Increased # questions in office hours on assignments • Increased time spent “off topic” in lecture – not a bad thing! • Changes implemented to date: • Lecture time is more interactive • Participation activities with the students (worksheets, group work, think-pair-share, flipped lectures, etc.) • Modifications to assignments and quizzes to “challenge” the students in preparation for class and exams

32. Advice for Instructors Looking to Assess Success • Don’t assess while you are teaching the course! • Assessment can be done any time! • Store the data, save it for later • Even if you have time … subconsciously not good for your teaching! • Don’t try to assess everything at once! • Pick one exam in your course, and compare the average ± standard deviation (standard error) between years • Even simpler? • Compare exam averages between class sections. TA DA! You just did an assessment!

33. Advice for Instructors Looking to Assess Success • Don’t expect MyLab / Mastering to make your students in to geniuses! • These are tools! And how the students implement the tools (accept the tools!) will be highly variable. • You may not see immediate changes in exam grades • Accept the “intangible” as a result! • You can not grade student enthusiasm (which means you can’t assess it – sorry!) • Student enthusiasm is contagious – not just to other students – but also to you!

34. Advice for Instructors Looking to Assess Success • What can you do with your data? • FLAUNT IT! • White papers • Peer-reviewed papers • Teaching portfolio • Promotion dossier • Assessment office

35. Final thoughts …

36. What else is assessable? • I got my class back! • Not spending class time on giving quizzes, answering questions about homework format, etc. • What do we do then? Talk about Biology! • AND … most of my assessment is built in to MyLab / Mastering and my course management system! • Don’t reinvent the wheel! • Download grades, assign your assessment strategy, .. TA DA!

37. What else is assessable? • HUGE time savings in my classes • I have “recovered” 35 minutes per week in class (of 150 min) with no quizzes and homework things in class • Prepared students means < 45 min lecture! > 105 min biology exploration! • Time saving is so huge … how huge is it? • Enough that I am completely “flipping” one section • DATA! DATA! DATA! DATA! DATA! DATA! DATA! DATA!

38. But .. The Most Important Thing to Me?

39. What the Students Have to Say! • "Mastering Bio helps a lot! The more you have us do, the more I learn!” • "I wish that our exams were as interactive as Mastering! I love learning that way!" • "(Favorite part about Mastering) aspects of the assignments helped me to better grasp the materials through the videos. I would not have done as well on those parts of the test without Mastering.” • "More Mastering - 'nuf said!"

43. Let’s Recap • How will YOU measure success? • Will YOU use historical data (common exams, etc.) to support the efficacy of your assessments (MyLab & Mastering)? • MyLab & Mastering assignments will contribute what percentage to YOUR students final course grade? = assessment • What do YOU do with the information evaluated?

44. The Dirty Words in Education • Success • Efficacy • Assessment • Evaluations

45. THOUGHTS!? Sandi Connelly Gosnell School of Life Sciences, RIT, Rochester, NY Sandra.Connelly@rit.edu or RITGenBio@gmail.com