Tracking Transformed Courses:

1. Tracking Transformed Courses: Impacts of Tutorials, Instructor, Text, … SJP Summer '05

2. Tutorials at CU • We’ve used “Tutorials” (a research-based replacement for recitations) in Phys 1110 and 1120 twice each in the last 2 years. • All four times have been very successful (data to follow, more if you’re interested!) => we are motivated to continue • Following is some evidence to make the case that we should support this teaching innovation.

3. Resources • Tutorials carry some space, material, and personnel needs. The latter are what need ongoing support • Need “Learning Assistants” (LA’s) - we hire undergrad STEM majors interested in teaching, who have excelled in the course previously.

4. Courses studied Recitation HomeworkText: • Phys 1120 • Fa 04 Tutorials CAPA HRW • Sp 05 Tutorials CAPA Knight • Phys 1110 • Fa 03 Tutorials CAPA HRW • Sp 04 Tutorials CAPA HRW • Fa 04 Knight workbookMP Knight • Sp 05 Trad recitations MP Knight

5. Phys 1120: Call them Tut1 (Fa04) and Tut2 (Sp05) Attempt at replication. Main differences: Tut2 has… • different instructor • different semester • follows up an 1110 taught without Tutorials • different textbook • no long answer on CAPA or exams

6. Summary (up front!) • Despite many other changes in course elements, we replicated Tutorials and basic course structure. • Result: nearly identical learning gains, very high by national standards. (The final score matches what our junior physics majors get on this hard exam!)

7. 1120 BEMA pretest (start of term, before instruction) BEMA = “Brief E&M Assessment”, validated research-based survey of Conceptual elements of E&M (kind of like the “FCI”) Blue data above is F04 (N=319) Pretest ave 26%

8. 1120 BEMA pretest(comparing different semesters) F04 (N=319) Pretest ave 26% S05 (N=232): 27% The two semester’s pretests match closely - no obvious different in starting population’s content knowledge.

9. 1120 BEMA post(how did they do at end of course?) F04 (N=319) 26% -> 59%, S05 (N=232) 27% -> 59% g(ave, F04) = .44+/- .01 g(ave, S05)=.43+/- .01 Posttest results are identical (an impressive replication) High by nat’l standards (typical trad courses, post score = 30-40% !)

10. 1120 mini-summary Despite different instructors, text, exam structures, semesters… Two tutorial implementations: no sig diff on BEMA, CLASS (an attitude/beliefs survey), or common exam questions overall. Final scores on nationally validated survey very high in both classes. Overall neutral results on surveys of student satisfaction. (No disasters, but still room to improve)

11. 1120 “replication study” shows that Tutorials are effective, and can be transferred between instructors. Next: 1110 study, comparing Tutorials with other forms of recitations…

12. Phys 1110: Distinguishing features • 1: “Tut” (Fa03/Sp04) Tutorials + CAPA • 2: “Workbook” (Fa04) No Tutorials, but used small groups/Knight workbook for 1st half of semester (also Mastering Physics, a fancier hw system) • 3: “(More) Trad” (Sp05) Reverted back to mostly traditional recit’s (Otherwise, same text and hw as #2, same course structure, lead instructor had team taught it in Fa04) (different instructors, semesters …)

13. 1110 summary - up front! • “Tutorial-based” courses consistently show stronger results on all measures of learning gains and attitude surveys • Middle course (“2”) used research-based text + hw system, clickers, small-group interactive recitations (but not Tutorials). It fared well, but consistently lower on all measures. (Despite significantly stronger conceptual pretest) • Last course (“3”) replicated “2” with one major change: traditional recitations. Learning and attitude shifts are worst for this course.

14. Pre/post FMCE (Sp04)

15. Phys 1110 normalized learning gains(on nationally validated exam “FMCE”) Tutorials Middle ground Trad rec. gain(1)= .66 +/-.02 g(2)= .585 +/-.02 g(3)= .45 +/-.02 Course (1) with Tutorials has highest gains. Course (2) is 8 points (several sigma’s) worse Course (3) (trad recit.) => significantly lower gains. (but still, double nat’l standards!)

16. Course (1) (2) (3) Impact on different pretest populations:"low starters" pretest <=12.5% Trad rec. Tutorials (% of class in this pool) (Tutorial course (1) is significantly more effective for “at risk” students than traditional-recitation course (3))

17. Course (1) (2) (3) Impact on different pretest populations:"high starters" 50<pre<93% Tutorials Trad rec. (% of class in this pool) => Even the best students at the start (high pretest) benefit more from Tutorials (or group activities) than from the trad recitation

18. 1110 Summary • Compare Tut-based with “workbook/small group” recitation (1 to 2) • measurable diffs in FMCE, common exam q’s, CLASS • Compare Tut-based with “more trad rec” (1 to 3), • significant diffs in FMCE, exams, and CLASS • Tutorials are only one effect. (Instructors, course structure surely play a big role) • But in 1120, changing instructors (and text) did NOT harm the course when the Tutorials were retained.

19. Bottom line • Not a controlled experiment (!) but data show Tutorials are a successful, valuable, and productive course element. • They come at a cost (and benefit) • Cost: $1500/LA/semester * (6-8 LA’s per course) • Need one hr/week training session (TA’s too!) • Add’tal benefits: Cadre of undergrads becoming LA’s, some moving on to K-12 certification. (+ their learning gains after LA’ing are through the roof) • Grad TA’s exposed to research based learning env’t.

20. Recommendation • We should continue implementing Tutorials (and collecting this kind of data) • To run Tutorials, we need support for LA’s, and the training infrastructure that goes along (=> a faculty or experienced grad student assigned to teach the TA’s/LA’s)