Andrew N. Christopher Albion College

1. Though this be madness, there is a method in it:Using methodological exemplars to improve pedagogical research Andrew N. Christopher Albion College Jordan D. Troisi Widener University Presented at the 12th Annual Society for the Teaching of Psychology Best Practices Conference October 11, 2013

2. “Though this be madness, there is a method in it” • A brief summary of the beginning of “Hamlet:” • Hamlet is visited by his father’s ghost (The King) • The ghost says that his own brother (Claudius) has killed him • 1-2 months later, Claudius has married Gertrude, Hamlet’s mother • The ghost says “kill Claudius” • Hamlet says “sure,” and that he’s going to act mad for awhile, presumably to cover his tracks • Perhaps there is a method in it?

3. “Though this be madness, there is a method in it” • Hamlet is acting(?) mad, and a bunch of characters develop hypotheses as to why • Gertrude hypothesizes he’s mad because his father has died and she married his brother in 1-2 months • Claudius hypothesizes he’s acting mad because he wants to steal the throne he has recently assumed • Polonius hypothesizes he’s mad in love with his daughter Ophelia and that he just wants to sleep with her • All valid hypotheses, which they each test (poorly) • But they ignore a crucial confound. . . • A freaking ghost is telling Hamlet what to do!  FAIR  FAIR  FAIR

4. Guidance in pedagogical research from Hamlet • There’s a lot of madness out there, and it’s often hard to understand • Luckily, psychologists are trained to determine the reasons why behavior occurs • Also luckily, such behavior is rarely guided by ghosts • What happens in the classroom may seem like madness, but there’s a method in it • We need the methodological tools to make sense of the madness

5. Overview • Scholarly teaching, the scholarship of teaching and learning (SoTL), and the learning sciences • Issues of control in pedagogical research • Effective methodology exemplars from SoTL and the learning sciences • Take-home messages for effective pedagogical research

6. SoTL & Learning SciencesDaniel & Chew (in press) SoTL Learning Sciences Extension of higher education Evaluates specific pedagogical methods Application-driven Constructs may lack theoretical precision Extension of “basic” research in psychology Findings may not translate across teaching contexts Theory-driven Construct operationalizations tend to be widely-accepted

7. SoTL & Learning SciencesDaniel & Chew (in press) SoTL Learning Sciences Predicting the weather Knowing the laws of thermodynamics

8. Research on Note-Taking • Williams and Eggert (2002) examined the notes of 125 students in an undergraduate human development course • Students took notes in a 125-page study guide on both their: • Class notes • Textbook readings

9. Research on Note-Taking • Three blind raters coded these notes for: • Completeness • Length • Accuracy

10. Research on Note-Taking • Completeness, length, and accuracy used to predict: • Brief essay quizzes linked to the notes • Exam questions from text only • Exam questions from lecture only • Exam questions from both text and lecture

11. Research on Note-Taking • Peverly et al. (2013) randomly assigned 200 undergraduates in an educational psychology course to either: • Outline provided • No outline provided • while taking notes on a 23 min video-taped lecture

12. Research on Note-Taking • Predictors • Handwriting speed • Wrote letters of alphabet horizontally in orders as many times as possible in 45 sec • Language comprehension • 7 reading passages with questions about each • Verbal working memory • Listening span test • Attention

13. Research on Note-Taking • Dependent Variables • Quality of notes • Written summary of the lecture

14. Research on Note-Taking • Used structural equation modeling to assess predictors’ relationships with quality of notes and written summary of the lecture

15. Overview • Scholarly teaching, the scholarship of teaching and learning (SoTL), and the learning sciences • Issues of control in pedagogical research • Effective methodology exemplars from SoTL and the learning sciences • Take-home messages for effective pedagogical research

16. SoTL: The Early Days • Focused on student attitudes (i.e., Did students like it?) • e.g., Christopher and colleagues (2004) • Wesp and Meile (2008, p. 362) • “It appears that student opinions about the effectiveness of teaching techniques are inaccurate.” • “Researchers should prefer direct measures (of student learning) because they provide a more accurate assessment of pedagogical effectiveness….”

17. “Learning Might not Equal Liking” • Attitudes about courses and topics may not equate to learning, and vice versa • After a course on research and statistics, students reported increases in knowledge but no changes in favorable attitudes toward the subject (Sizemore & Lewandowski, 2009)

18. SoTL: The Early Days = A Vicious Cycle • Psychologists are control freaks “Correlation does not equal causation” equates to “Correlational research is not as valuable as experimental research”

19. SoTL: Why is it so hard to do? • Special challenges in doing SoTL: • How often you teach a particular class • Number of students in the class • Time of day a class (or sections of a class) is offered • Lack of control relative to lab-based research

20. Overcoming the “Control” Issue in SoTL • As psychologists, we love control • As teachers, we want to do whatever we can to maximize student learning and sometimes produce attitude change • But, classrooms are very “noisy” places

21. Finding Appropriate Comparisons • Smith (2008) • Use a previous class • Compare and (ideally) statistically control for ACT/SAT scores • Use an in-tact class • Divide the class into 2 “equal” groups • Group 1 attends the first half of class; Group 2 attends the second half of class • Use your intervention on one of the 2 groups • Measure outcome variable on the same day

22. More Methods to Use Within 1 Class • Bartsch et al. (2008) • Typical pre/post contains same questions • Using different tests at pre and post = instrumentation confound • Focus on using a 1-group pre/post with alternate forms

23. 1-Group Pre/Post with Alternate Forms • Prepare 2 versions of an assessment instrument (i.e., Version A & Version B) • Give half the students Version A at pretest and Version B at posttest • Switch for the other half of the students

24. 1-Group Pre/Post with Alternate Forms • 2 (Version A first or Version B first) x 2 (pre and post) mixed ANOVA • Allows us to learn if: • Groups were equivalent to start with • Version A was equivalent to Version B • The intervention does or does not facilitate student learning

25. 1-Group Pre/Post with Alternate Forms • Use random assignment to: • Form your 2 groups of students • Put questions on the 2 versions of the assessment instrument

26. Overview • Scholarly teaching, the scholarship of teaching and learning (SoTL), and the learning sciences • Issues of control in pedagogical research • Effective methodology exemplars from SoTL and the learning sciences • Take-home messages for effective pedagogical research

27. Rapport, Student Motivation, and Course Attitudes • We’d all like good rapport, motivated students, and favorable attitudes toward our courses • Legg and Wilson (2009) examined how sending a welcome email to students before a course began could create these outcomes • Building rapport before the first day of class!

28. Rapport, Student Motivation, and Course Attitudes • Half of an Introduction to Psychology course’s students (n = 66) received a welcome email one week prior to the beginning of the semester • Randomly assigned, instructor blind to participant condition • Surveyed rapport, student motivation, and course attitudes at: • End of 1st class period • At mid-term • After the final exam

29. Rapport, Student Motivation, and Course Attitudes • Most favorable ratings for email group during the first class period • Fewer effects for subsequent measurement periods • Some interactions with gender of student • Students in the email group were less likely to drop the course

30. Kudos to Legg and Wilson (2009)! • A welcome email is a low-tech, time-efficient way to contact students and start building rapport and motivation • Authors randomly assigned students within one class and were blind to which students received the email • Measured multiple, relevant DVs, at 3 time periods

31. Student Attitude Change in a Prejudice Course • Reducing prejudice is often a goal of psychology courses • Reducing prejudice should be assessed over time, as done superbly by Kernahan and Davis (2010) • The utility of the classroom for ideas in basic research

32. Student Attitude Change in a Prejudice Course • Compared a Psychology of Prejudice and Racism course & a Statistics course • Utilized a pretest-posttest design with a control group, plus a one-year follow-up • Design allows for examination and comparison of attitude changes from beginning to end of semester, and beyond

33. Student Attitude Change in a Prejudice Course • Results showed that by the end of the semester, those in the Prejudice and Racism course showed greater: • Awareness of white privilege, white guilt, noticing of racism, and responsibility for taking action • At one year follow-up, some effects: • Plateaued (e.g., awareness of white privilege) • Waned (e.g., responsibility for taking action) • Increased (e.g., comfortability in mixed-race interactions)

34. Kudos to Kernahan and Davis (2010)! • Comparison between a group with an anticipated effect and a control group • Effects examined across 3 time periods, including one year later • High external validity, and the target outcome of interest for basic and pedagogical research!

35. Learning or Living-Learning Communities • Designed to create a coherent educational experience for a group of students centered on a particular topic • Can improve student performance and retention while fostering relationships • Buch and Spaulding (2008) spectacularly examined these important outcomes utilizing a longitudinal design with a matched comparison group

36. Learning or Living-Learning Communities • Compared GPA, retention, co-curricular involvement in psychology courses/activities over 7 semesters of college • Utilized a longitudinal design with a control course with students matched (n total = 40) on important variables (SAT score, gender, ethnicity, generation status)

37. Learning or Living-Learning Communities • Cumulative GPA better in the Learning Community group during the 1st, 2nd, and 4th semester of college • Retention at the school and progress within the major were better among Learning Community group (some significant findings, some non-significant but in the expected direction) • Co-curricular involvement higher in the Learning Community group • Involvement in psychology club, internship, research

38. Kudos to Buch and Spaulding (2008)! • Longitudinal design (see also Buch & Spaulding, 2011), with a comparison group matched on important variables • On a hot topic in higher education • 7 • Semesters • Of • Data • !

39. Laptop Multitasking and Performance • Multitasking divides attention and can lead to poor memory of course material and poor performance • Laptops in the classroom can provide distractions for the users and others in view of laptop screens • Sana and colleagues (2013) put distractions from laptop use to the test (literally!)

40. Laptop Multitasking and Performance • Two studies on how well students retained information during lectures for a multiple choice quiz • One study examined student performance when they were randomly assigned to multitask during the lecture (given some online tasks to perform) • A second study examined student performance when they were in view of confederates’ laptop screens on which they were multitasking

41. Laptop Multitasking and Performance • In the first study, students assigned to multitask performed worse on a quiz of the information of the lecture (with no difference between simple fact-based items or application items) • In the second study, students who were in view of a research confederate who was multitasking performed worse on the quiz

42. Kudos to Sana and Colleagues (2013)! • Examining a topic extremely relevant to teaching and learning • Including two studies, one on distraction from the self and one on distraction from others • Using an experimental design that simulated classroom experiences • High external validity and mundane realism

43. Time-of-Day Preference and Grade-Point-Average (GPA) • Preckel et al. (2013) examined whether high school students’ (N = 272) time-of-day preference was predictive of their GPAs in a variety of subjects • At the college level, we discuss the “best” times to offer classes from a students perspective

44. Time-of-Day Preference and Grade-Point-Average (GPA) • Preckel et al. (2013) statistically controlled for variables suggested to be predictive of academic performance (e.g., conscientiousness, cognitive ability, achievement motivation, gender) • Included measures of other-report as well as self-report

45. Time-of-Day Preference and Grade-Point-Average (GPA) • Entered time-of-day preference on the last step of three hierarchical linear regressions • Used three criterion: • overall GPA • math-science GPA • language GPA

46. Time-of-Day Preference and Grade-Point-Average (GPA) • Found that students with an evening preference had a lower overall GPA, lower math-science GPA, and language GPA than students with a morning preference

47. Time-of-Day Preference and Grade-Point-Average (GPA) • Use of statistical control, a commonly-used technique in personality research • Use of multiple predictors, including self- and other-report data

48. Academic Dishonesty and Narcissism • Much has been made about the “rise of narcissism” in Western society during the past 2-3 decades • Teachers have lamented (likely since the beginning of time) academic dishonesty among students

49. Academic Dishonesty and Narcissism • Brunnell et al. (2011) randomly assigned 199 Introductory Psychology students to one of two questionnaire conditions: • Questions referred to the Self • Questions referred to the Typical Student on Campus

50. Academic Dishonesty and Narcissism • Questions referred to: • Guilt experienced for cheating if: • Exam was overly difficult • Classmates did not help them study • Friends pressured them to cheat • Prevalence of Academic Dishonesty • The number of times they (others) cheated on a test in the past 12 months • Predicted how many times the typical student on campus would cheat on a test