Welcome!

1. Welcome! A Learning Communities Approach to Assessment of Higher Cognitive Skills Merilee Griffin Michigan State University grifflee@msu.edu

2. The simulation • Your college wants to measure leadership ability among students • You’re on the committee • The committee identified “ability to inspire and motivate peers” as a component of leadership • Students have submitted videos, which you are evaluating

3. As you view each video… • Trust your own judgment and experience • Note what the student does or says that is effective or ineffective • Jot down a few notes

4. Score each video • How well does each student demonstrate the ability to inspire and motivate peers? • Six-Point Holistic Scale where 6 = Outstanding Do you need a handout?

5. If viewing this slide show as a handout, skip over the simulation slides, 6 – 19. • Results of the simulation are fairly predictable: even without direction to do so, people gradually begin approximating others’ scores.

6. Tristan Wants to inspire and motivate fellow students to vote for him for Student Council (He’s filming the video in the Dominican Republic, but he’s campaigning for a seat at the American International U in Kuwait)

7. Tristan • Score 1 = • Score 2 = • Score 3 = • Score 4 = • Score 5 = • Score 6 =

8. Positives

9. Negatives

10. Karilyn She is trying to inspire and motivate college freshmen to get involved in the Student Senate

11. Karilyn • Score 1 = • Score 2 = • Score 3 = • Score 4 = • Score 5 = • Score 6 =

12. Positives

13. Negatives

14. Craig Wants to inspire and motivate students to vote for him for student body president

15. Craig • Score 1 = • Score 2 = • Score 3 = • Score 4 = • Score 5 = • Score 6 =

16. Positives

17. Negatives

18. Score-changing period

19. Score changes • How many changed? • How many changed in the direction of greater consensus? • How many changed in the direction of greater disparity?

20. “Norming” • Used by testing companies to train scorers of writing tests • Also occurs naturally, as people adapt to each other in groups

21. Why initial disparity? • Each assessor is a unique package of prior experiences • Each has a unique perspective • More disparity exists at the higher levels of cognitive skills

22. Revised Bloom’s Taxonomy

23. “Experts” • Expected to deliver consensus • Often disagree more than laypeople • Frustrating to non-academics!

24. Possible explanation • Assessors usually work from Positivist paradigm • Narrow focus and get very specific about defining terms • Eliminate confounding variables • Clarity • Reliability

25. Possible explanation, P. II • Faculty often work from a Constructivist paradigm • Embrace diversity of perspective, honor cultural and social influences • Broad, rich mix of elements • Validity

26. Closing the Loop Program improvement Curriculum Assessment Teaching

27. Closing the Loop Program improvement Curriculum Assessment Teaching

28. Possibly this happens… • The assessment team spends hours developing outcomes and measures • It feels like they are arriving at a positivist “truth” • In fact, they have developed an “interpretive community”

29. Stanley Fish • Is There a Text in This Class? (1980) • People who engage in communal discourses over a period of time develop shared assumptions, beliefs, and perspectives • They interpret texts with a high degree of accord

30. Closing the loop • Faculty cannot be left out of the dialog • Must become part of the interpretive community

31. Alternative • Forget about testing • Engage faculty in continuing process of assessment • Focus on actual samples of student work • Direction is bottom-up, not top-down • No face-to-face meetings • Interactive Web site

32. Web site • Pages for posting student writing, video, audio recordings, photos, etc. • Table of scores • Pages and audio files of faculty analyzing the writing and justifying their scores • Asynchronous discussion boards

33. Advantages of Web • Flexible time • Web application avoids tension of face-to-face meetings • Opportunity for review and reflection

34. Old method, New technology • Holistic scoring - about 30 years old • Generally successful • Meeting time is a burden on schedules and expensive • Web-based practice – flexible time and much less expensive

35. Choice • Everyone begins with equal status, or • A few people take the lead as assessors • Others use their work as guide • Gradually begin working as equal participants

36. Lave and Wenger • Legitimate Peripheral Practice (1991) • Wenger: Learning Communities (1998) • Novices learn by practicing alongside adept practitioners • Begin with simple skills, move to complex • They not only learn skills, but develop identities • They gradually move from novice to adept status, becoming leaders

37. Focus on student work • Avoids abstract arguments • Grounds words like “effective” and “adequate” • Keeps focus on desired outcomes

38. Kenneth Bruffee • Collaborative Learning: Higher Education, Interdependence, and the Authority of Knowledge (1999) • “Reacculturation” • Transition groups surface and challenge assumptions; build new perspectives • Learners don’t give up old ideas; they nest them in larger perspectives

39. Scoring • Use a rubric or not • Similar to the scoring done on SAT, GRE, etc. • People can achieve fairly high inter-rater reliability in a fairly short time • Different • Dialog will be broader, richer • More valid in the eyes of faculty

40. Assessment should be… • Ongoing, not a once-a-year event • Like professional reading, attending conferences • Ever-changing

41. What about accountability? How can comparisons be made from one institution to another? • On the Web – 24/7 access by anyone • Natural curiosity about how we stack up against others • Eventually, links between institutions • Incentives for norming with others by CHEEO’s, accrediting agencies, Ed

42. Who would develop framework? • Academic programs – Associations • Membership • Ethic of sharing knowledge • Academic expertise • Network • Non-academic programs – NASPA?

43. This is doable! • Technology is simple • U’s have hosting Web space and technical expertise • Extremely low-cost way to do assessment

44. Would faculty buy in? • Holistic scoring among writing teachers in 1970’s • Collegial dialog surprised people • Their own vocabulary, reference points, theoretical frameworks

45. Collaborative learning in online communities • Design must foster community among participants • Design must be sensitive to cultural and social contexts • Design should foster the collaborative development of new knowledge

46. Session Ending Details • Please fill out session evaluation as we do Questions and Comments • Sign up for emailed Reference List or leave your card • If you are working on a similar project, please let’s get in touch! Thank you!

47. Questions and Comments