Sizing up Your Students, Part 3: Designing Effective Evaluations

1. Sizing up Your Students, Part 3:Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free1-888-886-3951 and use PIN/code240228 Parts of this presentation were adapted from materials created by:Noelle C. Griffin PhD, Assistant Director, UCLA Center for the Study of Evaluation Original material can be found at: http://www.lmu.edu/Page18044.aspx

2. Housekeeping • Maximize your CCC Confer window. • Mute your phone (*6) if you have visitors or noise in your office. • Please note phone audio may be in presenter-only mode. • Ask questions over the phone when the presenter prompts. • Ask questions throughout presentation via the chat window. • Turn on or off Closed Captioning by clicking on the icon. • Save the presentation or chat by clicking on the icon.

3. Agenda Background and Terms Planning your Evaluation Strategy - Defining your Goals, Objectives & Outcomes - Choosing your Measurements - Online Quizzes and Exams - Performance based assessments Analyzing data – rubrics

4. Background and Terms Enormous body of literature on this subject Download list of resources at http://nlight.com/ONE/evalsSurveys_resources.doc

5. Background and TermsSome Basics Assessments versus evaluationsReliability - repeated measurements are same or very similarValidity – you are measuring what you say you are

6. Background and TermsKinds of Measures Quantitative: generates numeric data Qualitative: generates descriptive data

7. Background and TermsCollection Methods Survey, Quizzes, and Exams Interviews Observations Work Products

8. Background and TermsAuthentic – Performance & Behavioral Assignments Projects – individual and group Behavioral observations • General Participation • Discussion contributions Portfolio

9. Background and TermsTriangulation of Data

10. Planning Your Evaluation Strategy Define your course goals, learning objectives and outcomes Determine what kind data and measurement tools will best indicate how well your Learning Objectives & Outcomes are being met?

11. Planning your Evaluation StrategyGoals, Objectives, Outcomes Goals - Represent the overall mission Objectives - Broader perspective on what generally students should know, value or do Outcomes - What students should demonstrate/represent relative to objective

12. Why bother withGoals, Objectives, and Outcomes? They provide the basis for: • Course content • Evaluating students • Communications • Students • Other faculty and administration • Outside world • Curriculum planning

13. Example Goal: The course will prepare students to become a rocket scientist Objective: Students will be able to understand the working components of a rocket and methods for a successful launch Outcome: Students will be able to build and launch a rocket.

14. Learning Objectives and OutcomesBloom’s Taxonomy of Learning Knowledge of terminology (define, identify, describe) Comprehension (discuss, explain, summarize) Application (apply, utilize, interpret) Analysis (analyze, compare, contrast) Synthesis (integrate, organize, construct) Evaluation (appraise, defend, judge)

15. Measurement Tool – Surveys, Quizzes & ExamsTypes of Items Quantitative (numbers) often closed ended (limited responses) • True/False • Multiple choice • Ratings • Single answer Qualitative (narrative & content) often open ended items • Short answer • Essay

16. Measurement Tool - Quizzes & ExamsExamples of Closed Ended Items What is your gender? • Male – Female What is your age ? (10-25) (26-40) (41-55) (55+) Which college do you attend? EVC Butte Foothill De Anza Who is your favorite artist of the 20th Century? • Picasso • Joan Miro • Marc Chagall • Salvador Dali

17. Measurement Tool - Quizzes & ExamsExamples of Open Ended Items Describe the stylistic elements of Art Deco. Why did Van Gogh cut off his ear? What is art? What is your life story until this point?

18. Measurement Tool - Quizzes & ExamsWatch for Bias in Constructing Items How long ago did you stop using street drugs? • Just this week • A month ago • A year ago • Over a year ago

19. Measurement ToolPerformance Assessments Portfolios Capstone assignments Performances/projects Internships/field activities Students evaluating students

20. Measurement Tool – Performance AssessmentsPotential Strengths Direct evidence of student learning Students motivated Realistic context Can be integrated into the grading process Data collection can be relatively unobtrusive to students

21. Measurement Tool – Performance Assessments Potential Weaknesses Additional development/coordination time Faculty trust/buy in critical Faculty time for scoring Reliability/validity uncertain Lack of norms/comparative data

22. Measurement Tool – Performance AssessmentsPortfolios What is it? A collection of multiple targeted pieces of student work that provide evidence of a learner’s achievements along a continuum of learning. Can be formative or summative (i.e., developmental vs. showcase) Can be “online” or “old school”

23. Measurement Tool – Performance AssessmentsPortfolio Pitfalls “Kitchen sink” approach Lack of pre-planning Retrospective re-creation Request vs. requirement (i.e., built into program) Under-estimating faculty time involved

24. Measurement Tool – Performance AssessmentsCapstone Assignment/Project What is it? A culminating project or assignment that provides evidence of achievement of learning goals; generally integrative in nature. Does not only have to be last year; can be spaced in terms of developmental milestones in the program providing longitudinal/comparative data

25. Measurement Tool – Performance AssessmentsCapstone Considerations Does assignment capture the breadth of program learning expectations? Does assignment link back to what is actually being taught in the courses?

26. Measurement Tool – Performance AssessmentsData Collection Process How will assignments be collected/saved for analysis? Integrate into regular classroom process, otherwise compliance difficult Will collection be online or hard copies? Where will data be stored? How will anonymity be maintained?

27. Measurement Tool – Performance AssessmentsScoring Criteria Rubrics: A set of criteria that identify expected dimensions of achievement regarding specific assignments, as well as levels of achievement along those dimensions Checklists Other approaches

28. Professional org/ discipline standards Review representative student work (failing though exemplary) and analyze what criteria led to those grades Student feedback Analysis of grading criteria already used by faculty to grade similar assignments Ask faculty to grade sample work and provide criteria they used, looking for commonalities Pilot/practice use Measurement Tool – Performance AssessmentsStrategies for Developing Rubrics

29. Measurement Tool – Performance AssessmentsStudents Students evaluating themselves Students evaluating other students • Need to train students in critiquing guidelines http://nlight.com/ONE/critiquingGuidelines.doc • Use of rating scales useful Evaluating Team Projects

30. References “Assessing for Learning” (2004). Peggy L. Maki, Stylus Press “Assessment Clear & Simple: A Practical Guide for Institutions, Departments, and General Education” (2004). Barbara E. Walvoord, Jossey-Bass. “A Practical Guide to Alternative Assessment” (1992). Joan L. Herman, Pamela Aschbacher & Lynn Winters, ASCD. “Scoring Rubrics in the Classroom” (2001). Judith Arter & Jay McTighe, Corwin Press.

31. Q & A • Ask now, or via email if you prefer: • Marti Atkinson, matkinson@cccone.org • Bill Doherty, bdoherty@cccone.org

32. Evaluation Survey Link http://tinyurl.com/24n7lq

33. Upcoming/Archived Seminars For upcoming desktop seminars and links to recently archived seminars, check the @ONE Web site at: http://www.cccone.org/seminars/index.php

34. Thanks for AttendingFor information on @ONE’s trainings go to:http://www.cccone.org/index.php