It’s Raining CATs and DOGs: Increasing Student Learning

1. It’s Raining CATs and DOGs: Increasing Student Learning Members of the Assessment Advisory Group: Anna DelVitto, Maura Stevenson, Monica Washington and Mary Kate Quinlan June 29, 2010

2. CATs and DOGS Classroom Assessment Techniques Developing Outstandingly Good Strategies

3. Introduction • Overview of Assessment • Activities that are Working in Our Classrooms • Turning the Page from Evaluation to Assessment • Group Activity

4. Overview of Assessment The primary purpose of assessment is the improvement of student learning

5. Three Levels of Assessment

6. Classroom Assessment • What are students learning? • How well are they learning it? • Classroom Assessment Techniques reinforce student learning • Focus on what’s important • Provides additional practice • Students become self-aware learners

7. Characteristics of a Good CAT • They focus on things that can be changed during the course of one term. • They produce results that are easy to analyze. • They provide useful information about what students have or have not learned. • They provide information to students and teachers about the kinds of changes needed to improve learning. • They reinforce and enhance learning the content or skill being assessed. Source: Classroom Assessment Techniques by Angelo and Cross (1993)

8. Activities that are Working in Our Classrooms Rubrics Weekly Journals Anna

9. Rubrics

10. What is a Rubric? • Rubrics are essentially guides for scoring student assignments. • A tool to evaluate students’ products to determine if they achieved the learning outcomes • Rubrics exist in many different formats, from the simplest (checklist) to more complex (descriptive or holistic rubrics).

11. Shoe Rubricevaluation of shoes

12. Advantages of Rubrics for Students • Clarifies learning objectives – provides benchmarks • Promotes understanding of grading process • Focuses on what’s important – clearly defined • Improves performance and facilitates learning • Allows students to judge their own work • Provides concrete feedback

13. Advantages for Instructors • More objective, consistent, and reliable • Clarifies criteria for teacher • Identifies areas for re-teaching – how effective was instruction? • Facilitates and speeds up grading • Satisfies accrediting agency requirements for assessment

14. Simple Beginnings: The Checklist • Does the essay deal with the topic that was set? • Does the essay answer the question that was set? • Is the grammar, punctuation and spelling acceptable? • Is it neat and legibly written? • Does it cover all the main aspects and in sufficient depth? • Is the content accurate and relevant?

15. Math Graphing Rubric Grading Categories: (1) Completeness and accuracy of graphing outline (2) Accuracy of graph (3) Neatness and attractiveness

16. Quantifying the written work… • Based on the scoring rubric just created, the math graphing assignment is worth a maximum of 12 points (3 categories worth a maximum of 4 points each). • Student scores would be expected to range from 3 to 12.

17. Improve the Outcome !! • Give rubric to students along with the assignment. Review expectations. • Pass out samples of student work and have students use the rubric to evaluate them.

18. Evaluation of Assignment • Use the scoring rubric to grade student assignments. • Reevaluate the rubric – Did it work? Make changes to the scales and grading categories based on actual application of the tool.

19. Developing a Rubric • Determine learning outcomes • Keep it short and simple • Each item should focus on a different skill • Evaluate only measurable criteria • Try to fit it on one sheet • Reevaluate the rubric – did it work?

20. Math program assessment Monica

21. Weekly Journal

22. Instructor’s observation • What challenging situation existed in Marketing? • What would it take to overcome this challenge? • How would this solution prove beneficial?

23. Understanding The Challenge • Helping students go beyond memorization to application of marketing terms and techniques

24. Overcoming the challenge • Based on weekly study units, a marketing term was selected for students to examine closely. • Students developed weekly journals.

25. Benefiting from this CAT • Students experience: • Ability to apply marketing terms to real life situations • Growth in understanding of professional terms • Opportunity to connect marketing to business and economics

26. Regarding this CAT • Instructor Achieves: • More satisfaction • Meaningful feedback

27. Let’s review our main questions again. • What challenging situation existed in Marketing? • What would it take to overcome this challenge? • How would this solution prove beneficial?

28. It’s time to see who can identify the first question to help us understand the instructor’s observation.

29. If you were thinking…What challenging situation existed in Marketing?You are correct.

30. The second question… Do you remember ?

31. Were you thinking… • What would it take to overcome this challenge?

32. What about the third question to help us understand the Weekly Journal CAT?

33. Of course, you remembered this…. • How would this solution prove beneficial?

34. GOOD JOB!

35. Let’s review our main questions again. • What challenging situation existed in Marketing? • What would it take to overcome this challenge? • How would this solution prove beneficial?

36. Weekly Journal Assignment

37. Try this model with a course you teach in mind • What challenging situation existed in Marketing? • What would it take to overcome this challenge? • How would this solution prove beneficial? • What challenging situation exists in your course? • Would a weekly journal help to overcome this challenge? • If you can visualize a solution to this challenge, what are the benefits? To whom?

38. A Look at the actual CAT • Faculty Name: Tracy McDonough • Campus: North Date: 3-25-09 • Discipline/Program: Business • Course: Marketing Topic: Critical thinking • Classroom Assessment Technique (CAT) used: Weekly journal • Summary of Student Feedback: Test scores have improved over last semester a mean of 62 up to a 68 • How Were the Results Used to Improve Student Learning? I have noted for several semesters that students could memorize definitions in marketing but then could not apply them. Marketers are not asked to repeat definitions on the job; they are asked to apply them to complicated problems in communication, promotions, etc… My students could not make this jump. This semester I instituted a weekly journal format. They are to pick a topic from the week and apply it to marketing efforts they see in their daily lives within the journals. Students have talked about target markets for the shows they watch; direct mail pieces they receive at home and even how CCAC has been marketing to them. I have been amazed at this growth in understanding. --

39. A section of the CAT • Classroom Assessment Technique (CAT) used: Weekly journal Summary of Student Feedback: Test scores have improved over last semester a mean of 62 up to a 68

40. A Closer Look at a Section of the CAT • How Were the Results Used to Improve Student Learning? I have noted for several semesters that students could memorize definitions in marketing but then could not apply them. Marketers are not asked to repeat definitions on the job; they are asked to apply them to complicated problems in communication, promotions, etc… My students could not make this jump

41. The CAT ends…. • This semester I instituted a weekly journal format. They are to pick a topic from the week and apply it to marketing efforts they see in their daily lives within the journals. Students have talked about target markets for the shows they watch; direct mail pieces they receive at home and even how CCAC has been marketing to them. I have been amazed at this growth in understanding.

42. This CAT – Weekly Journal - appeared in the May 2009 edition of the Achieving the Dream newsletter • Submitted by • Tracy McDonough • North Campus Maura

43. Turning the Page from Evaluation to Assessment Item Analysis Conferencing

44. Item Analysis

45. Evaluation - Instructors utilize “Scantron” forms to make grading exams/quizzes easier and quicker.

46. Assessment • Scantron forms can be used for analysis of test items too. • You can use an item analysis form to determine how many/what percentage of students get a particular question wrong…

47. - In this example, 15 students took the exam. - For question #19, 12 students answered the question wrong. - Did the instructor’s answer key have an error? If not, this serves as a red flag.- The instructor needs to eliminate or re-word the question or improve instruction in the material related to the question.

48. Would it help or be of interest to know what “wrong” answer students might have selected for item #19? • Yes! • Such commercial analysis products exist, but start to get into a greater expense range.

49. Detailed Item Analysis • Are test items too hard or too easy? • Do test items differentiate between students who do well on the overall test and those who do not? • Difficulty Index • Discrimination Index

50. Difficulty Indexn=30 • Question #1 • 24 of 30 students answered correctly • 24 divided by 30 = .80 (difficulty index) • Difficulty index >.75 generally considered “easy” • Difficulty index <.25 generally considered “difficult” • So, question #1 is considered moderately easy • Question #2 • 12 of 30 students answered correctly • 12 divided by 30 = .40 (difficulty index) • “moderately difficult” • Bothered by # of students who selected B as the answer? • B should be analyzed and possibly reconsidered