Morphing Rubrics to Adapt to Developmental Student Needs

1. Morphing Rubrics to Adapt to Developmental Student Needs AjaHenriquez, MFA, EdD Student CSUSB English Instructor:Crafton Hills College, California Baptist University

2. What is a Rubric? • Definition of • rubric (ˈruːbrɪk) From Collins English Dictionary

3. Why Rubrics?

4. School Specific Issues • Approximately 31% of incoming students transfer into the developmental level (English 015) at CHC, while 36% transfer into a lower, remedial level course (914): Around 33% of students test into college level English courses (Research Briefs, 2011). • Approximately 52% of students who pass the “Preparation for College Writing” course (English 015) persist in the college and successfully complete a college level writing course (Student Equity Data, 2011).

5. What We Can Control • The success rates tell us that there is something not transferring in our courses. • Many things we cannot control: student readiness/commitment, family problems, financial problems, etc. • We can address our rubrics and how we communicate with the students. • As we know from research, the students may simply misunderstand what we say/write when we grade.

6. What is the Purpose of a Rubric? • We use it to grade for a few reasons: • To assess specific items in a given assignment • To help ensure or increase uniformity of assessment • It is a tool for instructors to communicate with students. • Students must be able to understand the communication.

7. Language Disconnect • We may be taking for granted student understanding of our rubrics. • Developmental courses often act as an introduction into the academic discourse community (Bizzell, 1982): • “students from different social classes come to school with different abilities to deal with academic discourse: middle-class students are better suited by their socialization in language use to deal with academic discourse's relative formality and abstraction than working-class students are. This unequal removal from academic language is, of course, exacerbated for students whose home language does not resemble the so called standard English” (192).

8. Student Perspective VS • “It’s like they [the affluent students] had their own classes, and we [the students from the poor/working class] had our own classes. We were, like, segregated in the same school. “ • --Prep for College Writing Student

9. Demographics of Basic Skills Students • “Crafton Hills College, then, serves a community where the constituents must commute to work or school, where the poverty and unemployment rates exceed the state average, and where the income is below the national average.” From Demographics, CHC webpage: http://www.craftonhills.edu/About_CHC/Demographics.aspx

10. We Should Keep in Mind: • Students may not understand the language we use on rubrics • We must ask ourselves if we have written the rubric for our understanding or student understanding.

11. Example: Unexamined Rubric

12. A Tool That Doesn’t Work • Does a developmental student know what these terms mean when it comes to writing? • If the students don’t understand what the rubric means, then they can’t use it to improve. • How can we make this more understandable?

13. Example: Revised Rubric

14. Other Concerns • Even with some clearer criteria, there are still some issues with this rubric, which are less simplistic than just the language used. • Validity-does this measure what we want it to measure? • Reliability-are the scores consistent? Messick (1995)

15. Validity • Content Validity: to ensure we are measuring what students should know and not some other thing (personal preferences), we must match each item on the rubric to a course objective. • Substantive Validity: we should make sure that the items on the rubric connect to different types of cognitive processes and are of varying difficulty Messick (1995)

16. How to Ensure Content Validity • Review the items on your rubric and make sure they match to the course objectives along with any departmental norms. • English departments often have norming sessions for grading, which these instructors must take into consideration when evaluating the content of their rubrics. • EXAMPLE partial rationalization of content validity  • Some things that make it into our grading that should not be there: stapling, niceness of binder or folder, whether it was accompanied with a latte, etc. • Item one corresponds with objectives 4, 5, 6, 11, as well as the grading criteria for writing (demonstrates skillful use of vocabulary and syntax; is generally free from errors in mechanics, usage, and sentence structure). • Item two corresponds with objectives 7, 8, 9, as well as the grading criteria for writing (focuses clearly on the topic and responds effectively to all aspects of the assignment; explores the issues thoughtfully and in depth; is coherently and logically organized with a thesis statement supported by apt reasons and specific, detailed examples).

17. Substantive Validity For explanation of Bloom's Taxonomy, watch http://youtu.be/qjhKmhKjzsQ

18. Ensuring Substantive Validity • First of all, we should examine the items in our rubrics to ensure there is a spread of difficulty. • By doing this, we are able to measure the ability of students who are at different levels ability (low, mid, and high).

19. Substantive Validity • Items do not match the student ability well. • It is difficult to establish which items should be more difficult, since each individual will have personal difficulties. • There should be a greater spread of difficulty.

20. Rasch Model • The previous slide showed an output table from Winsteps, which is based on the Rasch mathematical model. • We don’t all have this program or the ability to use it. • There is a simpler way to examine our student scores using Excel, which most of us already have.

21. Steps to Examining Scores • Open an Excel page • Input the student scores for each item and their total score. • You do not need to input student names. Simply using letters will work. • Using Excel, array the scores from highest to lowest. • Examine the scores. Step by step instructions linked on http://www.youtube.com/watch?v=nDez8VmlN9A

22. Classical Analysis

23. Fix Through Metamorphosis • Item one does not have consistent scores. • In order to fix this, I would expand this item, Writing Conventions, into its component parts. • Adding more items will help pinpoint the exact issues with writing conventions that give students trouble, which will in turn help me differentiate instruction based on student need.

24. Morphed Rubric, Item One

25. Interpreting the Morph • Because item one was so problematic, I can interpret that it is not very clear what the students can do to improve, even with explicit in-text notes. (As we may know) • I morphed it by expanding so that both the students and I could understand what wasn’t working in their assignments. • They can work on spelling instead of run-ons • I can work on student weaknesses with instruction

26. Hidden Issues with Rubrics • The Rasch analysis with Winsteps suggests that most of the grade scale is not necessary. • Grading on a 100 point scale suggest the ability to differentiate an essay in 100 ways (Dr. Jesunathadas). • As the Revised Rubric suggests, I would use a 1-4 point scale for each item and then report the final score as a percentage for student understanding. • The changes to the grading scale will help keep the scoring effective (no more half scores that don’t differentiate well).

27. Item 2

28. Revised Rubric

29. Morphing Your Rubric(s) • Be sure your items are aligned with objectives and department norms • Review your rubric for clear language and explicit (understandable) description of proficiency levels • Take one set of assignments and array them on Excel to find inconsistent items • Any inconsistent items should be expanded so you can pinpoint what is not working for students

30. Morph Your Instruction • If you notice there is something on the rubric that students are consistently unsuccessful at, then you can spend extra time on that in class instead of on items they have already mastered. • The morphed rubric will help you measure whether your instruction is helping or if you need to choose another tactic.

31. Is it Working? • Student scores on each item should be consistent with their total scores. • If not, keep morphing the rubric/instruction. • Look for growth in scores over the semester, if you are using the same rubric each time. (This also helps students understand consistent expectations) • Keep in mind student lives will occasionally impact their scores over the semester, so don’t freak out if all students don’t show improvement all the time.

32. Final Thoughts • Questions? • This presentation is linked at ajahenriquez.wordpress.com • A screencast of how to manipulate data in excel are available at • http://www.youtube.com/watch?v=nDez8VmlN9A

33. References • Bizzell, P. (1982). Review: College Composition: Initiation into the Academic Discourse Community. Curriculum Inquiry, 12(2), 191-207. • rubric. (n.d.). Collins English Dictionary - Complete & Unabridged 10th Edition. Retrieved October 20, 2011, from Dictionary.com website: http://dictionary.reference.com/browse/rubric