Puzzling over Rubrics: A Faculty Workshop on Finding the Right Tools for Assessment

1. Puzzling over Rubrics: A Faculty Workshop on Finding the Right Tools for Assessment DelanaGajdosik-Nivens, Judith Garrison, and David Wheeler

2. Scientific Teaching in STEM • “Scientific teaching” (Handelsman 2007) recommends that: • Faculty review and revise courses based on evidence that students are • learning ways of science • developing concepts • developing competencies • Faculty adopt appropriate assessment tools to evaluate TWE the student has met the goals • Facts, concepts, skills, attitudes and motivations • Aligned with learning goals • Assessment should be both formative and summative • Student-centered • Student background important • Faculty set clear and measurable goals • Frequently measured • Quantitative data - statistical analysis • Qualitative data • break into manageable units and define coding categories • search for patterns, quantify • interpret and synthesize • Valid and repeatable measures • Ideas and results are peer reviewed - formally and/or informally.

3. Backwards Design Process: Lesson/Topic Goals and Objectives before Activities Understanding By Design – (Developed by Grant Wiggins and Jay McTighe, 2002)

4. Example Methods of Assessment Indirect Direct Tests Assignments Observations Presentations Essays/reports Interviews Portfolios Surveys Peer-Assessment Assessment Gradient (Janet Batzli, UWM and Tammy Long MSU) DIFFICULT HIGH Interviews, Portfolios Potential for Assessment of Learning RUBRICS Essays, Reports, PBL, POGIL Ease of Assessment Short Answer, Surveys, Observations Models, Concept Maps, Quantitative Responses EASY LOW MC, T/F, Clickers

5. Additional points to remember • Just as assessment methods are selected based on their appropriateness for the particular goals for students, so it goes with developing rubrics. • There is no one right rubric to use. • Assessment rubric scores do not necessarily imply an assignment or course grade. “Grading is measure of students’ success in answering questions correctly--that is what students know or do not know; research based analysis helps explain why students do or do not know” Ebert-May, Diane, Jane Batzliand HeejunLim. “Disciplinary Research Strategies for Assessment of Learning.” BioScience53.12 (Dec 2003): 1221-1228. • http://www.serve.org/uploads/publications/HowtoAssess.pdf • http://www.aacu.org/resources/assessment/index.cfm • http://visionandchange.org/ • Diane Ebert-May, Jan Hodder, Kathy Williams, and Doug Luckie 2004. Pathways to scientific teaching. Frontiers in Ecology and the Environment2: 323–323. • Handelsman, J., Miller, S., Pfund, C., & Wisconsin Program for Scientific Teaching. (2007). Scientific teaching. Madison, Wisconsin: Wisconsin Program for Scientific Teaching. • HANDOUTS: Rubrics for Problem Solving, Critical Thinking, Inquiry and Analysis, Scientific Reasoning, Quantitative Literacy, Teamwork

6. CORE A2 and D SLOs • A2: Students will demonstrate the ability to use mathematical information and concepts in verbal, numeric, graphical or symbolic forms to solve problems • D: Students will demonstrate a collegiate-level understanding of the method by which scientific study is conducted and, students will accurately evaluate data in scientific reasoning problems

7. Student Learning Outcomes (SLOs) What do you want your students to know, or to know how to do, when they have completed your program?

8. English Major SLOs 1. Students will demonstrate writing skills at the stylistic, structural, and grammatical levels. Students will demonstrate that they can develop a thesis, with a sustained argument that supports that thesis. 2. Students will demonstrate a clear understanding of primary literary texts and a familiarity with the culture, genre, and place in literary history from which they come. 3. Students will demonstrate proficiency in discipline-specific skills: literary research, literary analysis and interpretation, oral communication, facility with the vocabulary of the discipline.

9. Assessment Measurements How do you know that your students know or can do what you set in your SLOs? Develop measurements specific to your outcomes. Make those measurements easy to execute. Common measurements: capstone course products, standardized tests, student portfolios, exit interviews and/or exams.

10. English Measurement by SLO • Evaluations of final essay in required capstone course, ENGL 5990—both by the course instructor and collaboratively by the English Assessment Committee. • Subtest scores on the ETS Major Field Test in English, a nationally-normed exam. Collaborative faculty evaluations of capstone oral presentations. • (Research) Performance on capstone project—both written and oral. (Analysis) Performance on capstone project and on analysis subtest of MFT. (Oral comm.) Capstone oral presentation. (Vocabulary) Test given in required ENGL 3010 course.

11. Student Learning Outcome #1: Students will demonstrate writing skills at the stylistic, structural, and grammatical levels. Students will demonstrate that they can develop a thesis, with a sustained argument that supports that thesis. Stylistic Criteria: • Demonstrates overall sophisticated prose • Uses varied sentence patterns • Employs appropriate word choice • Writes clearly

12. Structural Criteria: • Maintains recognizable organizational pattern • Writes unified and coherent paragraphs • Introduces points early in paragraphs in clear topic sentences • Guides reader with mapping statements and transitions Grammatical/Mechanical Criteria: • Is adept in sentence construction • Uses appropriate punctuation • Spells words correctly Thesis and Argument Criteria: • Presents an interesting and significant interpretive problem • Has a strong thesis in response to the problem • Makes own argument with awareness of alternative positions Supports argument effectively with textual detail

13. Student Learning Outcome #3: Students will demonstrate proficiency in discipline-specific skills: literary research, literary analysis and interpretation, oral communication, facility with vocabulary of the discipline. Literary Analysis Criteria: • Has an appropriate critical voice • Offers original, perceptive reading of text • Employs recognizable theoretical framework Literary Research Criteria: • Has conducted thorough research of topic, using a variety of appropriate sources • Uses sources with sophistication and purpose • Integrates source material successfully into paper • Properly cites sources

14. Oral Presentation Rubric For each of the learning outcomes below, indicate how strongly you agree with the statement. A score of “5” indicates “strongly agree,” “4” indicates “agree,” “3” indicates “somewhat agree,” “2” indicates “disagree,” and “1” indicates “strongly disagree.” If the oral presentation does not provide adequate evidence with which to evaluate a particular outcome, leave it blank.  1. Student demonstrates proficiency in oral communication 1 2 3 4 5  2. Student demonstrates facility with the vocabulary of the discipline 1 2 3 4 5 3. Student demonstrates proficiency in literary research 1 2 3 4 5  4. Student demonstrates skill in literary analysis and interpretation 1 2 3 4 5  5. Student shows a developed thesis and supporting argument 1 2 3 4 5 6. Student demonstrates a clear understanding of primary literary 1 2 3 4 5 text(s) and a familiarity with the culture, genre, and place in literary history from which it comes Other Comments:

15. Rubric for First Class Information Literacy • Rubric is for program assessment • Not for assigning student grade • Provides useful information, but not precise • Based on VALUE rubric from Association of American Colleges & Universities • Applied to research assignment in core class • Aiming for “Developing” level

16. SLO 2.1: Determine the nature and extent of information needed to complete a project or solve a problem.

17. SLO 2.1: Determine the nature and extent of information needed to complete a project or solve a problem. • Reflected in the thesis statement The telephone has developed massively over the centuries, becoming one of the most relied on inventions allowing people to communicate with others at ease. ~~~ As the popularity of social networking sites increase, the communication between people has changed from face-to-face interactions to screen-to-screen interactions. ~~~ When students are exposed to graphic documentaries that show the dangers of texting and driving, they become more responsible drivers.

18. SLO 2.1: Determine the nature and extent of information needed to complete a project or solve a problem. • Thesis statement & relevant sources As the popularity of social networking sites increase, the communication between people has changed from face-to-face interactions to screen-to-screen interactions. Efraim Turban, et al. "What Drives Social Commerce: The Role Of Social Support And Relationship Quality." International Journal Of Electronic Commerce 16.2 (2011): 69-90. Business Source Complete. Web. 25 Aug. 2013.

19. SLO 2.1: Determine the nature and extent of information needed to complete a project or solve a problem.

20. “Closing the Loop” • Revised type of sources required • credible vs. scholarly • Revised writing prompt to focus on evidence instead of sources • Added more practice in refining topics to seminar class • Topics stated as questions • Thesis statements reflecting key concepts

21. Web Resources • Internet Resources for Higher Education Outcomes Assessment (North Carolina State University). • opened practices: a community of practice for teaching and learning with open/community-source tools. • VALUE: Valid Assessment of Learning in Undergraduate Education • Rubrics Repository College of Charleston