Writing Effective Course Assessment Plans

1. Writing Effective Course Assessment Plans Kim Anderson ASLO Subcommittee Chair Fall 2009

2. Why Assessment Plans? • "the systematic collection, analysis, interpretation, and use of information to understand and improve teaching and learning" (Tom Angelo). • Facilitates periodic, not episodic assessment of student learning and program outcomes • Serves as a planning management guideline for program review process • Leads to improvement in student learning, retention and program completion • Enables faculty to play central role in managing student learning • Facilitates collaboration within and among departments • Required by accreditation

3. Assessment Plan DevelopmentThings to Consider • Time • How consistent is the evaluation? • How robust is the final assessment information? • How will this process meet the standards set by the college’s planning process? • How effectively does this process provide guidance to our stakeholders? • How well can this information be used to improve student learning? • How will our process meet the standards set by ACCJC?

4. Key Elements • Common Methods and Criteria • In total • In part • Common core of evaluation standards • Discussions • Regularly scheduled time (department meetings, work group meetings, planning days, school meetings) • Participation (full-time and part-time) • Embedded Assessments • Decide on course work • Established assignment(s) and incorporate

5. Outcomes Assessment Management • Internal Management Structure • Who (identified) • When (note in assessment plan) • How (implement by course, not SLO) • Examples • Gantt Chart/Excel • Web based tool • TracDat

6. Course Outcomes Assessment Cycle • Routine Course Review Cycle - at least every six (6) years • Starting up will require shortened cycle for some • The Three “Ms”: • Meaningful • Measureable • Manageable

7. What Are The Components of Assessment Plans? • First, clearly state outcomes • Second, identify an assessment tool • Third, establish the criteria and level of expectation for success • Fourth, gather and analyze the results • Fifth, create a plan for improvement

8. Assessment Plan Format

9. 1. Identify intended outcomes SLOs = Core, overarching, broad statements of knowledge and skills learned in a course or in an outside of class learning opportunity • Avoid complex statements—if outcome contains more than one measurable item, then separate into different outcomes • Use an active verb indicative of expected learning or experience • State what want at end of experience—the outcomes reflect what you want to achieve • The outcomes are achievable, reasonable, and measurable • If accredited by a national organization, the outcomes address the expected student learning or service standard of such organizations • If it is part of a sequence so is well-reasoned in relationship

10. Outcome Statements Format

11. 2. Choose Assessment Task • Assessment Tools ✓Direct reasonable replications of real world tasks; capstone exam/project, pre- and post-test, rubric, licensing exams, case study {Indirect performance proxies; surveys, student interviews, employee questionnaire (cannot be used exclusively for course assessment)} ✓Quantitative numerical or statistical values; use actual numbers (scores, rates, etc.) to express quantities of a variable {Qualitative descriptive information from questions or statements that provides feedback and possible suggestions; surveys, or summary reports} Formative ongoing; to improve learning or process steps Summative final; to gauge quality end product or process • Common agree so can aggregate results; time efficient & ease of use • Who will conduct the course assessment • When time frame & cycle; rotate so not every year but all every cycle • How embed in course assignment or service experience

12. Suggested Assessments

13. Checklist for Assessment Tools • Embedded in everyday activities. • Authentically measures student skills, knowledge, or ability. • Based upon clearly defined material. • Evaluated by explicit criteria. • Collects valid evidence concerning a specific outcome. • Provides reliable results or reproducible evaluative results. • Assessment criteria and expectations have been validated through discussion with colleagues. • Systematically administered to students. • Conforms to institutional mission. • Consistent with department’s program outcomes and the college’s Instructional Program Outcomes. • Cost effective concerning resources, supplies and administration.

14. 3. Establish criteria/expectedlevel of achievement • Minimum expectation for achievement (%, fraction, actual number); 1st time determine a majority then base expectations on previous assessment information when available • What is the success level in regards to the assessment tool/task (“satisfactory” on the rubric, a passing grade, numeric score, narrative indicator) • Which students count in data (successful completion of course only, all queried, representative sample, random sample)

15. Sample Document Layout

16. 4. Gather and analyze results of assessment • Gather, organize, aggregate, analyze, and report • Process protocol and who’s involved; periodicity; deadlines; regular cycle in conjunction with routine course review cycle dates • Discussions at various meetings – need time • Aggregate • Within the course/department/program • Office of Institutional Effectiveness • Analysis and evaluation • Through discussion with vested individuals, which can mean within and outside of the department/program • Was the data valid and helpful; what does it say that is working well; what could be improved • Discuss the discipline issues

17. 5. Create a Plan • Based on results ascertain a step or steps to improve student learning • Curriculum (content emphasis, sequencing, assignments) • Pedagogy (individual/group work, technology, learner styles) • Assessment (SLO, assessment tool) • Establish definite actions and who is responsible • Determine time frame for implementation • Identify follow-up process • On-going assessment cycle

18. Outcomes Assessment Loop

19. Capstone • Exam or Essay • Same in all sections • Shared questions that assess SLOs • Common essay topic but with agreed upon scoring system (rubric) • Project • Individual = culminating/summative • Collaborative = formal/informal, typically lab settings • Agreed upon scoring system (rubric)

20. What is a Scoring Rubric? A rubric is a continuum of scoring categories and guidelines for teachers to evaluate and to give feedback to individual students AND to evaluate groups of student workto assess for course effectiveness.

21. Essentials of a Good Scoring Rubric • Criteria must align with the SLO by identify the various characteristics of the outcome. • Scoring categories specify various levels of achievement in each characteristic. • Assessment task should mirror real-life challenges as appropriate. • Two independent raters will arrive at a similar score for a given response based on the scoring rubric.

22. Components of a Scoring Rubric • Assignment = task components of a product or performance/clear definitions of each characteristic to be assessed • Levels of Achievement = scale of success by letter grades, numbers, or narrative • Evaluation = criteria standards or competencies • Documentation

23. Assignment = task • Product • Essay • Portfolio • Final project • Shop product • Document/Map • Performance • Speech • Dance/Act • Play an instrument/Sing • Skill activity • Lab procedure Elements or characteristics of the assignment

24. Achievement = scale • Letter Grade A, B, C, D, F • Numbers 1-3 (simple) ; 1-4; 1-5 (detailed) • Narrative “beginning, developing, competent, exemplary”; many versions

25. Evaluation = criteria • Evaluation criteria & standards • Demonstration of requirements • Evidence of specific progress • Feedback on competencies • Miss, meet or exceed standards

26. Documentation = samples • Product • Performance • Letter grade • Number scale • Narrative explanations

27. “Product” ExcerptExcel Spreadsheet Final Project

28. “Performance” SampleGolf Bunker Shot

29. “Grade” SamplePolitical Science Essay • A = The A-range essay implies a strong argument and provides convincing specific support from the various readings. The writer demonstrates mature command of language through a variety of sentence structures, word choices, quotes or paraphrases from the readings (consistently cited correctly). Control of usage and mechanics, despite occasional flaws, contributes to the writer’s ability to communicate the purpose of the paper. The writer thoroughly understands the concepts/theories involved and through the essay can convince others of his/her viewpoints or helps make the reader aware of something completely new or original. These are powerful due to organization and creativity. • B = The B-range essay shows effort and promise for the writer. It presents a thesis (argument) and often suggests a plan of development that is carried out effectively. Mastery of the readings/theories may not be fully indicated with the use of quotes or paraphrases, but the writer provides enough supporting details, makes competent use of language, and sometimes varies sentence structure. Occasional errors in usage and mechanics do not interfere with the writer’s ability to communicate the purpose of the paper. • C= The C-range essay presents a thesis (argument) and often suggests a plan of development which is generally c arrived out. The writer may or may not have completed all of the required readings and utilizes generalizations or list for support. Command of the theories under consideration is weak or shaky. Sentence structure tends to be repetitious, and errors in usage and mechanics sometimes interfere with the writer’s ability to communicate the purpose of the paper. • D = The D-range may present a thesis (argument); however, the plan of development is usually not carried out. This indicates the writer may have spent little time with the readings or thinking about the concepts involved. The writer provides support that tends to be sketchy and/or illogical. Sentence structure is simplistic, repetitious and occasionally awkward. Language is often inappropriate in tone or style. Errors in usage and mechanics are frequent. • F = This paper presents a thesis that is vaguely worded, weakly asserted or there is no central argument present. Support, if any, tends to be rambling and superficial. Sentence structure is difficult to follow and errors in usage and mechanics interfere with the writer’s ability to communicate the purpose of the paper.

30. “Numeric” SamplePhilosophy - Critical Thinking

31. “Narrative”ExcerptEnglish 1/ESL 1 Final Essay Examination Sampling of Narrative Criteria A. Respond to the writing task. The paper must respond to the specific topic and the reading and must show understanding of the reading. A paper simply restating the content of the reading is not an adequate response. The paper must be at least 400 words long. B. Treat the reading fairly. The paper must avoid plagiarism by attributing the words and ideas of others to their source. Borrowed words and combinations of words must be set off by quotation marks. The paper must not misinterpret or misquote others’ words. C. Maintain a consistent style and sense of audience. The paper must not have abrupt or confusing shifts in person, voice or tense. E. Be organized logically and coherently. The paper must be organized according to a discernible plan, with the main points supporting the thesis. The reader must be able to follow the plan with ease. I. Adhere to the conventions of Standard English. The paper should have no more than minimal errors in spelling, punctuation, capitalization, apostrophe, and case, and should use standard forms of verbs, adjectives, and adverbs.

32. Course Assessment PlanRecap • Establish periodic assessment of student learning in courses to improve student success • Column 1: SLOs • Column 2: Assessment Tool-What, How, Who, When • Column 3: Achievement/Success-Expected Achievement, Success Level, Students Included