Program Level Assessment

1. Program Level Assessment Workshop for College of the Redwoods Fred Trapp August 18, 2008

2. Assessment of Program Effectiveness • What the program will do or achieve • Curriculum • Retention • Graduation • Placement • Satisfaction (graduate and employer)

3. Assessment of Student Learning Outcomes • What students will do or achieve • Direct methods of assessment

4. Direct Measures of Learning • Capstone experience • Standardized tests • Performance on national licensure certification or professional exams • Locally developed tests • Essay questions blind scored by faculty • Juried review of projects • Externally reviewed exhibitions performances • Evaluation of internships based upon program learning outcomes

5. Indirect Measures of Learning • Alumni, employer, and student surveys (including satisfaction surveys) • Exit interviews of graduates and focus groups graduate follow up studies • Retention and transfer studies • Length of time to degree • Graduation and transfer rates • Job placement rates

6. Steps of Implementation • Step One: Developing Assessment Plans • Step Two: Developing Implementation Strategies • Step Three: Collecting Data, Analyzing & Reflecting on its meaning • Step Four: Implementing Change as the result of assessment

7. Assessment Language • Program Outcomes/ Topic Areas • Student Learning Outcomes • Means of Assessment/ Evidence • Criteria for Assessment • Standards of Success/ Quality of Performance

8. Program Outcome Topic & Student Learning Outcomes Communication (program outcome topic) Write Relate Speak Listen Participate Student learning outcomes

9. Means of Assessment/Evidence- Circumstances (learning events) • Assignments (in class and out of class) • Feedback on practice • Self evaluation • Peer evaluation • Role Play • Pre Tests • Simulation

10. Means of Assessment/Evidence- Deliverable or Performance Goal Outcome Outcome Outcome Outcome Outcome Learning events Object (deliverable or performance)

11. Means of Assessment/Evidence- Deliverable or Performance Communication Write Relate Speak Listen Participate Speech Essay Interview

12. Criteria for Assessment Communication Write Relate Speak Listen Participate criteria Verbal delivery Speech Nonverbal delivery Organization Content

13. Standards of Success/ Quality of Performance Communication Write Relate Speak Listen Participate Quality of Performance Verbal delivery Speech Nonverbal delivery organization content

14. Student Learning Outcomes • Students should be able to <<action verb>> <<something>> • Learner Centered • Specific • Action oriented • Cognitively appropriate for the program level

15. COMPREHENSION EVALUATION APPLICATION ANALYSIS SYNTHESIS KNOWLEDGE Associate Classify Compare Compute Contrast Differentiate Discuss Distinguish Estimate Explain Express Extrapolate Interpolate Locate Predict Report Restate Review Tell Translate Analyze Appraise Calculate Categorize Classify Compare Debate Diagram Differentiate Distinguish Examine Experiment Inspect Inventory Question Separate Summarize Test Arrange Assemble Collect Compose Construct Create Design Formulate Integrate Manage Organize Plan Prepare Prescribe ProducePropose Specify Synthesize Write Appraise Assess Choose Compare Criticize Determine Estimate Evaluate Grade Judge Measure Rank Rate Recommend Revise Score Select Standardize Test Validate Cite Count Define Draw Identify List Name Point Quote Read Recite Record Repeat Select State Tabulate Tell Trace Underline Apply Calculate Classify Demonstrate Determine Dramatize Employ Examine Illustrate Interpret Locate Operate Order Practice Report Restructure Schedule Sketch Solve Translate Use Write

16. COMPREHENSION EVALUATION APPLICATION ANALYSIS SYNTHESIS KNOWLEDGE Associate Classify Compare Compute Contrast Differentiate Discuss Distinguish Estimate Explain Express Extrapolate Interpolate Locate Predict Report Restate Review Tell Translate Analyze Appraise Calculate Categorize Classify Compare Debate Diagram Differentiate Distinguish Examine Experiment Inspect Inventory Question Separate Summarize Test Arrange Assemble Collect Compose Construct Create Design Formulate Integrate Manage Organize Plan Prepare Prescribe ProducePropose Specify Synthesize Write Appraise Assess Choose Compare Criticize Determine Estimate Evaluate Grade Judge Measure Rank Rate Recommend Revise Score Select Standardize Test Validate Cite Count Define Draw Identify List Name Point Quote Read Recite Record Repeat Select State Tabulate Tell Trace Underline Apply Calculate Classify Demonstrate Determine Dramatize Employ Examine Illustrate Interpret Locate Operate Order Practice Report Restructure Schedule Sketch Solve Translate Use Write Course outcomes

17. COMPREHENSION EVALUATION APPLICATION ANALYSIS SYNTHESIS KNOWLEDGE Associate Classify Compare Compute Contrast Differentiate Discuss Distinguish Estimate Explain Express Extrapolate Interpolate Locate Predict Report Restate Review Tell Translate Analyze Appraise Calculate Categorize Classify Compare Debate Diagram Differentiate Distinguish Examine Experiment Inspect Inventory Question Separate Summarize Test Arrange Assemble Collect Compose Construct Create Design Formulate Integrate Manage Organize Plan Prepare Prescribe ProducePropose Specify Synthesize Write Appraise Assess Choose Compare Criticize Determine Estimate Evaluate Grade Judge Measure Rank Rate Recommend Revise Score Select Standardize Test Validate Cite Count Define Draw Identify List Name Point Quote Read Recite Record Repeat Select State Tabulate Tell Trace Underline Apply Calculate Classify Demonstrate Determine Dramatize Employ Examine Illustrate Interpret Locate Operate Order Practice Report Restructure Schedule Sketch Solve Translate Use Write Program outcomes

18. Possible Outcomes? • Students will appreciate…. • Students will be exposed to…. • Students will demonstrate knowledge of the history, theories and applications of… • Students will attend…. • Faculty will provide students with opportunities to….

19. Possible Outcomes • Students will appreciate… • Students will be exposed to…. • Students will demonstrate knowledge of the history, theories and applications of… • Students will attend…. • Faculty will provide students with opportunities to….

20. Writing Program Level Student Learning Outcomes • Student Learning Outcomes • Students will recognize, analyze, and interpret human experience in terms of personal, intellectual, and social contexts. • Students should be able to critically comprehend, interpret, and evaluate written, visual, and aural material. Compound outcome statements will be problematic to assess.

21. Program Level Student Learning Outcomes • Students will be able to: • Design a solution to a technical problem • Interpret performance data • Demonstrate the functions of key operating systems • Articulate the ethical standards of the profession • Produce diagnostic quality radiographic images • Evaluate the advantages of alterative solutions

22. Course LevelStudent Learning Outcomes • Students will be able to: • Students will have a basic knowledge of human anatomy • Students will be able to define 200 medical terms • Students should know the history of the United States • Students will be able to translate dollars into pounds and Euros

23. Single course Course concepts Fewer confounds from outside the course detractors Faculty (indiv. & small groups) “Micrometer” precision Less fear generated in faculty Faculty professional obligation, less overhead cost No external certifications Multiple courses Program performance competencies (post grad) More confounds from outside the course detractors Faculty (small & large groups) “Two sticks & a chain” precision Potential fear generated in faculty Faculty professional obligation, but some overhead cost External certifications possible Course vs. Program Assessment

24. Identify Criteria for Assessing Program Level Outcomes • Must be shared across faculty • A way to further define the student learning outcome • Provides a common language for describing student learning • Must be outcome specific • Number of criteria will vary by outcome

25. Means of Assessment/Evidence- Deliverable or Performance + Criteria Communication Write Relate Speak Listen Participate Eulogy Sales presentation delivery content organization Oral Report

26. The Reality of Assessing Program Level Student Learning Outcomes Course Course Course Course Course Speaking eye contact gestures volume sources transitions style rate poise examples verbal variety appearance evidence conclusion organization attention getter

27. The Reality of Assessing Program Level Student Learning Outcomes Can our students deliver an effective public speech? eye contact gestures volume sources transitions style rate poise examples verbal variety appearance evidence conclusion organization attention getter

28. Program Learning Outcome Example #1 Gather factual information and apply it to a given problem in a manner that is relevant, clear, comprehensive, and conscious of possible bias in the information selected BETTER: Students will be able to apply factual information to a problem Criteria: Relevance Clarity Comprehensiveness Aware of Bias

29. Program Learning Outcome Example #2 Imagine and seek out a variety of possible goals, assumptions, interpretations, or perspectives which can give alternative meanings or solutions to given situations or problems BETTER: Students will be able to provide alternative solutions to situations or problems Criteria: Variety of assumptions, perspectives, interpretations Analysis of comparative advantage

30. Program Learning Outcome Example #3 Formulate and test hypotheses by performing laboratory, simulation, or field experiments in at least two of the natural science disciplines (one of these experimental components should develop, in greater depth, students’ laboratory experience in the collection of data, its statistical and graphical analysis, and an appreciation of its sources of error and uncertainty) BETTER: Students will be able to test hypotheses. Criteria: Data collection Statistical Analysis Graphical Analysis Identification of sources of error

31. Standards for Success/Quality of Performance • Rubric • Scale or description for assessing each of the criteria • Two to Five-point scales for each criteria • Anchored with descriptions and supported by examples • Portion of the Whole Group Which is Successful

32. Standards for Success/Quality of Performance Rubric OUTCOME: Performance Anchors: Does not meet Expectations Meets Expectations Exceeds Expectations Criteria Descriptions of Performance

33. Standards for Success/Quality of Performance Rubric OUTCOME: Performance Anchors: Not Yet Almost YES!!!! Components

34. Standards for Success/Quality of Performance Rubric OUTCOME: Performance Anchors: Basic Essential Exemplary Components

35. Student Learning Outcome Criteria Test Questions 1 1 2 3 4 2 5 6 7 8 9 3 10 11 12 4 13 14 15 16 17 18

36. Student Learning Outcome Criteria Test Questions % correct 1 1 2 3 4 78% 2 5 6 7 8 9 56% 3 10 11 12 82% 4 42% 13 14 15 16 17 18

37. Student learning outcomes Course 1 Course 2 Course 3 Course 4 Course 5 1 X X X 2 X X X 3 X X X X 4 X X 5 X X X X X 6 X X X 7 X Identify Assessment Points in the Program

38. Phase 1 Student learning outcomes Course 1 Course 2 Course 3 Course 4 Course 5 1 X X X 2 X X X 3 X X X X 4 X X 5 X X X X X 6 X X X 7 X

39. Communicate Effectively Demonstrate Oral Communication Skills Program-wide Assessment Report Does not meet Meets Exceeds Verbal Delivery 20% 65% 15% 57% 30% Nonverbal Delivery 13% 58% Organization 24% 18% Is there a problem with performance in any of the criteria?