Strategies, Tips, and Tools for Facilitating Learning Outcomes Assessment in Student Services

1. Student Learning Outcomes Strategies, Tips, and Tools for Facilitating Learning Outcomes Assessment in Student Services Jerry Rudmann, Irvine Valley CollegeFebruary 2008

2. Overview - Student Services • Fine-tuning assessment • Tips for writing survey items • Focus groups • Helpful technology tools • Clickers - promote active learning and record SLO information • Rubric generators - a way to measure most anything • PDF Acrobat forms - autoscoring and recording student input • Portfolios - making students responsible and reflective • Scanning - some ideas • Tracking software - organizing all this stuff • Several options / strategies for making SLOs meaningful • SSO versus SLO • Problem focus • Less is better • Use what you already have • Think of SLOs in the context of student development • Qualitative assessment in OK • Other…?

3. Some Options / Strategies for Making SLOs Meaningful • Address “robust” SLOs (overarching outcomes) • Problem focus • Less is better • Share SLOs with students • Use what you already have • Think of SLOs in the context of student development • Qualitative assessment is OK • SSOs vs. SLOs…

4. General Tip 1: Problem Focus Approach • What competencies do students have difficulty mastering? • Focus SLO activities on problem areas.

5. General Tip 2: Keep It Simple But Meaningful • Corollary - Often, less is better.

6. General Tip 3: Student Development Approach • Student development • Academic self-efficacy (Bandura) • Academic self-regulation • Campus involvement (Astin) • Mentoring professor studies • Student Services DO help student success

7. Surveys

8. Students self-rate their competencies on program or college level learning outcomes. Students’ satisfaction with various student services. Surveys - SLO Uses

9. Types of Questions Open-ended – respondents answer in own words Closed-ended – respondent limited to a finite range of choices

10. Types of Questions Open-ended Flexible Hard to code answers Good for preliminary work to finalize a survey Closed-ended Easier to code answers, process and analyze Hard to write good closed-ended items

11. Item Format Visual Analogue Scale Food in the cafeteria is… Poor_ _ __ _ _ _ _ _ _ _ _ _ _ _ _ _Excellent Likert Scale Food in the cafeteria isoutstanding! SDDN A SA (Strongly Agree) (Disagree) (Neutral) (Agree) (Strongly Agree)

12. Nine tips for designing and deploying a survey Don’t call it a survey Provide a carefully worded rationale or justification at the beginning Group items by common format Start with more interesting items Put demographic items last Mix in negative wording to catch acquiescence (aka “response set”) Automate scoring when possible If asking for sensitive information, use procedures designed to assure anonymity Always, always, always pilot test first

13. Survey Administration Methods Face to Face Written Group administration Mail Computerized http://research.ccc.cccd.edu Password protected Validation rules Branching and piping Telephone

14. Focus Groups Focus groups can be especially insightful and helpful for program and institutional level learning outcome assessment. Have your college researcher provide some background materials. Focus Groups: A Practical Guide for Applied Research By Richard A. Krueger, Mary Anne Casey The RP Group sponsored several “drive in” workshops over the last few years.

15. Goal for This Section • Technology Uses • Technology ToolsExpected Outcome: Be able to select and use technology-based approaches to assess student learning outcomes

16. Assessment Challenges • Engaging Students in Self-Evaluation • More Efficient Assessment

17. Some Technology Tools • Online Rubric Builders • eLumen (SLO Assessment/Tracking) • Classroom Responders (“Clickers”) • Scannable and Online Tests and Surveys • ePortfolios • Adobe Acrobat Forms • Excel Spreadsheets

18. Rubrics • Way to measure the heretofore immeasurable: products and performances. • A rubric breaks the assessment into important components. • Each component rated along a scale well-labeled scale.

19. Let’s Develop an Assessment Rubric for a Resume

20. Chocolate Chip Cookie Rubric Chocolate Chip Cookie Rubric

21. Rubrics are Good! • Facilitate staff dialogue regarding satisfactory performance. • Create a more objective assessment. • Make expectations more explicit to the student. • Encourage metacognitive skill of self-monitoring own learning. • Facilitate scoring and reporting of data.

22. Online Discussion Rubric http://www.uas.alaska.edu/sitka/IDC/resources/onlineDiscussionRubric.pdf

23. Design Your Own Rubric • Please work in groups and use the worksheet in your packet to design a scoring rubric for assessing one of the following: • Coffee shops • Syllabi • Customer service at retail stores • Grocery stores • Online courses

24. Online Rubric Builders • Rubrics to guide and measure learning • Tools • Rubistar http://rubistar.4teachers.org • Landmark Rubric Machine http://landmark-project.com/rubric_builder

25. Rubistar Art History Rubric Rubistar

26. Rubric Builder Screen Shot

27. Adobe Acrobat Forms • Make form using MS Word • Import form and save as PDF form • Adjust the fields • Add fields to tally sub scores and total scores

31. How Do You Report Results?

32. eLumen to Assess SLOs • Reduce Time Spent Creating Reports • Assess Course, Program, and/or Degree-Level Outcomes • Share Assessment Rubrics Across Classes and Programs • View Individual or Aggregated Results • Use Online or Offline http://www.elumen.info

33. Use Online or Offline

34. Criterion-Based Assessment • Rubrics are attached to each SLO Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

35. Rubrics Describe Criteria • Writes prose clearly Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

36. Library of Degree-Level SLOs Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

37. And Rubrics Link to SLOs Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

38. from the Science committee from the Biology Department from the faculty committee on critical thinking from the faculty committee on communication skills Science and Gen Ed SLOs/Rubrics Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

39. Scorecard for All Students in the Course Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

40. Class Scores by Student Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

41. Aggregated Data for Course Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

42. Course Aggregates by Program Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

43. Classroom Responders • Engage students • Monitor student understanding • Quickly and easily collect and store assessment data • Use publisher item banks or create your own

44. Renaissance Classroom Response System PBS Demo

45. Most valuable tip is… • Finding ways to use technology to make SLO and SSO assessment easier and more efficient • Concentrating SLO work on skills students have difficulty mastering • Building SLOs around student development (self-efficacy, goal clarity, etc.)

46. Renaissance Learningfor clicker training resources http://www.renlearn.com

47. Scanning Technology • A way to gather survey input from students • A way to test students’ knowledge http://www.scantron.com and http://www.renlearn.com

48. Surveys and Tests • Online or Scannable • Surveys • Pre and post surveys of student self evaluation of progress • Gather stakeholder (faculty, business community leaders, advisory groups) input on expected learning outcomes • Student satisfaction with service (SSO) • Quizzes/Tests • Practice and graded

49. Some Survey Software Options • Scannable surveys and quizzes - Optical Mark Reader by Remark (OMR Remark) • Need software and a Fujitsu scanner • Use word processor to create scannable bubble-in surveys or answer sheets. • Produces item analysis output. • Online survey tools • eListen (Scantron Co.) • SelectSurvey.net http://www.classapps.com/SelectSurveyNETOverview.asp

50. Excel Spreadsheets Example of autoscoring and record keeping in a Japanese Program.