Annual Planning & Assessment

1. Annual Planning &Assessment WEAVE February 2010

2. Purpose today: • Review our planning and assessment process and why it is important • Train you on the latest version of WEAVE

3. Why do we do it? • Because we want to fulfill the College’s mission • Because we want to keep getting better • Because we want to know what works & what doesn’t • Because we want to communicate improvements • Because we must be accredited

4. SACS Accreditation • Not stating outcomes, in many cases stating tasks and action steps • More stretch targets. (Keep getting better.) • Assessment weak and not sophisticated. Can’t answer “why.” • Improvements not coming from assessment. Improvements not documented. • Not closing the loop. • Not documenting & communicating our results. FINDINGS FROM LAST SACS VISIT:

5. Assessment Process Adapted from Trudy Banta, IUPUI

6. Institutional Effectiveness Cycle Delineate Mission Discern Expected Outcomes Establish Measures/Assessment and Targets Review Actual Performance against Targets Understand your results – the why Make improvements through action

7. Keep an open mind • You do much of this every day • Logical steps but thought process is different. • Will not replace your intuition and experience. • Payoff – Keeps your mission clear. See the strides you’ve made and why you’ve made them. Ability to communicate to others.

8. Mission Statement • The mission statement links the functions of your unit to the overall mission ofUVa-Wise. • A few questions to consider in formulating the mission of your unit: • What is the primary function of your unit? • What core activities are involved? • What should those you serve experience after interacting with your unit?

9. Characteristics of a Well-Defined Mission Statement • Brief, concise, distinctive • Clearly identifies the program’s purpose • Clearly aligns with the mission of your area and the College • Explicitly articulates the essential functions/activities of the program • Clearly identifies the primary stakeholders of the program: i.e., students, faculty, parents, etc.

10. Learning Outcomes When writing Learning Outcomes, the focus must be on the stakeholders (e.g., students, faculty, staff, and others) and what they will think, know, do, or value following the provision of the service.

11. Program Outcomes • Process statements • Relate to what the unit intends to accomplish • Level or volume of activity • Efficiency with which you conduct the processes • Compliance with external standards of “good practice in the field” or regulations • Satisfaction statements • Describe how those you serve rate their satisfaction with your unit’s processes or services

12. Common Issues Addressed in Program Outcomes • Facilities & Equipment • Faculty • Financial Support • Student Preparation • Productivity • Technology • Access • Accountability • Affordability • Climate • Educational Effectiveness/Quality

13. Writing Program Outcomes Consider such questions as: • What are the most important results or impacts that should occur as a result of your unit’s activities? • What are your critical work processes and how should they function? • What does the end user experience through interaction with your unit?

14. Examples of Outcomes • Learning Outcomes • Students receiving advising services will be able to identify and utilize campus resources. • Staff and faculty will be able to use the Oracle system for purchasing. • Process statements • The number of faculty training workshops and participation rates for WebCT will relate to the needs. • The travel office will promptly process travel requests. • Satisfaction statements • Faculty will report satisfaction in the training received for WebCT. • Faculty and staff will report satisfaction with travel processing under the Oracle system.

15. Writing Outcomes Outcomes should be: • linked to the unit’s mission • realistic and attainable • limited in number (manageable) • something that is under the control of the unit • measurable and/or observable • meaningful • target key services or change points • use action verbs

16. Assessment Measures After establishing your outcomes… • Define and identify the sources of evidence you will use to determine whether you are achieving your outcomes. • Detail what will be measured and how • Identify or create, if necessary, measures that help your unit in making critical decisions about its processes and services.

17. Choosing Assessment Measures Some things to think about: • How would you describe the end result of the outcome? • How will you know if this outcome is being accomplished? • What will provide you with this information? • Where are you currently delivering the outcome? • Are there any naturally occurring assessment opportunities? • What measures are currently available?

18. Characteristics of an Effective Assessment Measure • Measurable and/or observable • You can either observe it, count it, quantify, etc. • Meaningful • If captures enough of the essential components of the objective to represent it adequately • It gives you enough information • Manageable • It can be measured without excessive cost or effort

19. Direct Measures Direct measures are those designed to directly measure: • what a stakeholder knows or is able to do (i.e., requires a stakeholder to actually demonstrate the skill or knowledge) • The benefit of programming or intervention

20. Common Direct Measures • Participation data • Observation of behavior • Culminating experiences (e.g., presentation, project, internships, etc.) • Collection of work samples (portfolios) • Pre- and post-measures • Volume of activity • Level of efficiency (average response time) • Measure of quality (average errors)

21. Indirect Measures Indirect measures focus on: • stakeholders’ perception of their level of learning • stakeholders’ perception of the benefit of programming or intervention • stakeholders’ satisfaction with some aspect of the program or service

22. Common Indirect Measures • Surveys • Exit interviews • Retention/graduation data • Focus groups

23. Achievement Targets • An achievement target is the result, target, benchmark, or value that will represent success at achieving a given outcome. • Achievement targets can be specific numbers or trends.

24. Examples of Achievement Targets • Students will achieve a score of 3 or 4 on the rubric. • Average score on rubric is a 3 or better (on a scale of 0 to 4). • 90% of the transcripts will be sent within three days. • Each employee will participate in a minimum of two training/development programs per year. • Acquisition statistics will indicate growth in the overall collections that support academic programs.

25. Putting It All Together Outcome: Improve and strengthen media relations and publications Measure 1 (Direct): Track media pitches/releases. Achievement Target: Average of 250 media pitches/releases for the year. Measure 2 (Indirect): Conduct annual readership survey of university magazine. Achievement Target: Feedback via readership survey will indicate average ratings of 7 or better on a 1-10 scale.

26. Putting It All Together Outcome: To provide a library web site that enables users to locate and use information on their own Measure 1 (Direct): Usability testing with small groups Achievement Target: Participants in usability testing demonstrate web site’s ease of use -- ### Measure 2 (Indirect): LibQUAL+ assessment program Achievement Target: LibQUAL+ results indicate faculty and student satisfaction - ####

27. Findings Findings = Assessment data On an Assessment Report, “findings” refers to a concise summary of the results you gathered from a given assessment measure. The language of this statement should parallel the corresponding achievement target Describe your results in enough detail to prove whether you have met, partially met, or not met your achievement target. It is not necessary to provide any interpretation of your data in your findings summary.

28. Findings • (Optional) Attach documents to support your data. These can include survey instruments or results, reports, committee members and minutes from meetings, etc.

29. Examples of Findings Statements • Example 1: • Achievement Target: Overall mean score of students from program meets or exceeds state average score. • Findings: The overall mean score of students from Praxis exam exceeded that of the state average score. Results: Program overall mean scaled score—91.50, State overall mean scaled score—79.13.

30. Examples of Findings Statements • Example: • Achievement Target: Decrease by 40% the number of undeclared students from the entering major (Fall 2008) to current major (Fall 2009) • Findings: 34% of undeclared students who had registered for Fall 2009 classes by the beginning of June had declared a major.

31. Examples of Findings Statements • Example: • Achievement Target: 10 campus-wide workshops conducted in November and December of 2008. • Findings: 8 campus-wide workshops were conducted in November and December of 2008.

32. Analyze your Findings • Reflect on what has been learned during an assessment cycle • Identify areas that need to be monitored, remediated, or enhanced • Three key questions are at the heart of your analysis: • What did you find and learn? • So What does that mean for your academic program or support unit? • Now What will you do as a result of the first two answers?

33. Analyze your Findings • You will want to reflect on the following areas: • StudentLearning Outcomes, if applicable • Program Outcomes, if applicable • The Assessment Process

34. Action Plan • After reflecting on the findings, you and your colleagues should determine appropriate action to improve the program. This will lead to at least one action plan. • Actions outlined in the action plan should be specific and relate directly to the outcome and the results of assessment.

35. Action Plan Recommendations • We do not recommend having an assessment report without any action plans. • SACS 3.3.1 • We do recommend that you keep the number of action plans manageable.

36. Annual Reporting • The final step in your assessment report is to consider how your program contributed to one or more of the following: • Departmental goals or imperatives • College mission, goals or imperatives

37. Take-Home Messages • You do not have to assess everything every year • Modify something already being done that is meaningful to the program • Summarize your Findings • Limit your Action Plans and keep them manageable • Consider how your program fits within theCollege’s mission • Be flexible—this is an iterative process

38. Resources • http://www2.acs.ncsu.edu/UPA/assmt/admin_examples_others.htm • http://oeas.ucf.edu/doc/adm_assess_handbook.pdf • http://www.assessment.tcu.edu/ Special thanks to Texas A& M and the WEAVE Google Group.

39. One Minute Evaluation • What was the most valuable thing you learned? • What is one question that you still have? • What do you think is the next step that your program needs to take in order to implement systematic program assessment?