The Nuts and Bolts of Outcome Assessment

1. The Nuts and Bolts of Outcome Assessment Terri Manning Denise Wells Planning and Research Central Piedmont Community College

2. Much confusion! • There are so many words being used in education right now that involve some sort of measurement that people are generally confused, throw up their hands and walk away. • What are the differences if any between testing, measurement and assessment? • Are objectives and outcomes the same thing? • What is the difference between a learning outcome, an outcome target, an educational output, an outcome objective….. and the list goes on and on……

3. Our Task…... 1. To explain all these definitions (with examples) so you can understand them. 2. To prepare you to guide your department/unit in establishing and measuring outcomes.

4. Stages of Grief for Outcome Measurement Stage 5 Acceptance & adaptation Challenge & competition Catalyst - Proactive Stage 4 Depression Compliance - Passive reactive Stage 3 Bargaining - no time/no money Seek outside sources Stage 2 Anger and antagonism Resistant & Reactive Disbelief & Denial Paralysis - Passive resistance Stage 1

5. In Education… • We’ve learned that things come and they go • Most of these trends are purely academic exercises • What we do on a daily basis doesn’t change regardless of the mission, vision and goals • There will be new “edu-babble” things to do next year

6. We have moved from: a culture of unexamined assumptions a culture of implicit, individually held hopes, preferences and beliefs a model of higher education as primarily a quantitative, additive process To: a culture of assessment and evidence a culture of explicit broadly shared goals, criteria and standards a model that is fundamentally qualitative and transformative Changes in Higher Education

7. We have moved from: a culture that tends to ignore costs a teaching culture which ignores what is known about human learning a culture that emphasizes and privileges individual struggle for private advantage To: a culture that attempts to realistically account for direct, deferred and opportunity costs one which applies relevant knowledge to improve learning one which encourages collaboration for the common good and individual advancement Changes in Higher Education, continued

8. Long Story Short…. • Accountability in higher education is here to stay! • Measuring Outcomes is here to stay and makes good sense! • The southern region of the country is way ahead of the game……

9. What SACS Says We Must Do • Comprehensive Standard 3.3.1 states “The institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results.” • Comprehensive Standard 3.4.1 The institution demonstrates that each educational program for which academic credit is awarded (a) is approved by the faculty and (b) establishes and evaluates program and learning outcomes. • Comprehensive Standard 3.5.1The institution identifies college-level competencies within the general education core and provide evidence that graduates have attained those competencies.

10. How it Works in Most Schools…

11. Mission Statement • It is the first step in the planning process • It establishes an identity for the organization that is more expansive than what one individual or small group can bring to the organization • It has to be an enduring vision of future direction and values • It identifies the organizations worthwhile work

12. Mission Statement • Helps the organization accomplish goals, receive recognition, accomplish set tasks, observe program success, move the organization forward and earn respect • Is wide in scope, directional in purpose, and allows for a wide range of goals • It should be broad enough to allow for creativity, expansion and new possibilities

13. The Mission Statement Misunderstanding…. • It doesn’t necessarily drive all the daily activities of a unit. • Counselors are going to advise students into classes at a community college regardless of the college’s mission. • However, each unit needs to own the mission and set some goals to help the college meet its mission. • Therefore some of what a unit chooses to do is driven by the mission. • People who say “everything we do is driven by the mission” are either naïve or don’t really know what is going on across the college.

14. What is a Goal • A statement of a quantifiable desired future state or condition • It is driven by the mission • A goal: • lacks deadlines • is usually long-range • is relatively broad in scope • provides guidance for the establishment of objectives (the specifics)

15. What is a Goal • It is reported on anecdotally • It is accomplished by activities • It takes several goals to accomplish a mission statement • It takes several objectives to accomplish a goal

16. Due to the nature of goals, a unit may never accomplish a goal… …but the unit makes progress toward the fulfillment of a goal.

17. We are used to setting goals • But goals are not outcomes • Outcomes are program-specific • Outcomes represent a new way of thinking • Outcomes have become widely accepted by our various publics • They are here to stay • We used to measure ourselves by our activities

18. Program Outcome Model INPUTS ACTIVITIES OUTPUTS ResourcesServicesProducts or Results of Activities Staff Education (classes) Numbers served Buildings Services FTE (input next year) Facilities Counseling # Classes taught State funds Student activities # Students recruited FTE Constraints Laws State regulations United Way model

19. Program Outcomes Model INPUTS > > > Benefits for People *New knowledge *Increased skills *Changes in values *Modified behavior *Improved condition *Altered status *New opportunities ACTIVITIES OUTPUTS OUTCOMES (Outcomes answer the “so what” question)

20. Community College Performance Funding Indicators by Type (8 states)

21. Which Is It? • An input • An activity • An output • An outcome

22. 1st Activity

23. GED Preparation • College provides well trained faculty. • 200 students complete their GED. • Students move from public housing. • 30 courses are offered each semester. • 150 FTE are generated. • Students’ reading level improves. • Students are gainfully employed.

24. Student Services • Four counselors are hired. • Students successfully transfer courses and enter the university system. • Students receive financial aid at the university. • Students attend campus activities. • Students are able to pay back their student loans.

25. Nursing Program • 100% of students pass their state board exam. • 10 faculty members deliver 600 hours of clinical experience for students. • Two students become actively involved in the state nursing association. • Agreements are established between the college and area hospitals for student training opportunities.

26. General Education Courses • Students receive creative classroom experiences. • Faculty members receive a grant to offer multimedia opportunities to students. • Students become active participants in County civic activities. • 700 FTE are generated in core courses. • Students’ math skills improve.

27. Outcomes are ……... • Driven by the mission. • Related to overall program goals. • Specific to the teachings/activities of your program. • Determined by faculty and front-line staff. • Measured carefully and specifically.

28. Inputs through Outcomes:The Conceptual Chain Long-range Intermediate OUTCOMES Initial Outputs Activities Inputs

29. But are there different types of outcomes?

30. Different Types of Outcomes • Learning Outcomes • Program Outcomes • Administrative Outcomes

31. Definitions and Examples • Learning Outcomes: • What changes in knowledge, skills, attitude, awareness, condition, position (etc.) occur as a result of the learning that takes place in the classroom. These are direct benefits to students. • Examples: general learning skills (e.g. improved writing and speaking abilities), ability to apply learning to the work environment (e.g. demonstrate skills in co-op), program-specific skills developed or enhanced (e.g. take blood pressure.)

32. Definitions and Examples • Program Outcomes: • The benefits that results from the completion of an entire program or series of courses. Are there benefits for students who get the AAS in welding versus those who take a few courses? If so what are they? • Typical examples are: licensure pass rates, employment rates, acceptance into 4-year schools, lifelong learning issues, contributions to society, the profession, etc.)

33. Definitions and Examples • Administrative Outcomes • These are outcomes established by programs and services that have nothing to do with student learning or completion of programs. • They have to do with benefits to the faculty/staff, the department and to the college. • The English Department wants all faculty to attend one professional meeting annually so they can stay up-to-date in their field. What are the benefits to the faculty, the college, the students? • Counseling may want students to lose few credits when they transfer or to recruit a new counselor with expertise in working with first-generation students (huge population for us). What are the benefits to the counseling department, the college and students? • Facilities services may want all college units to feel that they respond quickly to maintenance needs and security issues. It they do this, what are the benefits to the college and to students?

34. What is an Outcome Objective? • A short-term, measurable, specific activity having a time limit or timeline for completion around a specific outcome • They measure outcomes and are used to show progress toward goals • They specify who, will do what, under what condition, by what standard and within what time period

35. Go Over Definitions

36. Example – The CPCC Foundation • Mission Statement (portion): To enable county residents to accomplish their educational goals.Goal: Enhance accessibility to the college through building private sector support for scholarships.Objective: The College Foundation will increase private sector contributions for scholarships in the amount of 5% by June 30, 2006. Outcome Objective: 100 additional students will receive need-based scholarships by August 31, 2006.

37. How to Set Objectives • There’s no magic number • e.g. 80% or 90% • What is reasonable? • What can you afford? • What realistically can your staff accomplish? • What percent shows you’re not committed and what percent shows you’re naïve?

38. How to Set Objectives • Examples: • Fifty percent of students will be able to communicate effectively in writing (complete the writing exam with a grade of 60 [D] or better) • By the end of the spring term, 95% of faculty and staff will have completed 20 contact hours of professional development (workshops, college courses, conferences, onsite trainings, etc.)

39. More Realistic • Seventy percent of students will be able to communicate effectively in writing (complete the writing exam with a grade of 75 [C+] or better) • By the end of the spring term, the professional development office will increase their offerings for faculty and staff by 10% over what was offered last year (workshops, college courses, conferences, onsite trainings, etc.)

40. How to Set Objectives • The first time you set objectives, be conservative • Allow yourself a pilot semester or year to determine the appropriate levels of change that can be expected • Don’t pull a rabbit out of your hat (e.g. let’s grow enrollment by 10%) • May need to benchmark (what does it mean)

41. Fall Curriculum Enrollment at CPCC (1.2%) (1.5%) (2.6%) (1.6%) (1.7%) (1.5%) (1.8%) (1.6%) Should we set a goal for a 5% enrollment growth for next fall?

42. Administrative Objectives • Many units do not directly serve students or they want results within their units that are not truly outcomes. • They want to improve services or approach an old problem in a new way. • They want to become more efficient and effective. • They will set administrative objectives.

43. My Administrative Objectives 1. 80% of faculty/staff responding to the faculty/staff survey will perceive that Planning and Research responds quickly to their requests for data. 2. 80% of faculty/staff responding to the survey will perceive that Planning and Research makes a significant contribution to the College. 3. 80% of faculty/staff responding to the survey will perceive that Planning and Research contributes to the effectiveness of CPCC. 4. 80% of faculty/staff responding to the survey will indicate that Planning and Research produces enough reports to meet the planning and information needs of faculty and staff.

44. Why is This Hard? • Because it is education • Because the best results may not happen for years • Because we are so busy doing what we are doing…. we forget why we are doing it

45. Let’s Look at Healthcare • When you have strep throat and go to the doctor for your antibiotics (your intervention) • What are your intended outcomes? • Would the doctor ever tell you: • We believe curing disease is a developmental process • We believe there is value in the activity of taking pills and receiving shots • We’re not sure if you’ll get better, it is how all doctors have treated the disease since we learned about it • What would you say to that?

46. Health Outcomes Are Easy • Your health faculty have been doing this for many years • Everything in healthcare is about outcomes • People go to doctors, receive treatment, endure surgery and physical therapy – only for the outcomes… no other reason

47. How to Measure Program/Student Outcomes

48. Identifying Outcomes • Faculty/staff in an area are often the worst in defining outcomes because they are too close to the subject.

49. Sources of Ideas for Outcomes * program documents * program faculty and staff * national associations/credentialing boards * key volunteers * former students * parents of students * records of complaints * programs/agencies/employers that are the next step for your students * other colleges with similar programs, services and students as yours * outside observers of your program in action

50. How Often • Should we measure objectives or student learning outcomes every year? • When does measurement become too time consuming? • Units need time to put into effect the changes made as a result of outcome assessment before they are thrown back into another cycle. They need time to reflect on changes and results.