Institutional Effectiveness: Developing an Institutional Philosophy for the Quest for Quality

1. National Council for Instructional Administrators April 4, 2008 Institutional Effectiveness:Developing an Institutional Philosophy for the Quest for Quality Terri M. Manning, EdD. Central Piedmont Community College

2. Institutional Effectiveness (IE) • A philosophy with a set of processes • A combination of the content from the following fields: • Assessment • Program Evaluation • Quality Assurance • Organizational Development • You combine these areas and apply the result to higher education and you get IE

3. A Definition • Institutional Effectiveness is: • an ongoing, integrated and systematic set of institutional processes that include planning, the evaluation of programs and services, the identification and measurement of learning outcomes, the use of data and assessment results for decision-making that results in improvements in programs, service and institutional quality.

4. You have had a speaker on this topic for the last several years – yet you ask for additional training on this topic.Why is this hard for institutions to accomplish?

5. Why This is Hard for Many Colleges • Colleges want someone to tell them exactly how to do this – what is expected. • They think they are doing it for their accrediting agencies. • They don’t create processes that are valuable to the institution. • They are either too simple or too complex. • They try to drive it from the top down – administration tries to tell the faculty what needs to be done. • It has no central coordination of efforts – no natural connection between processes.

6. Why It’s Hard, cont. • Their approach is like students in the classroom - “what exactly do we have to do to get an A on the test.” They completely miss the concept. They don’t “get it.” • We do not measure learning outcomes so we can say we did it. • This is not an end into itself. • It is part of a process leading to a much greater end…. Improvements in institutional quality.

7. What You Are Really Doing • Is attempting to create institutional change by: • Getting faculty to value something different • Measure themselves differently • Have a different philosophy • Reward themselves for something different • Shake up their power structure • Is it any wonder we have trouble with this?

8. Lewen’s Organizational Change Model • Unfreezing – reducing the forces that maintain the organization’s current behavior. One introduces information that points out discrepancies between “what we have” and “what we want.” This is supposed to motivate members of the organization to engage in change behaviors. • Moving – shifts the desired behavior to a new level by intervening in the system to develop new values, attitudes and behaviors through changes in processes. • Refreezing – stabilizing the new behaviors by the use of support mechanisms such as a new culture, policies, norms and structures. Source: Essentials of Organization Development & Change, Cummings & Worley

9. What Holds Us Back • Barriers to Individual and Organizational Change • Failure to recognize the need for change • We are fine like we are – no need for improvement • Habit • That is not how we have always done it • Security • My position or rank or the department’s standing could be affected • Fear of the unknown • What if we don’t meet our outcomes (our students aren’t learning), we might be held accountable for it • Previous unsuccessful efforts • We tried that before and it failed

10. What Holds Us Back, cont. • Threats to expertise • Assessment results are not going to force me to teach my class differently or to improvement • Threats to social and power relationships • The one on top may no longer be on top, no longer the favorite or in control • Threats to resource allocation • My department or program might lose funding if we do not perform as well as other departments or programs (from Greenberg & Baron, 2000; Jones, 2004; Robbins 2000)

11. Mandates for Institutional Effectiveness • In 2006, U.S. Secretary of Education Margaret Spellings released “A Test of Leadership: Charting the Future of U.S. Higher Education.” This report addresses the need to transform higher education. • “There is inadequate transparency and accountability for measuring institutional performance, which is more and more necessary to maintaining public trust in higher education.” (p. 14) • “To meet the challenges of the 21st century, higher education must change from a system primarily based on reputation to one based on performance. We urge the creation of a robust culture of accountability and transparency throughout higher education.” (p. 21)

12. Mandates for Institutional Effectiveness • “The commission supports the development of a privacy-protected higher education information system that collects, analyzes and uses student-level data as a vital tool for accountability, policy-making, and consumer choice.” (p. 22) • “Faculty must be at the forefront of defining educational objectives for students and developing meaningful, evidence-based measures of their progress toward these goals.” (p. 24) • “The results of student learning assessments, including value-added measurements that indicate how students’ skills have improved over time should be made available to students and reported in the aggregate publicly.” (p. 24)

13. Accreditation Has Changed • Over the past 3-5 years, all six accrediting agencies have made significant changes in their processes – all requiring assessment and evaluation and the inclusion of student learning outcomes. The words “institutional effectiveness” are mentioned in the criteria for most accrediting agencies. • College and universities can no longer wait until 1-2 years before an accreditation visit and “gear up” for it. • Colleges and universities are starting to get in trouble with their accrediting agencies. • It’s going to get worse before it gets better .

14. What Do The Accrediting Agencies Want Source: Gita Wijesinghe, Florida A&M

15. The Reality • At our institutions, our policies and practices are set up to perfectly deliver the results we are currently achieving. • Are they good enough?

16. I ask you: … if you are going to ask your faculty to participate in and embrace institutional effectiveness, to invest the time to develop effective processes, to assess, analyze and use results - - - ….once it is all done, what would you want to know?

17. Here is what others have said… • What characteristics should our graduates have and do they have them? • Are our students learning? • Are our students succeeding? • Are some programs having trouble? • What will programs need to succeed in the future? • What is new and innovative coming our way (by discipline)? • Where should our enrollments be (growing vs. declining)? • What are the costs?

18. Institutional Effectiveness • Is a way of thinking…a philosophy • Is the way an institution keeps its finger to the pulse of its various communities. • It is about student success and student learning. • It is mostly about an institution’s continuous quest for quality, efficiency, effectiveness and innovation.

19. Institutional Effectiveness • It answers the questions: • Are our students learning what we intended for them to learn • Can they apply what they learned in the real world • Are we serving our students well • How can we improve, innovate and create • What is the real value of an education obtained from this institution

20. Institutional Effectiveness • There is no prescription – no commonly accepted set of practices or processes • Can be culturally changing to serious institutions • Is a unique system of questions and inquiry • Means looking at the questions before we come up with answers • Means looking at the data • It facilitates a culture of evidence

21. Educating Students • Is a developmental process • Unique to each institution • Yet with some common core values • It is like parenting • Would you want anyone to give your children an exam that measured how good of a parent you were, the quality of your family life… • Your first question would be: who made the decision on what a “good parent” is or what qualities are acceptable in families. While we all hold some common values about families and children, we are very different.

22. Difficulty for Faculty • Issues to discuss/work on: • What do we value? • How do we operationally define it? • Can we all agree on a few characteristics we value and measure them? • Is that enough? • Who decides if it is enough? • How much more are we going to add to their plate without removing anything?

23. Effective Institutions • Don’t make decisions, change policy or practice without the evidence to give them direction • When they start something new, they evaluate it heavily to make sure it produces the desired effect • When things don’t work they fix it • Don’t set students up to fail (policy and practice) • They create a continuous flow of information to the opinion leaders, decision-makers and stakeholders • Anticipate change and are prepared • Listen to their front-line employees – those most in touch with their students/customers/clients (faculty and counselors/advisors)

24. Good Practices • Institutions who fully participate in IE: • Figure out why students are not progressing and make changes • Discover student barriers to success and remove them • Assess effective teaching modalities and expand and enhance them • Openly uncover their weaknesses and work toward strengths

25. It is all about… • Your students • Their success • Faculty and staff as facilitators • The quality of your institution • The responsibility we have to tax payers, students, our community and state • A desire for continuous improvement

26. It is Not About • Doing what we have always done but in a different format • A prescribed set assessment tools • Filling out forms and putting documents in order • Paying lip-service to processes • Getting through an accreditation visit • This is so much bigger than an accreditation process or agency

27. Four Most Important Elements • Support from the administration • “Buy in” and trust from the grassroots level – the development of relationships • Freedom and fairness for those being reviewed • Effective processes that work for your institution - that are supported and followed

28. Support from the Administration • If you don’t have it nothing works • It is hardest where there is the unspoken rule “no bad news” • Faculty/staff can’t fight like salmon swimming up-stream • You have to be creative and develop perseverance • You are there to get the job done • Some times it takes more than once for your message to get across

29. Support from the Administration • To obtain this support, you have to: • Know what you are doing • Develop simple and clear processes • Fit the “administrative response” into your processes • Report results to gain visibility • Show how this relates to the bottom line and will increase enrollment growth and retention

30. Buy In From the Grassroots Level • Allow the faculty to create the process themselves (as much as possible) • You have to believe that what they are doing is critical to the success of the institution • You have to understand and validate that “they don’t have time for this” • Build friendships and trust. Function as a helper. • Be a faculty advocate and be proactive for their issues

31. Buy In From the Grassroots Level • Never be perceived as the one who is “grading their performance” • Roll up your sleeves and get involved • Seek to understand their issues • Give them tools and templates – put things online • Create rewards • Realize their #1 role is to teach, not to be assessment experts

32. Freedom and Fairness • Never use data against them • Give them as much input as possible – give them flexibility when you can • Allow them to measure what they believe is important • Aggregate data when necessary • Try to get the message across “This is a process with the goal of a continuous quest for quality. It is okay if things are not perfect – then we know where strategies for improvement need to be utilized.”

33. Effective Processes • Why develop a process? • There is an expectation of a due date • There is a format to follow that utilizes standards • Processes usually have outlines and calendars • Processes can be revised to include everything important to the institution

34. Typical Processes • A Mission Statement • A Strategic Plan • Institutional Goals and Outcomes • Annual Goal/Objective Cycle for All College Units/Departments • Annual Research and Assessment Reports • Program, Learning and Administrative Outcomes • Collect through processes • General Education Assessment • Program/Unit Review • Outcome Assessment Matrices

35. Today we are going to talk about: • Two areas critical to instruction/academic affairs: • The Development and Assessment of Learning Outcomes • The Development and Assessment of General Education Learning Outcomes

36. Why are We Moving from Goals to Outcomes? • Outcomes are program-specific • They measure the effect of classroom activities and services provided. • Outcomes represent a new way of thinking • Outcomes have become widely accepted by our various publics • They are here to stay • They are skills-based variables you can observe, measure, scale or score

37. Program Outcome Model ResourcesServicesProducts or Results of Activities Staff Education (classes) Numbers served FTE Services FTE (input next year) Facilities Counseling # Classes taught State funds Student activities # Students recruited Ability of Students Constraints Laws State regulations INPUTS OUTPUTS ACTIVITIES Theory of Change Model

38. Program Outcomes Model OUTCOMES OUTPUTS INPUTS ACTIVITIES Benefits for People *New knowledge *Increased skills *Changes in values *Modified behavior *Improved condition *Altered status *New opportunities (Outcomes answer the “so what” question)

39. Inputs through Outcomes: The Conceptual Chain Long-range Intermediate OUTCOMES Initial Outputs Activities Inputs

40. Different Types of Outcomes • Learning Outcomes (can be at course, program or institutional level) • Program Outcomes • Administrative Outcomes

41. Definitions and Examples • Learning Outcomes: • What changes in knowledge, skills, attitude, awareness, condition, position (etc.) occur as a result of the learning that takes place in the classroom. These are direct benefits to students. • Examples: general learning skills (e.g. improved writing and speaking abilities), ability to apply learning to the work environment (e.g. demonstrate skills in co-op), program-specific skills developed or enhanced (e.g. take blood pressure.)

42. Definitions and Examples • Program Outcomes: • The benefits that results from the completion of an entire program or series of courses. Are there benefits for students who get the entire degree versus those who take a few courses? If so what are they? • Typical examples are: licensure pass rates, employment rates, acceptance into 4-year schools or graduate programs, lifelong learning issues, content mastery, contributions to society, the profession, etc.)

43. Definitions and Examples • Administrative Outcomes • Units/programs want to improve services or approach an old problem in a new way. • They want to become more efficient and effective. • They establish an outcome objective for the administration. • Typical examples are: • All faculty will attend one professional meeting annually so they can stay up-to-date in their field, or: • Counseling wants to recruit a new counselor with expertise in working with first-generation students, or: • Facilities services wants students, faculty and staff to feel that they are safe on campus.

44. Why is This Hard? • Because it is education • Because the best results may not happen for years • Because we are so busy doing what we are doing…. we forget why we are doing it

45. Assessment • “The assessment of student learning can be defined as the systematic collection of information about student learning, using the time, knowledge, expertise, and resources available, in order to inform decisions about how to improve learning. (p.2)” Source:Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004

46. Assessment Characteristics • It is intended not to generate broad theories but to inform action. • Educational situations contain too many variables to make “proof” possible. • Assessment gathers indicators that will be useful for decision making. • It is not limited to learning that can be objectively tested. A department can state its highest goals and seek the best available indicators about whether those goals are being met. • It does not require standardized test or objective measures. • Faculty regularly assess complex work in their fields and make judgments about its quality. • Faculty can make informed professional judgments about critical thinking, scientific reasoning, or other qualities in student work, and use those judgments to inform departmental and institutional decisions. (page 2) Source:Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004

47. Assessment Characteristics • Assessment means basing decisions about curriculum, pedagogy, staffing, advising and student support on the best possible data about student learning and the factors that affect it. • A lot of assessment is already going on in responsible classrooms, departments, and institutions, though we have not always called it that. • Assessment can move beyond the classroom to become program assessment: • Classroom assessment – faculty evaluates her own students’ assignments in the capstone course and uses the information to improve her own teaching the next semester • Program assessment – faculty evaluated her own students’ assignments in the capstone course, outlining the strengths and weaknesses of the students’ work in relationship to departmental learning goals. The department uses the data to inform decisions about curriculum and other factors that affect student learning. (page 2-3) Source:Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004

48. Methods of Assessment • Course imbedded assessments • Written works • Student journals • Speeches • Skills-based assessment (demonstrated skills) • Observation checklists • Teamwork assessments • Surveys that ask about specific behaviors indicative of changes in values and attitudes • Not self-evaluation of actual learning

49. Great Fallacy #1 • Grades • In this day of social promotion, grade inflation and different teaching/learning philosophies, grades tell you virtually nothing. • They are not a measure of outcome achievement. • Two teachers will grade a student differently for the exact same work. • They cannot be used!

50. Great Fallacy #2 • The evaluation of teaching • Only tells you who is happy…”happiness coefficient” • Measures more about the business and science of teaching – not learning • Learning outcomes are not measures of teacher effectiveness or students satisfaction with the teaching/learning process • Learning outcomes are skills based • Students opinions are poor indicators if asked to self-assess • “In your opinion, are you more accepting of cultural differences in people since completing your coursework at the college?” • It is why we don’t ask faculty “On a scale of 1-5, how good of a teacher do you feel you are?”