Utilizing an Outcomes-Based Program Review to Inform Resource Allocations

1. Utilizing an Outcomes-Based Program Review to Inform Resource Allocations Lauren Weiner, Ed.D. Director, Associated Students Administration University of California-San Diego Marilee J. Bresciani, Ph.D. Professor, Postsecondary Educational Leadership San Diego State University Ben Gillig, M.A. Candidate Assistant Coordinator of Academic Initiatives San Diego State University Lisa McCully, M.A. Director, College of Education Office of Student Services San Diego State University Bresciani, Gillig, Weiner, & McCully

2. Session Overview • Presentation of framing questions • Review of research study • Discussion of findings and application to practice • Presentation of a preliminary framework • Discussion with small groups Bresciani, Gillig, Weiner, & McCully

3. Framing Questions • What evidence is used to inform your resource re-allocation or allocation processes? • How does your outcomes-based program review process inform resource re-allocation or allocation process? • Why would you even want to have this conversation? Bresciani, Gillig, Weiner, & McCully

4. Background • Resource allocation is about more than just money… • Human resources • Facilities • IT • Time spent by senior leaders • Budget allocations • Evidence that priorities are being effectively met • Resource allocation and re-allocation are cornerstones of senior student affairs officers’ jobs • How resources are allocated says something about how the division operates and how it sets its priorities Bresciani, Gillig, Weiner, & McCully

5. Student Affairs Leaders Pulling The Pieces Together Resource Allocation Bresciani, Gillig, Weiner, & McCully

6. Literature Review Budgeting Frameworks • Program budgeting • Formula budgeting • Zero-base budgeting • Performance budgeting • Incremental budgeting Require a set of purposes, priorities, and/or strategies to work. What framework do we use to ensure that our purposes, priorities, and/or strategies will be effective in accomplishing intended results? Bresciani, Gillig, Weiner, & McCully

7. Purpose of the Study • This mixed method study explores how institutional/divisional leadership use outcomes-based assessment results to inform resource allocations and re-allocations for student learning and development. • Explores the need to utilize outcomes-based assessment in times of budget cuts, reduction in revenues, and a higher demand for services • The findings intend to identify frameworks that align resource allocations and re-allocations with stated priorities and provide evidence to refine programs and services that align with those priorities. Bresciani, Gillig, Weiner, & McCully

8. Hypotheses • The manner in which institutions are funded does not influence how institutional leadership uses outcomes-based assessment results to inform resource allocations and re-allocations for student learning and development. • The timeframe in which institutional leadership allocates and re-allocates resources for student learning and development does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development. Bresciani, Gillig, Weiner, & McCully

9. Hypotheses, Cont. • The budgeting framework that institutional leadership uses to allocate and re-allocate resources for student learning and development does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development. • The manner in which institutional leadership engages in strategic planning does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development. Bresciani, Gillig, Weiner, & McCully

10. Hypotheses, Cont. • The manner in which institutional leadership engages in outcomes-based assessment program review does not influence how they use outcomes-based assessment results to inform the resource allocation and re-allocation of resources for student learning and development. Bresciani, Gillig, Weiner, & McCully

11. Methodology • This mixed methods study included the following data collection processes: • Survey list started with institutional members (n=1,030) • Researchers stratified the institution by type • A random sample of 257 invitations were then sent through Student Voice. • Survey invitations were sent to the institution’s Senior Student Affairs Officer • E.g., VP of Student Affairs or Student Services Bresciani, Gillig, Weiner, & McCully

12. Methodology • After data was collected, researchers analyzed the information using descriptive statistics • Researchers looked for correlations between institutional/divisional budget process and how they utilized their outcomes-based assessment results for their resource allocations and re-allocations • Researchers conducted a document analysis based on the information that was provided from their institutions URL addresses • Interviews will be conducted in the upcoming weeks Bresciani, Gillig, Weiner, & McCully

13. Survey Participant Demographics • Institution type Private 4-year Liberal Arts Colleges (35%) Public 4-year Comprehensive (20%) Public 2-year (15%) • Institution size Less than 4,000 (37%) 4,000 to 8,000 students (24%) 15,001 to 25,000 students (22%) • Position title Senior Student Affairs Officer (93%) Bresciani, Gillig, Weiner, & McCully

14. Instrument Development • Survey included 25 questions and a request for institutions to provide their URL address for additional assessment documents • Survey focused on asking institutional leaders to discuss if they utilize (a) outcomes-based assessment, (b) program review, (c) annual reports, and (d) strategic planning to inform the budget and resource allocation and re-allocation process at the institutional and divisional level • Survey included questions pertaining to institutional/divisional leadership practices and questions relating to the practices used for allocating and re-allocating resources to student learning and development programs Bresciani, Gillig, Weiner, & McCully

15. Document Analysis • Survey respondents were asked to provide their URL addresses • Document analysis selection criteria: • All materials needed to be official institutional documents, which included: • Annual reporting processes • Outcomes-based assessment processes • Program review processes • Assessment plans from the institutional, divisional, and departmental levels • Researchers analyzed documents and developed themes by using the comparative method for category development • Researchers determined essential descriptive groupings during the initial coding phase Bresciani, Gillig, Weiner, & McCully

16. Instrument Development Interview Protocol: • Researchers will conduct semi-structured interviews with institutional leaders, such as vice-presidents, deans, faculty, and administrators directly involved in the outcomes-based assessment program review process • Researchers will also conduct interviews with institutional leaders involved in the resource allocation and re-allocation processes Questions focus on: • Types of evidence based decision-making processes that institutional leadership use to inform the resource allocation and re-allocation process for student learning and development • The alignment between the results generated from outcomes-based program review and resource re-allocations and allocations Bresciani, Gillig, Weiner, & McCully

17. Limitations • Since the researchers used a self-reporting survey, participants did not have to provide their actual budgeting documents, thus researchers cannot triangulate that data; however, outcomes-based and planning practices can be triangulated through document analysis • 25% response rate; survey is back out in the field Bresciani, Gillig, Weiner, & McCully

18. Preliminary Findings Bresciani, Gillig, Weiner, & McCully

19. The manner in which institutions are funded does not influence how institutional leadership uses outcomes-based assessment results to inform resource allocations and re-allocations for student learning and development. WE DO NOT HAVE CONCLUSIVE DATA FOR REJECTING THIS HYPOTHESIS Bresciani, Gillig, Weiner, & McCully

20. There were significant correlations between certain revenue sources and use of outcomes based assessment at certain points in the resource allocation/re-allocation process. • Use of tuition and fee revenue was associated with use of OBA to formulate budget requests at both the institutional and division levels but not to inform actual resource allocation (t=-.43, p>.01; t=-.48, p>.01). • Use of state allocated revenue was associated with OBA playing a role in re-allocation only (t=-.34, p=.02). • Use of state allocation revenue was also associated with OBA playing a weighted role in the formulas used to allocate money to departments (t=.83, p=.01). • Use of endowment revenue was associated with OBA playing a weighted role in the formulas used to allocate money to divisions (t=.31, p=.04). • Use of revenue from auxiliary services was associated with OBA playing no role in resource allocation/re-allocation at all (t=-.32, p=.02). Based on these preliminary findings, we will be conducting interviews and gathering additional documents to clarify Bresciani, Gillig, Weiner, & McCully

21. The timeframe in which institutional leadership allocates and re-allocates resources for student learning and development does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development. • 98% of survey participants responded that their resource allocation process takes place on an annual basis • Since the majority of respondents selected the same timeframe, we can not determine the extent that the time frame influences this process. • Thus, we can not accept or reject this hypothesis Bresciani, Gillig, Weiner, & McCully

22. The budgeting framework that institutional leadership uses to allocate and re-allocate resources for student learning and development does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development. • 46% report their institutions using incremental budgeting processes, while 35% use program-based at the divisional level • There are no significant correlations between budgeting framework and the use of OBA for resource allocation and re-allocation • We accept this hypothesis Bresciani, Gillig, Weiner, & McCully

23. The manner in which institutional leadership engages in strategic planning does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development. • Strategic planning is the most common practice used to inform the resource allocation or re-allocation process • Institution level: 22% of survey respondents reported that strategic planning informs the resource re-allocation process • Institution level: 20% of survey respondents reported that strategic planning played a role in the way they construct the budget, but not the allocation of resources • Division level: 22% of survey respondents reported that strategic planning informs the resource re-allocation process • Division level: 22% of survey respondents also reported that strategic planning played a weighted role in the formulas used to allocate dollars • There are no significant correlations between the use of strategic planning and the use of OBA for resource allocation and re-allocation • We accept this hypothesis. Bresciani, Gillig, Weiner, & McCully

24. The manner in which institutional leadership engages in outcomes-based program review does not influence how they use outcomes-based assessment results to inform the resource allocation and re-allocation of resources for student learning and development. • Institution level: 24% of survey respondents reported that program review does not have any role in the allocation or re-allocation of resources • Institution level: 20% of survey respondents reported that program review plays a role in the re-allocation of resources • Division level: 27% of survey respondents reported that program review plays a role in the re-allocation of resources • Division level: 22% of survey respondents reported that program review does not play any role in the allocation or re-allocation of resources Bresciani, Gillig, Weiner, & McCully

25. The manner in which institutional leadership engages in outcomes-based program review does not influence how they use outcomes-based assessment results to inform the resource allocation and re-allocation of resources for student learning and development. • However, uses of outcomes-based program review data was correlated with several uses of outcomes-based assessment data: • Use of program review data in a weighted role in budget formulas used to allocate money to departments was correlated with the use of outcomes based assessment data for the same purpose (t=.43, p<.01). • Use of program review data in a weighted role in the budget formula used to allocate money to programs and processes correlated with the use of outcomes based assessment data for the same purpose (t=.47, p<.01). • Use of program review data to construct budget proposals, but not necessarily to allocate resources was correlated with the use of outcomes based assessment data for the same purpose (t=.67, p<.001). • Use of program review data to construct budget proposals, but definitely not to allocate resources was correlated with the use of outcomes based assessment data for the same purpose (t=.69, p<.001). • Thus, we cannot reject this hypothesis. Bresciani, Gillig, Weiner, & McCully

26. Preliminary Findings from the Document Analysis • Documents analyzed from the URL addresses included annual reports, assessment plans, program reviews, strategic planning processes, assessment committee structures, and institutional planning documents • Themes from the documents: • Departmental goals and objectives align with the student affairs and institutional goals and objectives • Institutions use assessment results to inform the program improvement process, not in the resource allocation process • Identification of departmental and divisional successes, but do not inform the resource allocation process Bresciani, Gillig, Weiner, & McCully

27. Similarities and Differences Between Survey and Documents • Institutional leaders identified and communicated values, objectives, and priorities • Departments aligned programs with the institutional goals and objectives • Input being solicited from faculty, staff, and administrators • Strong participation of faculty, staff, and administrators in the planning process • Differences between survey and documents: • Survey respondents indicated that resource re-allocations were influenced by strategic planning and outcomes-based assessment program review findings; documents showed no linkage to the resource re-allocation process Bresciani, Gillig, Weiner, & McCully

28. Summary of Conclusions • There is a disconnect between what revenue sources influence versus what the budgeting process influences. • There is a disconnect between the institutional budgeting processes and the ability to use allocation of resources to improve student learning and development. Does that make sense to continue? • If 74% of the initial allocation of funding is done annually, why aren’t outcomes-based program review results used to inform initial allocations of resources? Bresciani, Gillig, Weiner, & McCully

29. Summary of Conclusions, Cont. • If strategic planning and outcomes-based assessment program review findings only influence resource re-allocations and not initial allocations, how will leadership be able to influence long-term improvement in student learning and development? • More to come…. Bresciani, Gillig, Weiner, & McCully

30. Framework with Modifications(Bresciani, 2010 & Bresciani, et.al, in progress) • Identify and articulate values • Prioritize values • Allocate resources • Align outcomes to values and Implement outcomes-based assessment • Define the criteria for quality within the context of the values and identify capacity for meeting the criteria of quality • Gather the results and determine at which level the decision for resource re-allocation or allocation resides Bresciani, Gillig, Weiner, & McCully

31. Framework with Modifications(Bresciani, 2010 & Bresciani, et.al, in progress) • Allocate or re-allocate resources to improve your outcomes within your context and capacity for quality and in alignment with your values Bresciani, Gillig, Weiner, & McCully

32. Small Group Discussion Questions • How well do your resource budgeting and allocation processes align? • How much do you utilize planning and assessment processes to allocate or re-allocate resources to refine your priorities? • Does this proposed framework offer any practical value for your division/institution? • What are some immediate next steps that you can implement? Bresciani, Gillig, Weiner, & McCully

33. Planned Next Steps for the Study • Complete correlation analysis • Re-invite participants to complete the survey • Request more URLs for expanded analysis - particularly budget process documents • Conduct interviews to test the viability of the framework • Write-up results for publication submission Bresciani, Gillig, Weiner, & McCully

34. Special Thanks to… • Student Voice • JD White • Annemieke Rice • Participating institutions • To you Bresciani, Gillig, Weiner, & McCully

35. Session Citation • This presentation: • Weiner, L., Bresciani, M.J., Gillig, B., McCully, L. (2011). Using outcomes-based assessment to inform resource allocation in higher education. Workshop presented at the annual meeting of the American College Personnel Association, Baltimore, MD. • Future paper: • Bresciani, M.J., Weiner, L., Gillig, B., McCully, L.(in preparation). Using outcomes-based assessment to inform resource allocation in higher education. Journal of Student Affairs Research and Practice submission Bresciani, Gillig, Weiner, & McCully

36. References Bresciani, M.J. (2010). Aligning values with resources and assessment results. Student affairs leader (38), 13, p. 1-2. Bresciani, M. J., Gardner, M. M., & Hickmott, J. (2009). Demonstrating student success in student affairs. Sterling, VA: Stylus Publishing. Berg, D.J. & Skogley, G.M. (1985). Editors’ notes. In D.J. Berg & G.M. Skogley (Eds.), Making the budget process work.  New Directions for Higher Education No. 52. San Francisco: Jossey-Bass. Carnegie Council on Policy Studies in Higher Education (1980). Three thousand futures: The next twenty years for higher education.  San Francisco: Jossey-Bass. Clark, R. & d’Ambrosio’s, M. (2006). The new balancing act in the business of higher education. Northamton, MA: Edward Elgar Books. Bresciani, Gillig, Weiner, & McCully

37. References, Cont. Merriam, S. B. (1998). Qualitative research and case study application in education. San Francisco: Jossey-Bass. Mayhew, L. (1979). Surviving the eighties: Strategies and priorities for solving fiscal and enrollment problems. San Francisco: Jossey-Bass. Strauss, A., & Corbin, J. (1990). Basics of qualitative research. Newbury Park, CA: Sage. Taylor, S. J., & Bogdan, R. (1998). Introduction to qualitative research methods. New York: John Wiley. Bresciani, Gillig, Weiner, & McCully

38. Contact Information • Marilee J. Bresciani – marilee.bresciani@mail.sdsu.edu • Ben Gillig –gillig.ben@gmail.com • Lauren Weiner – lsweiner@ucsd.edu • Lisa McCully – lmccully@mail.sdsu.edu Bresciani, Gillig, Weiner, & McCully