Using Institutional Effectiveness to Build a Culture of Performance Improvement

1. Using Institutional Effectiveness to Build a Culture of Performance Improvement Revised version for the 2011 SACS Working Group Department of Institutional Research and Effectiveness St. Petersburg College P.O. Box 13489 St. Petersburg, FL 33733 (727) 341-3059 FAX (727) 341-5411 2007 SACS-COC Annual Meeting

2. Issues with IE • SACS’s Peer Review Research Project revealed that in the area of Institutional Effectiveness (IE) • 62% of institutions in off-site reviews were deemed non-compliant and • 27% of institutions in on-site visits were deemed non-compliant • IE was the second most identified area behind faculty qualifications 2007 SACS-COC Annual Meeting

3. SPC Experience with IE • As part of the recent SACS reaccreditation process, SPC spent a considerable amount of time and effort documenting and detailing their IE processes • The result…SPC had “zero” compliance issues in the area of IE during the off-site reviews 2007 SACS-COC Annual Meeting

4. SACS Requirements for IE • Core Requirement 2.5: Integrated Planning & Evaluation Processes Institutional Effectiveness • “…ongoing, integrated, and institution-wide research-based planning and evaluation processes…” • Comprehensive Standard 3.3.1: Institutional Effectiveness • “…identifies expected outcomes…,assesses whetherthe extent to which it achieves these outcomes, and provides evidence of improvement…” • Comprehensive Standard 3.5.1: General Education Core College Level Competency • “… identifies college-level general education competencies and the extent to which graduates have attained them. within the general education core and provides evidence that graduates have attained those competencies.” 2007 SACS-COC Annual Meeting

5. SACS Requirements for IE • Additional IE related requirements include: • Comprehensive Standard 3.1.1: Institutional Mission • Federal Requirement 4.1: Evaluating Success Student Achievement • Federal Requirement 4.2: Program Curriculum 2007 SACS-COC Annual Meeting

6. IE Evidence • SACS Suggested Documentation for CR 2.5 • Evidence of linkage of IE to institutional mission • Institutional plans and budgets that demonstrate linkage of assessment findings to planning at all levels • Minutes of appropriate (IE related) unit, committee, task force meetings… • Documentation that relates to IE, such as budget preparation instructions, minutes of budget presentation meetings, annual reports, annual assessment updates, IE reports • Samples of specific actions taken to improve the IE process and/or results from that process 2007 SACS-COC Annual Meeting

7. Performance Improvement • From Compliance to Performance Improvement 2007 SACS-COC Annual Meeting

8. SPC’s Definition of IE • “At SPC, Institutional Effectiveness is the integrated, systematic, explicit, and documented processes of measuring performance against the SPC mission for purposes of continuous improvement of academic programs, administrative services, and educational services offered by the college.” “Closing the Loop” 2007 SACS-COC Annual Meeting

9. IE Phases • IE Processes are yearly processes that consist of three distinct phases: • Evaluation/assessment phase • Strategic/budget planning phase • Implementation/operation phase 2007 SACS-COC Annual Meeting

10. Evaluation/Assessment Phase • Reflective look back at the prior year’s assessment and evaluation results. • educational outcomes • institutional surveys • departmental-specific assessments • Individual assessments results are reviewed by the respective oversight group which in turn aggregates the information and presents recommendations to the President’s Cabinet. • These recommendations are evaluated and are routinely summarized in the form of the Institutional Objectives (IOs). 2007 SACS-COC Annual Meeting

11. Strategic/Budget Planning Phase • Transition from assessment/evaluation results to planning for the upcoming operational year • Includes assessment-driven unit planning • Unit managers state their intent for the upcoming year in measurable objectives • Unit managers ensure alignment with the institution’s mission and goals, Strategic Directions and Institutional Objectives (SD-IOs) 2007 SACS-COC Annual Meeting

12. Implementation/Operation Phase • Execution of the developed unit plans • Unit managers are required to provide periodic reports of their progress to ensure the fidelity and stability of these unit plans • Periodic reports include a “mid-year status report” and an “end-of-year final results report.” The end-of-year report includes assessment results in addition to the unit’s use of these results in improving their unit. 2007 SACS-COC Annual Meeting

13. Individual IE Processes • Individual IE processes are designed to result in continuous improvement and comprise the following: • Mission statement process • Evaluation and assessment processes for academic programs, student services, and administrative services • Strategic Directions and Institutional Objectives (SD-IOs) process • Unit / budget planning process • Collegewide operating budget development process • Budget approval, implementation, and annual operating results 2007 SACS-COC Annual Meeting

14. a. Mission Statement Process Mission Statement • IE process begins and ends with the College Mission statement • College’s Mission is formally reviewed every five years by a committee representing faculty, administrators, students, community leaders, and members of the Board of Trustees (BOT) • Mission is also reviewed annually by President’s Cabinet before the new strategic planning cycle begins 2007 SACS-COC Annual Meeting

15. a. Mission Statement Process Assessing the Mission SPC Mission – Assessments - Closing the Loop 2007 SACS-COC Annual Meeting 5

16. b. Evaluation and Assessment Processes Evaluation and Assessment • Evaluation and assessment processes at SPC are centered on mission-driven outcomes in the following program and service areas: • (i) 26 Organizational units comprising 40 Academic programs (lower division); • (ii) 8 Colleges and Schools comprising 20 Majors (upper division), and • (iii) key administrative and educational support services. • Analysis of outcome results is on-going and captured in various assessment reports. • SPC evaluates all student services to ensure students are provided the best support possible to ensure student success. 2007 SACS-COC Annual Meeting

17. b. Evaluation and Assessment Processes Assessment Areas • Academic Outcomes • General Education • BAS/BS • AAS/AS • Key Administrative and Educational Support Services • Viability • Importance • Satisfaction 2007 SACS-COC Annual Meeting

18. b. Evaluation and Assessment Processes Assessments Ed Gen Reviews Surveys Assessment Focus • Continually evaluate assessment instruments and current processes • Integrate assessment with educational process • Focus on fewer, yet meaningful improvements (action items) • Use multiple assessment methods • Involve faculty in process • Integrate timing of assessments (3-year cycle) 2007 SACS-COC Annual Meeting

19. b. Evaluation and Assessment Processes Direct and Indirect Measures • Direct Measures • Internally Developed General Education • Externally Developed General Education (e.g., MAPP, iSkills) • AAS/AS Academic Program Assessment Report (APAR) • BAS/BS Program Assessment • Comprehensive Academic Program Review (CAPR) • Indirect measures • Entering Student Survey • Enrolled Student Survey • Graduating Student Survey • Recent Alumni Survey • Employer Survey • Community College Survey of Student Engagement (CCSSE) 2007 SACS-COC Annual Meeting

20. b. Evaluation and Assessment Processes Three-year Assessment Cycle APARs Academic Program Viability Reviews (APVRs) conducted yearly CAPRs 2007 SACS-COC Annual Meeting

21. b. Evaluation and Assessment Processes Key Assessments Notes: • SACS Compliance Requirement • Florida DOE Recommendations • BOT Rule 6Hx23-3.04 Requirement

22. b. Evaluation and Assessment Processes Academic Program Assessment • Academic Program Assessment Reports (APARs) • Prepared for the Gen Ed, A.S., and A.A.S. programs according to the three-year cycle • Include a summary of assessment findings and a discussion and analysis of those assessment findings (including use of results, action plan items, and budgetary implications) • Process was initiated college-wide in 1999 • Program data is gathered and analyzed on a continuous basis 2007 SACS-COC Annual Meeting

23. b. Evaluation and Assessment Processes Academic Program Viability Review • Academic Program Viability Review (APVR) • Published yearly • Included measures: • Program Graduates • Course Enrollment • Student Semester Hour (SSH) productivity • Relative Profitability Index (RPI-T) • Economic Trend Data 2007 SACS-COC Annual Meeting

24. b. Evaluation and Assessment Processes Program Review Process • Comprehensive Academic Program Review (CAPR) is a summative evaluation that includes multiple measures such as: • program-specific performance • profitability measures • academic outcomes, and • stakeholder perceptions 2007 SACS-COC Annual Meeting

25. b. Evaluation and Assessment Processes Program Review Objectives • CAPR was designed to meet three objectives: • To provide a comprehensive report that summaries all elements of the program’s viability and productivity from a 360-degree perspective, • To provide comprehensive and relevant program specific information to key College stakeholders such as the President Cabinet members in order to make critical decisions regarding the continued sustainability of a program, and • To provide program leadership a vehicle to support and document actionable change for the purposes of performance improvement. 2007 SACS-COC Annual Meeting

26. b. Evaluation and Assessment Processes Ed Outcomes Online • To provide a medium for completing the draft educational assessment reports as well as establishing a repository for program specific information, SPC developed an Educational Assessment Web site (https://it.spcollege.edu/edoutcomes/) • College administration and instructional staff are provided access to “completed” assessment reports including the APAR (Program & Gen Ed) and the CAPR • Online access further encourages the use of assessment data as well as highlighting “best practices” across the college 2007 SACS-COC Annual Meeting

27. b. Evaluation and Assessment Processes CollegeAccess 2007 SACS-COC Annual Meeting

28. b. Evaluation and Assessment Processes Program Director Access 2007 SACS-COC Annual Meeting

29. b. Evaluation and Assessment Processes Student Survey Process • Data collection • Increased student awareness of survey process and accessibility • Minimized use of class time to complete surveys • Use multiple points of student contact • Online survey submission and scanning of paper forms • Quality of survey forms • Professional “look and feel” of survey format • Items designed to gather pertinent data • Included online branching models to reduce survey length 2007 SACS-COC Annual Meeting

30. c. Strategic Directions and Institutional Objectives Process Strategic Directions • College’s SD-IOs are comprised of longer-term directional statements and shorter-term, annually measurable objectives. • Strategic Directions are: • Updated on a three-year cycle and are based on achieving the College’s mission. • Developed to make improvements in learning outcomes, quality, productivity, and other environmental-scan considerations, as well as inputs from IE Team, President’s Cabinet, faculty and student representatives, the BOT, and external/community stakeholders. 2007 SACS-COC Annual Meeting

31. c. Strategic Directions and Institutional Objectives Process Institutional Objectives • Institutional Objectives are • Reviewed and updated annually, with key on-going initiatives updated based on progress made during the prior year and carried forward, where appropriate. • Designed for the purpose of achieving the corresponding longer-term strategic direction. 2007 SACS-COC Annual Meeting

32. c. Strategic Directions and Institutional Objectives Process SD-IO Alignment Mission – Assessments - Closing the Loop Examples 2007 SACS-COC Annual Meeting

33. d. Unit / Budget Planning Process Unit / Budget Planning • SPC’s unit/budget planning is centered on: • more than 150 Unit / Budget Managers, • development of departmental unit plans and budget requests, and • use of the national award-winning Strategic Planning Online (SPOL) system. • Unit / Budget Managers are urged and encouraged to access, review, and make use of all available assessment outcome data (i.e., College’s IE website, oversight group reports, etc.) 2007 SACS-COC Annual Meeting

34. d. Unit / Budget Planning Process Unit / Budget Planning • Budget guidelines are prepared by the Collegewide Budget Committee (CBC), which provides • Information about the College’s expectations for State appropriations, • Collegewide budgetary outlook, and • Salary package and other key strategic expectations. • Budget guidelines also provide details on how capital expenditure requests are to be entered into the SPOL system • SPOL system requires the unit / budget managers to select one of the SD-IOs for each of the unit plan objectives entered 2007 SACS-COC Annual Meeting

35. d. Unit / Budget Planning Process Unit / Budget Planning • Supporting tasks, expected outcomes, and related budget funding are also captured in the SPOL system • After all unit plans and budget requests are entered, they undergo management review in which refinements and adjustments may be made based on funding availability and overall College priorities • Approved Unit plans (by College President’s direct reports) are aggregated and submitted for further review and evaluation by the CBC 2007 SACS-COC Annual Meeting

36. e. Collegewide Operating Budget Development Process Collegewide Budget Committee • Each year, the President confirms the membership of a CBC (SVPs, VPs, provosts, and other key functional leaders) • Certain members of the committee are rotated on an annual basis for cross-training purposes. • The charge of the CBC includes the following: • Develop budget guidelines, • Review of all budget requests approved by department heads, vice presidents, and senior vice presidents, • Ensure that the SD-IOs are funded according to their priorities and funding availability, and • Develop the College’s Operating Budget. 2007 SACS-COC Annual Meeting

37. f. Budget Approval, Implementation, and Annual Operating Results Budget Approval • Operating Budget is approved by the President’s Cabinet and recommended to the BOT • Once approved by BOT, the budget is communicated and uploaded into the College’s central financial system • College financials and operating results are monitored on a monthly basis and reported quarterly to the BOT 2007 SACS-COC Annual Meeting

38. f. Budget Approval, Implementation, and Annual Operating Results Operating Performance and Results • Operating performance and results are centered on: • Executing SD-IOs • Supporting unit plans and quality improvement initiatives • Adhering to the operating budget • Operating performance and results are accomplished through the following key steps: • Implementing budgets and action plans • Monitoring achievements and related circumstances • Evaluating mid-year unit plan reviews • Review end-of-year unit plan reports 2007 SACS-COC Annual Meeting

39. IE Next Steps • Ultimate goal is to provide stakeholders ‘timely’, ‘relevant’, ‘accurate’, and ‘interpretable’ data through: • Formatted (dashboard) style reports, and • On-demand customizable reporting, with • Valid, reliable, and standardized measures. 2007 SACS-COC Annual Meeting

40. Links to Additional IE Materials http://www.spcollege.edu/Central/IE/2007_SACS_Presentation.htm 2007 SACS-COC Annual Meeting

41. Key Points • IE is more than assessment • A lot has changed since 2007 • IE is less centralized • New planning system has been implemented • Assessment activities have expanded 2007 SACS-COC Annual Meeting

42. Next Steps • Need to capture/define the current processes • Need to ensure alignment among current processes • Need to incorporate the new Student Achievement Committee Structure • Finally, need to tell the story… 2007 SACS-COC Annual Meeting

43. Questions 2007 SACS-COC Annual Meeting

44. Using Institutional Effectiveness to Build a Culture of Performance Improvement Revised version for the 2011 SACS Working Group Department of Institutional Research and Effectiveness St. Petersburg College P.O. Box 13489 St. Petersburg, FL 33733 (727) 341-3059 FAX (727) 341-5411 2007 SACS-COC Annual Meeting