Assessment & Review of Graduate Programs - Doctoral

1. Assessment & Review of Graduate Programs - Doctoral Duane Larick, NC State University Michael Carter, NC State University Margaret King, NC State University Council Of Graduate Schools Pre-Meeting Workshop December 7, 2005

2. Guidelines for This Presentation • Please feel free to raise questions at anytime during the presentation • We have included discussion questions along the way • We are very interested in your participation through questions and sharing of experiences from your campus • We will also leave time at the end for general discussion.

3. Agenda • Introduction and Objectives • Overview of Graduate Program Review • Reasons for Graduate Assessment • General Process of Program Review • Process or Processes for Development of a Program Review Procedure • External program review • Outcome based – continuous & ongoing review • Comparative Data Sources • Summary and Discussion

4. Objectives • Discuss various motivators for undertaking graduate assessment • Increase overall awareness of recent trends in Graduate Program Review • Demonstrate practical experience/knowledge gained related to development and implementation of external reviews and outcome-based continuous and ongoing procedures for Graduate Program Review • Illustrate examples of data and managerial tools developed/utilized to improve the efficiency of the process

5. Background Information About Our Audience • How many of you are responsible for graduate program review at your institutions? • How many of you have this as a new responsibility? • How many of you have recently (or are considering) changing your procedure?

6. Why Assess Graduate Programs?

7. Why Assess Graduate Programs? • The primary purpose should be to improve in the quality of graduate education on our campuses • By creating a structured, scheduled opportunity for a program to be examined, program review provides a strategy for improvement that is well-reasoned, far-seeking, and as apolitical as possible

8. Why Assess Graduate Programs? External Drivers: • To help satisfy calls for accountability • Especially at the State level

9. State Mandated Evaluation of New Programs • All new degree program proposals must include an evaluation plan that includes: • the criteria to be used to evaluate the quality and effectiveness of the program • measures to be used to evaluate the program • expected levels of productivity of the proposed program for the first four years of operation (number of graduates) • a plan and schedule to evaluate the proposed new degree program prior to the completion of its fifth year

10. State-Mandated 5th-Year Review - Issues • Statewide Productivity Assessment of Graduate Programs • Capacity in Relation to Student Demand • Capacity in Relation to Occupational Demand • Centrality in Relation to Instructional Mission • Success of Graduates • Program Costs

11. Low Productivity Analysis - Elements of Statewide Analysis for Each Program Area to be Reviewed • Trends in enrollment and degrees granted • Student characteristics • Program costs • Occupational demand • Recommendations for expansion or elimination of programs on a statewide basis

12. Why Assess Graduate Programs? External Drivers: • Requirement for regional accreditation, licensure, etc.

13. Regional Accreditation Agencies • Southern Association of Colleges and Schools • Western Association of Colleges and Schools • Northwest Association of Colleges and Schools • North Central Association • New England Association of Schools and Colleges • Middle States Commission on Higher Education

14. SACS Principles of Accreditation • Core requirement #5: “The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that incorporate a systematic review of programs and services that (a) results in continuing improvement and (b) demonstrates that the institution is effectively accomplishing its mission.”

15. SACS Criterion for Accreditation • Section 3 – Comprehensive Standards - #16 • “The institution identifies outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results.

16. Western Association of Schools & Colleges • Accreditation Standards 1.2. Educational objectives are clearly recognized throughout the institution and are consistent with stated purposes. The institution has developed indicators and evidence to ascertain the level of achievement of its purposes and educational objectives. 4.4. The institution employs a deliberate set of quality assurance processes at each level of institutional functioning, including new curriculum and program approval processes, periodic program review, ongoing evaluation, and data collection. These processes involveassessments of effectiveness, track results over time, and use the results of these assessments to revise and improve structures and processes, curricula, and pedagogy.

17. Intent of Accreditation Agency Effort • The intent of the regional accrediting agencies is to “encourage” institutions to create an environment of planned change for improving the educational process

18. Other Accreditation Agencies • Education, Architecture, Engineering, etc. • Often focused on minimum standards required • Department approach to development of the self-study and the review is focused on demonstration of achievement of those standards – not necessarily program improvement

19. Why Assess Graduate Programs? Internal Drivers: • Meet short-term (tactical) objectives or targets • Enrollment Growth & Funding • Meet long-term (strategic) institutional or departmental goals • Funding allocation/reallocation • Understand sources of retention/attrition among students and faculty • Funded project evaluation (GAANN, IGERT)

20. Discussion Questions • What other external and internal drivers exist on your campuses?

21. So The Questions We Need To Ask Ourselves Are • What are we currently doing? • Why are we currently doing it? • Is what we are currently doing accomplishing the external goals described above? • Is what we are currently doing accomplishing the internal goals described above? • Is there a better way? • Who defines better?

22. Procedure(s) for Review of Doctoral Graduate Programs • External program review conducted on a 5 – 10 year cycle • Standard practice at most Institutions • Outcome-based continuous and ongoing program review • Being implemented by many in response to regional and state accreditation requirements and institution needs

23. Key Characteristics of External Program Reviews • Program review is evaluative, not just descriptive • More than merely a compilation of data , it requires academic judgment of the data • Review of graduate programs is forward looking • It is directed toward improvement of the program, not simply assessment of its current status

24. Key Characteristics of External Program Reviews - continued • Programs should be reviewed on the basis of academic strengths and weaknesses, not on their ability to generate funding • Finances and funding should be relevant only as they affect the quality of the academic program • To the extent possible, program review should be an objective process

25. Key Characteristics of External Program Reviews - continued • Graduate program review should be an independent process, distinct from any other review • Efficiency can be gained by incorporating graduate program review with other internal or external reviews but, to be effective, graduate program review must lead to its own set of conclusions and direct its recommendations to the faculty and administrators who have the power to improve the graduate program

26. Key Characteristics of External Program Reviews - continued • Most importantly, program review MUST result in action • Based on the self-study, reviewers’ comments and recommendations, and faculty and administrator response to the review report, the institution develops and agrees on a plan to implement the desired changes • This plan must be linked to the institution’s planning and budget process

27. Successful Graduate Program Review Answers the Following Questions • Is the program advancing the state of the discipline or profession? • Is its teaching and training of students effective? • Does the program meet the institution’s goals? • Does it respond to the profession’s needs? • How is it assessed by experts in the field?

28. General Process for External Reviews Operational Procedures: • 5 - 10 year review cycle • Components • Internal self-study - report • External team review • Review team’s report • Program’s response • Administrative Meeting

29. Issues to be Resolved Before Beginning Program Reviews Locus of Control – Administration of Review Process • Comprehensive reviews are often coordinated by the office of the college or school dean or the chief academic officer • Graduate program reviews are often coordinated by the graduate dean

30. Issues to be Resolved Before Beginning Program Reviews - continued Regardless of who controls the review, the following principles should apply: • All reviews should involve the college or school administration • The graduate dean should play a major leadership role in all graduate reviews • The essential participants in any graduate program review are the chief academic officer, college administration, graduate dean, department chair, graduate program administrator, graduate program faculty, review team(s) and graduate students in the program

31. Issues to be Resolved Before Beginning Program Reviews - continued Counting – and Paying – the Costs • A realistic estimate of the costs must be made and an agreement must be reached regarding who will pay them • Costs include: • Travel, accommodations and meals for reviewers, honoraria for reviewers, etc. • Costs for developing and reproducing review documents, etc.

32. Issues to be Resolved Before Beginning Program Reviews - continued Graduate Versus Overall Program Review? • Advantages to graduate-only review • Allows for a thorough, in-depth review of the graduate program • Attention focused on quality indicators unique to graduate education • No risk of the graduate program review being “overwhelmed” by the size of the undergraduate program

33. Issues to be Resolved Before Beginning Program Reviews - continued Graduate Versus Overall Program Review? • Advantages to comprehensive review • Potential savings in time and money • Does not subject departments to multiple separate reviews • Graduate and undergraduate programs, as well as research and outreach activities are interdependent • Matters like faculty teaching loads, program and departmental budgets, facilities and quality of teaching and research experience may be more adequately addressed

34. Issues to be Resolved Before Beginning Program Reviews - continued Scheduling Reviews • More well-meaning plans for graduate program review have foundered on an unworkable timetable than any other obstacle! • Recommendation is a 5 – 7 year cycle • This depends on the number of programs and resources available • Programs may be grouped by department, college, etc. for review. The review “unit” should be established prior to scheduling

35. Issues to be Resolved Before Beginning Program Reviews - continued Scheduling Reviews – continued • Factors to consider in determining the order of programs for review: • Length of time since last review • Compelling financial problems or resource needs • Major proposals for curricular change • Upcoming accreditation or external reviews • Faculty or administration desire for review

36. Issues to be Resolved Before Beginning Program Reviews - continued Scheduling Reviews – continued • The schedule MUST be published far in advance • Programs generally need 9-12 months to prepare the self-study, etc. • Once established, every effort should be taken to maintain the schedule, BUT things happen!

37. Issues to be Resolved Before Beginning Program Reviews - continued Coordination With Accreditation Reviews • Graduate program reviews should be a separate process from accreditation reviews, but much can be gained by conducting them in tandem, sequentially, or at least in the same academic year: • Efficiency of data collection • Graduate program review team can benefit from the expertise and report of the accreditation team • When done in tandem, it is extremely important that the accreditation team acknowledge the difference(s) in the nature of the two reviews

38. Issues to be Resolved Before Beginning Program Reviews - continued Masters Versus Doctoral Programs • Whether it leads to a doctoral program or not, a master’s degree should have its own academic integrity • At those institutions with research-oriented master’s and doctoral programs in the same department, programs at both levels should be reviewed simultaneously • The institution should examine the unique characteristics of each master’s program and develop criteria of evaluation appropriate for that program

39. Issues to be Resolved Before Beginning Program Reviews - continued Research Based Versus Practitioner Graduate Program Reviews • Traditional research-based and practitioner programs often exist within the same department • Despite the differences in their educational goals, they should be reviewed together • It is essential that they be reviewed using different criteria • Should not rely on the use of professional accreditation review in place of internal review

40. Issues to be Resolved Before Beginning Program Reviews - continued Interdisciplinary Programs • Truly interdisciplinary programs cause special problems for review • Faculty and students are often arranged into academic departments • Those academic departments often control resources, faculty hiring, student admissions, course offerings, etc. • In spite of the administrative convenience of working through existing departments, interdisciplinary programs should be reviewed independently

41. Issues to be Resolved Before Beginning Program Reviews - continued Integration of Formal Review with Continuous Outcomes Assessment • It is important that formal review and continuous and ongoing assessment be seen as part of the same whole, with a common goal of improving graduate education • To accomplish this, they should somehow be coordinated and integrated • We will discuss how we do that at NC State later in the presentation

42. Discussion Questions • What other issues have you had to resolve on your campuses? • How have you resolved them?

43. Key Elements of a Successful Program Review Clear, Consistent Guidelines • These guidelines should describe: • The purpose of graduate program review • The process to be followed • Guidelines for materials to be included in each phase • A generic agenda for the review • The use to which results will be put • These guidelines should be posted on the Graduate School or Academic Affairs web page

44. Key Elements of a Successful Program Review Administrative Support • Adequate staffing and general administrative support are vital to the success of any program review • Departments can provide their own support for the self-study • The larger review process should be staffed centrally

45. Key Elements of a Successful Program Review Administrative Support - continued • Successful reviews depend on accurate institutional data • This data should be developed and maintained centrally but should be reviewed and evaluated by the program faculty • A standard report format using a single set of definitions should be developed in advance • The best information often comes from a combination of central and departmental sources

46. Managerial Tools Created for Program Review - Website

47. Managerial Tools Created for Program Review - Website

48. Managerial Tools Created for Program Review - Profile Data

49. Managerial Tools Created for Program Review - Profile Data

50. Managerial Tools Created for Program Review - Profile Data