Using CAS Standards and Review Process for Improving Student Learning and Program Outcomes

1. Using CAS Standards and Review Process for Improving Student Learning and Program Outcomes Patricia Carretta George Mason University CAS Officer-at-Large NACE Representative to CAS NASPA 2008 International Assessment & Retention Conference

2. Purpose • CAS Standards and Review Process • Integrating CAS into assessment of programs and services and student learning outcomes

3. Role and Use of CAS Standards: • Student Affairs / Student Services at the division level? • Student Affairs / Student Services at the departmental or program level? • Other? • Have looked at the CAS Standards for at least one functional area? • Have participated in a department/program review process using CAS standards? • Have used CAS standards for other purposes?

4. Participant Outcomes • Define the role of CAS self-assessment / review process within a comprehensive assessment plan • List the seven step process of conducting a CAS review • Appreciate the value of engaging a variety of institutional stakeholders in the review process • Strategize how to conduct a comprehensive assessment of programs, services and student learning using CAS Standards • Formulate an assessment plan • Understand the importance of assessing and reporting on input data as well as student learning and development outcomes as a strategies for improving learning and program quality

5. About CAS: The mission of the Council for the Advancement of Standards in Higher Education (CAS) is to promote the improvement of programs and services to enhance the quality of student learning and development. CAS is a consortium of professional associations who work collaboratively to develop and promulgate standards and guidelines and to encourage self-assessment.

6. CAS Resources • CAS Professional Standards, 2006 edition • Self-Assessment Guides • CAS Statement of Shared Ethical Principles • CAS Characteristics of Individual Excellence • Frameworks for Assessing Learning and Development Outcomes

7. CAS Guiding Principles • Students and Their Institutions • Diversity and Multiculturalism • Organization, Leadership, and Human Resources • Health Engendering Environments • Ethical Considerations

8. Mission Program Leadership Human Resources Ethics Legal Responsibilities Equity and Access Diversity Organization and Management Campus and External Relations Financial Resources Technology Facilities and Equipment Assessment and Evaluation Organization of CAS Standards

9. Student Learning & Development • Must promote student learning and development outcomes • Must identify relevant and desirable student learning & development outcomes • Must assess outcomes • Must provide evidence of their impact on student learning and development • Must articulate how they contribute to or support student learning and development

10. CAS: Student Learning & Development Outcomes • Knowledge acquisition, construction, integration and application • Cognitive complexity • Intrapersonal development • Interpersonal competence • Humanitarianism & civic engagement • Practical competence

11. Student Learning & Development Outcomes • Consistent with the institutional mission, programs and services must identify relevant and desirable student learning and development outcomes from among the six domains. When creating opportunities for student learning and development, programs and services must explore possibilities for collaboration with faculty members and other colleagues. • Programs and services must assess relevant and desirable student learning and development outcomes and provide evidence of their impact on student learning and development. Programs and services must articulate how they contribute to or support student learning and development in the domains not specifically assessed.

12. Essential Elementsin the CAS Approach • Culture that values involvement of all its members • Quality indicators • Use of standards and guidelines • Willingness/capacity: • To examine itself and assemble results • To report and use the results

13. Uses of CAS Standards • Credibility, accountability, improvement: • Program & service improvement; measures of quality and effectiveness; measures of impact on learning • Design of new programs & services • Restructuring / reorganization • Change in leadership • Institutional self-studies • Preparation for accreditation • Staff development • Academic preparation

14. CAS: Standards and Guidelines STANDARD (BOLD TYPE; AUXILARY VERB MUST) To support the institution’s mission and goals, Career services must provide faculty and staff and administrative units with information, guidance, and support on career development and employment issues and linkages with the broader community. GUIDELINE (LIGHT-FACED TYPE; VERB SHOULD) Career Services should support faculty and staff and administrative units by: • Identifying and disseminating information on employment trends, top employing organizations, and co-op and internship sponsors.

15. Comprehensive Assessment Plan Includes: • Assessment of programs and services • Assessment of student outcomes, including student learning and development

16. Program Evaluation • Is the program or service functioning effectively to achieve its mission? • What evidence is available to support the determination? • How is evidence used to make program decisions?

17. Student Outcomes Assessment • What is the effect of our work on students? • How are they different as a result of interacting with our programs and services? • How do we demonstrate their learning? • How do we know? Measure?

18. The CAS SAG: A Self-Assessment Guide • Translates CAS standards into an effective workbook / evaluative format • Promotes program self-assessment and development • Informs on program strengths and weaknesses • Supports professional staff development • Leads to enhanced programs and services that benefit student learning and development

19. CAS Self-Assessment Process • Establish and prepare the Self-Assessment Team • Initiate the Self Study • Compile and review Documentary and Evaluative Evidence • Identify Discrepancies • Determine appropriate Corrective Action • Recommend Steps for Program Enhancement • Prepare an Action Plan

20. The Self-Assessment Team Appoint and Train the Team: • Size of team • Composition: internal, external • Coordinator/Leader • Establish team ground rules • Discuss meaning of each standard • Establish team’s inter-rater reliability • Encourage team discussion; expect disagreements; commit to consensual resolution

21. Team Actions • Decide whether to include guidelines or other measures that go beyond the standards • Gather and analyze quantitative and qualitative data • Individuals rate each and every criterion measure • Obtain additional documentary evidence if required to make an informed team decision • Complete the assessment, ratings & action plan worksheets

22. Evaluative Evidence STUDENTRECRUITMENT MATERIALS: • Brochures & other program information • Participation policies & procedures PROGRAM DOCUMENTS: • Mission statements; program purpose & philosophy statements • Catalogs & related materials • Staff & student manuals; policies & procedures statements ADMINISTRATIVE DOCUMENTS: • Organization charts; student & staff profiles • Financial resource statements & budgets • Annual reports STAFF ACTIVITY REPORTS: • Curriculum vitae & resumes; professional activity • Service to other programs, departments, or community STUDENT ACTIVITY REPORTS: • Portfolios, developmental transcripts, resumes • Reports of student service RESEARCH & EVALUATION DATA: • Needs assessments & self-studies • Program evaluation; graduate & follow-up studies

23. Quantitative Data • Summary of objective responses on a questionnaire or program evaluation • Statistics about use by students and various population subgroups • Needs assessments, follow-up studies, and self-study reports • Institutional research reports and fact books • Comparative or benchmark data at local, regional or national level

24. Qualitative Data • Focus group information • Written summaries of responses to open-ended questions in interviews and on evaluations • Client satisfaction surveys, self-reports, and written comments, both solicited and unsolicited

25. Using the SAGs • SAGs offer a ready format for evaluation • In most instances, there are multiple criterion statements for each standard • Each criterion measure focuses on a particular aspect of the standard, allowing raters to express more detailed and specific judgments

26. Criterion Measure Rating Scale ND 1 2 3 4 NR Not Done Not met Minimally Well Fully Not Rated Met Met Met ND 1 2 3 4 NR Not Done Not met Minimally Well Fully Not Rated Met Met Met • Using this scale, consider each criterion statement and decide… • The extent to which each criterion measure has been met by the program or service

27. Assessment Criteria Example Part 6: FINANCIAL RESOURCES Counseling Services (CS) must have adequate funding to accomplish its mission and goals. Funding priorities must be determined within the context of the stated mission, goals, objectives, and comprehensive analysis of the needs and capabilities of students and the availability of internal or external resources. CS must demonstrate fiscal responsibility and cost effectiveness consistent with institutional protocols. ND 1 2 3 4 NR Not Done Not Met Minimally Met Well Met Fully Met Not Rated PART 6. FINANCIAL RESOURCES (Criterion Measures) Rating Scale NOTES 6.1 The program has adequate funding to accomplish its mission and goals. ND 1 2 3 4 NR 6.2 Funding priorities are determined within the context of program mission, student needs, and available fiscal resources. ND 1 2 3 4 NR 6.3 The program demonstrates fiscal responsibility and cost effectiveness consistent with institutional protocols. ND 1 2 3 4 NR Part 6: Financial Resources Overview Questions A. What is the funding strategy for the program? B. What evidence exists to confirm fiscal responsibility and cost-effectiveness?

28. CAS: Work Form AAssessment, Ratings, and Significant Items CAS Work Form A Assessment, Ratings, and Significant Items INSTRUCTIONS: This work form should be completed following individual ratings of the participants. For each of the 13 Parts, identify (circle) the criterion measure item number(s) in the column labeled for which there is a substantial rating discrepancy (two or more ratings apart). Items not circled should reflect consensus among judges that practice in that area is satisfactory. Items where judgment variance occurs need to be discussed thoroughly by team members. Follow this action by determining which practices (criterion measures) can be designated as “excellent” or “unsatisfactory” and record them in the Step One column. In Step Two, list the items requiring follow-up action including any criterion measure rated as being unsatisfactory by any reviewer.

29. CAS: Work Form A continued

30. CAS Work Form B: Follow Up Actions INSTRUCTIONS: The purpose of this work form is to begin the planning for action to be taken on practices judged to merit follow-up (See Step 1, Work Form A). In Step Three, transfer short descriptions of the practices requiring follow-up and detail these items using the table format provided. Step Three: Describe the current practice that requires change and actions to initiate the change

31. CAS Work Form C:Summary Action Plan Step Four: This form concludes the self-assessment process and calls for action to be taken as a consequence of study results. Write a brief action plan statement in the spaces below for each Part in which action is required.

32. Closing the loop • Purpose & scope of study is defined • Self-study team is trained • Self-study is conducted • Evaluative evidence is collected • Discrepancies are identified • Appropriate actions are determined • Special actions for program enhancement are recommended • Action plan is developed and communicated

33. Student Outcomes Assessment • What is the effect of our work on students? • How are they different as a result of interacting with our programs and services? • What have students learned? • How have they developed?

34. Why Focus on Student Learning? • Accountability • Alignment with institution’s mission • Consistent with language of the academy • Quality improvement • More effective learning • Better defined and more effective planning and strategy development

35. Defining Learning Outcomes “Outcomes usually identify growth in some dimension of knowing, being or doing.” Susan Komives & Sarah Schoper, “Developing Learning Outcomes,” Learning Reconsidered 2 (2006) • Knowledge (cognitive) • Attitude (affective) • Skill (psychomotor)

36. CAS: Student Learning & Development Outcomes • Knowledge acquisition, construction, integration and application • Cognitive complexity • Intrapersonal development • Interpersonal competence • Humanitarianism & civic engagement • Practical competence

37. Plan for Assessing Student Learning Outcomes • Review/revise mission • Identify major services, programs, activities • Specify desired student outcomes • Identify information needed to show student learning • Identify existing and new data needed • Develop process for measuring outcomes • Identify ways to use findings to change/improve

38. Writing Learning Outcomes • Identify words/phrases that describe what an ideal program participant or college graduate would be able to: • Know – Be – Do … as a result of engaging with your program, activity or service. • What would he/she learn from the experience?

39. Student Outcomes Matrix

40. Writing Learning Outcomes • Describe the learner – characteristics of users/participants of programs and services. What knowledge, skills, experiences do they start with? • Determine acceptable levels of performance • Consider how you would provide evidence of achievement

41. Ways to measure learning outcomes • Surveys, questionnaires, self-report (indirect) • Observations • Interviews • Focus groups • Reflective journals • Performance evaluations • Pre-test/post-test • Portfolios • Capstone projects • Review of project/performance by peers and internal/external reviewers

42. Tips, suggestions • Identify all your program outcomes • Rank outcomes based on importance • Activity outcome • Program outcome • Institutional or division outcome • Identify where learning opportunities are occurring (mapping) • Design and implement assessment cycle – one outcome at a time

43. Remember: • Start small • Start doable • Start narrow • Start targeted • Start! • One program, one workshop, one desired outcome, one measured tool

44. Additional CAS Resources • www.cas.edu • Standards book • Prologue & Context with “How to” information • FALDOs – book and CD formats as package • SAGs – CD or individually