Download
1 / 73
730 likes | 1.05k Views
Beyond SLOs and SAOs: The Importance of Rubrics and Completing the Assessment Cycle CSSO Southern Drive-In Workshop Menifee Campus, Mt. San Jacinto College Friday, October 31, 2008. Juan Avalos, Ph.D. Dean Student Services and Counseling Cosumnes River College. A little bit about myself….
E N D
Beyond SLOs and SAOs: The Importance of Rubrics and Completing the Assessment Cycle CSSO Southern Drive-In WorkshopMenifee Campus, Mt. San Jacinto CollegeFriday, October 31, 2008 Juan Avalos, Ph.D. Dean Student Services and Counseling Cosumnes River College
A little bit about myself… • Formal training in Higher Education. • 10 years of direct experience with educational assessment and research(national research institutes, 4-yr private, and 4-yr public). • 12 years of management experience in higher education(4-yr private, 4-yr public, and 2-yr public). • Happily married w/ four beautiful kids and a black & white Chihuahua named “Oreo”.
Presentation Deliverables: Concepts Program Models Sample Processes Examples Hands-on Exercise Resources
Presentation Goals: • Everyone walks away with at least one new understanding or piece of information. • Everyone walks away with at least one process or concept that can be implemented or used on your campus next week.
Taking a Quick Pulse… “Everyone, Please Close Your Eyes…” Thumbs Up, Down, or Side Ways? • I am confident in my ability to lead my Student Services division through the development of a robust “continuous improvement” system on my campus? • My understanding of assessment concepts and theoretical frameworks guides my day-to-day conversations with program leads in development of said system? • I am aware of specific assessment processes and am confident in my ability to lead my managers, faculty, and staff through said processes?
Main Points Understanding the core elements of continuous improvement models, having a research conceptual framework, and being able to gauge the relative strength of the evidence you collect is critical to keeping you from being overwhelmed by the complexity of the assessment process.
Main Points…(‘cont) Attention has been and will continue to be devoted to the articulation of SLOs and SAOs, don’t forget about the importance of rubrics and going through the entire assessment cycle. …There will be time for further refinement, increased rigor, and broader scope with each iteration.
Core Elements of “Continuous Improvement” Models • Desired outcome(s) • Activities designed to address stated outcomes • Assessment of progress on said outcomes • Dialog & reflection of research/findings • Implementation of improvement plan based on previous four steps (a.k.a. “closing the loop”). …..then back around again.
Various Business and Education “Continuous Improvement” Methodologies More in-depth descriptions and additional information can be found at http://www.isixsigma.com/me/ Balanced Scorecard Baldrige Benchmarking Business Process Reengineering Deming Document Control DMADV DMAIC Financial Analysis/Cost Quality ISO 9000 Lean Plan, Do, Check, Act—PDCA Process Management Project Selection Simulation Six Sigma Total Quality Management Theory of Inventive Problem Solving
Common Images of “Continuous Improvement” Models… Plan: Design or revise business process components to improve resultsDo: Implement the plan and measure its performanceCheck: Assess the measurements and report the results to decision makersAct: Decide on changes needed to improve the process
Common Images of “Continuous Improvement” Models… Baldridge Framework Compared to ISO
BEWARE OF…. NOISE Lots of terms, concepts, models, tools, vendors, and phrases tied to this process that can be tremendously overwhelming, be counter productive, and turn into NOISE. One Example: Over 200 terms present in Instructional Assessment Resources website. //www.utexas.edu/academic/diia/assessment/iar/glossary.php
One of our challenges… Overcoming a “Transactions” Framework Do you teach? If yes, you have SLOs. If no, you are likely to only have SAOs.
One of our challenges… (‘cont) We are probably “ok” with Service Area Outcomes since Student Service areas are traditionally “used to” thinking about operational efficiencies, service to students, student satisfaction w/ services, etc. …the quality of service we provide and it’s impact on the student (i.e. student satisfaction, responsiveness of service, indirect impact on other student success measures, etc.).
One of our challenges…(‘cont) Student Learning Outcomes will be more difficult as Student Service areas have not traditionally viewed themselves as contributing to Student Learning...viewed themselves primarily as administrative support units. The necessary paradigm shift from “transactions” to “student learning” will require us to view what we do from a “do I teach?” lens. If “yes”…What do I teach? Where is my “classroom”? What are my “lesson plans”? How do I know students are learning what I’ve intended for them to learn? etc.
Walking the Walk at CRC Prior to 2004 at CRC Limited opportunity for Student Services faculty and staff to come together and dialog about organizational success. Student Services participation in campus Program Review and Unit Planning processes were isolated and misaligned. Noel-Levtiz SSI findings pointing to potential performance gaps, in key Student Services, over an eight-year span, viewed as “inaccurate” due to perceived concerns over rigor of survey instrument and proctoring process. Focus of many departmental meetings on “updates” and reasons why improvements cannot be made without additional people and/or funding.
Walking the Walk… Over the Past Two Years at CRC Increased opportunity for Student Services faculty and staff to come together and dialog on organizational success. Second Student Services Summit scheduled for October 24, 2008. Dean’s counseling meeting only focuses on issues for dialog and reflection…need for “updates” handled via weekly email. Weekly email also includes a “Leadership Quote for the Week”, which provides an opportunity for reflection on important concepts connected to issues we are experiencing at the time.
Walking the Walk… Over the Past Two Years at CRC (‘cont) Student Services participation in campus Program Review more inclusive, but product a bit disjointed. Unit Plan, however, more clear and precise in articulating department-wide issues, identified strategies, and necessary resources, where applicable. Could have dwelled on the “imperfections” of our Performance Review document, but decided to accepted it as is, take from it as much insight as we could, and focus on production of our Unit Plan. Moving forward paid off.
Walking the Walk… This Year…reaping the benefits of laying our foundation four years ago. Reduction in resistance to accept that twelve-year trend of Noel-Levtiz SSI findings pointing to potential performance gaps may be “real”. Counselors willing to consider conversation on what actions we can take in response to concerns surfaced by SSI findings. Counseling faculty engaged in Fall 08 Convocation SLO Dialog Exercise. Willing to “go off task” and entertain initial conversation on SLO competence levels, as a precursor to a broader discussion about appropriate assessment measures.
Walking the Walk… This Year…(‘cont) Faculty willing to continue work started during Convocation. They will propose a method for going through the exercise. It may involve smaller groups of faculty initially (…producing draft work to share with Counseling core) and a process that is not as “boring”. Next series of conversations likely to focus on how we will ensure that we provide opportunities for students to achieve our desired SLOs within the context of our regular counseling sessions (i.e. “what will you do, in sessions with students, to help them learn the skills, abilities, etc. we expect them to learn?”).
Getting to Work… • Instructions: • You will be walking to Bldg. #800, Community Room. • You will be grouped by candy wrapper. • Exercise directions and worksheets are at your table. • I need you back in this Assembly Hall by 11:19 am.
Break-Out Session Debrief Think about the PROCESS…. …and what you were feeling as you went through the process. • What were your conversations like? • What were you able to do with relative ease? • What did you need to M-u-d-d-l-e Through?
One Research Framework to Consider Astin’s I-E-O Framework ENVIRONMENT OUTCOME INPUTS
A Guide to Using Evidence in the Accreditation Process: A Resource to Support Institutions and Evaluation Teams Working Draft (January 2002) WASC Senior Colleges & Universities Primary Author: Peter Ewell
What is Evidence?(from WASC Evidence Guide) Evidence is the substance of what is advanced to support a claim that something is true. Evidence… • Intentional and purposeful. • Entails interpretation and reflection. • Integrated and holistic. • Can be both quantitative and qualitative. • Can be either direct or indirect.
What is Evidence? (‘cont)(from WASC Evidence Guide) Syndromes to avoid when using evidence: • Trying to measure everything. • Trying to be too precise. • Premature closure.
What Constitutes Good Evidence?(from WASC Evidence Guide) Five principles to using good evidence: • Relevant…demonstratively related to the question being investigated. • Verifiable…is documentable and replicable. • Representative…typical, not an isolated case. • Cumulative…gains credibility as additional sources or methods are employed. • Actionable…reflectively analyzed and interpreted in way to provide guidance for action/improvement.
Are you asking yourself… How do I ensure that my Student Service operations are effective?
Are you asking yourself… (‘cont) In addition to working closely with your Student Services faculty and staff in the identification of SAOs and SLOs, attention must be given to the following: Moving your operations through the entire Assessment Cycle. It’s okay not to be perfect the first time around. There will be time for further refinement, increased rigor, and broader scope with each iteration….
