650 likes | 683 Views
CREATING A CULTURE OF ASSESSMENT. Closing the Loop. Dean Kenneth B. Roberts, Ph.D. Heidi M. Anderson, Ph.D. Eleanora Bird, M.S. Jeff Cain, M.S. Office of Education Innovation University of Kentucky College of Pharmacy. Outline. Literature “Creating a Culture of Assessment”
E N D
CREATING A CULTURE OF ASSESSMENT Closing the Loop Dean Kenneth B. Roberts, Ph.D. Heidi M. Anderson, Ph.D. Eleanora Bird, M.S. Jeff Cain, M.S. Office of Education Innovation University of Kentucky College of Pharmacy
Outline • Literature “Creating a Culture of Assessment” • UKCOP Case Study • Created an office to handle the College assessment efforts • Faculty development and involvement • Student involvement • Documenting current assessment efforts • Future Directions at UKCOP
Culture of Assessment • Leadership must be ‘on board’ • Mission, resources, use results for improvement • Involve all stakeholders • Faculty • Students • Alumni
Culture of Assessment • Make assessment relevant • Course reviews • Surveys, rubrics • Focus on teaching and learning rather than assessment • Focus conversations not on assessment, but on helping students learn
Culture of Assessment • Provide opportunities to learn about assessment • Minimize the burden of assessment on faculty and students • Provide centralized leadership, coordination and support for assessment • Provide resources • Keep paperwork to a minimal
Culture of Assessment • Start small, Set up clear expectations and be flexible • Celebrate and reward assessment efforts • Give faculty and staff written recognition • Encourage leaders to commend publicly • Provide special funding • Honor faculty and staff who obtain external grants that will strengthen assessment
Culture of Assessment • Use assessment results appropriately • Keep faculty, students and staff informed • Don’t penalize faculty whose assessments results are less than positive • Don’t expect everyone to get on board
(UKCOP) 9
Vision • Office of Education Innovation (OEI) was developed to assist faculty • Teaching • Learning • ASSESSMENT • Technology • Scholarship of Teaching Today
Mission • The mission of the “Office of Education Innovation” (OEI) is to provide assistance to faculty in developing and evaluating the professional pharmacy and graduate programs within the University of Kentucky, College of Pharmacy (UKCOP).
Community • Dean’s Leadership Team • Office of Academic Affairs • Office of Education Innovation (4 FTEs) • Assistant Dean • Assessment Coordinator • Information Technology Officer • Administrative Assistant • Faculty • Students • Other Participants TEAM
Who the office works with? • The “Office of Education Innovation” (OEI) works closely with: • UKCOP Course Coordinators/Directors • UKCOP Individual Faculty • UKCOP Departments • UKCOP Students • External Constituents to College
OEI Activities and Services • Curriculum • Assessment • Instructional Technology • Faculty Development (Enrichment) • Scholarship of Teaching Today
UKCOP Faculty Development: How, What and Why of Assessment? 16
Why do assessment? • Accountability • ACPE Accreditation • University Review • Curriculum committee requirements • Enhance student’s educational development • Decision-making
Accreditation • July 2000 accreditation standards require assessment activities to be aligned with mission and goals: • achievement indicators, • formative and summative formats, • outcomes assessment, and • systematic assessment plan.
Used for Improvement Ongoing Use Multiple Measures Assessment ProcessDriven by Mission Reassess
What is the Assessment Process? • “Systematic collection, review and use of information about educational programs undertaken for the purpose of improving student learning and development” • Palumbo and Banta, 1999, p. 3
Institutional Academic CURRICULUM COURSES Teaching-learning Methods Assessment Methods Individual STUDENTS: Learning Performance Attitudes Behaviors Skills Satisfaction Levels of Assessment
Institutional UNIV REVIEW ACCREDITATION Individual Faculty SELF Teaching Portfolio Peer Observation Scholarly Productivity Levels of Assessment
Student Student Learning Outcomes Performance Faculty Contribution to teaching Scholarly productivity National visibility Program Graduation rates Licensure pass/fail Curriculum Courses Teaching-Learning Methods What is Assessed?
Difference: Student Assessment vs. Program Assessment • Student Assessment is a teaching process designed to improve learning • Program Assessment is an institutional process designed to improve the program
Student Assessment • Carefully define your student learning outcomes • If you don’t know what you’re looking for, you will have a difficult time measuring it. • Outcome statements should describe student learning or behavior • Outcome: “the result”
Student Assessment • Start simple. Choose a limited number of outcomes to measure at first. • Begin modestly so that the data collected can help to define and refresh accomplishments
Faculty Involvement • Faculty learning about assessment: • Website developed to keep them abreast of activities • Faculty participation on committees: • Assessment committee • Curriculum committee
Faculty Involvement • Self-study reports • Curricular purposes • Curricular Outcomes • Course Review (online submission) • Course Evaluation (online) • Survey development • PS Graduate Program • Requested peer evaluation of teaching
UKCOP General Outcomes • Foundation Knowledge • Critical Thinking • Information Management • Communications • Professionalism • Social Interaction, Citizenship, Leadership • Life-long Learning
UKCOP Professional Outcomes • Patient Care • Medication Distribution • Human Resources Management and Operations • Research • Teaching
Faculty Involvement • Focus on Teaching/Learning • Generate the appropriate assessment questions • Determine appropriate ways to measure outcomes • Decide on when to assess the students • Prepare and utilize the reports
Faculty Involvement • Developing Questions to Ask: • What do we want students to know, and do? • How will we know they are learning what we hope they will? • What is the effect of our curricula, courses, teaching, styles and educational tools? • Did our changes bring about the effect we desired?
Student Involvement • Incoming orientation: • Explain the College’s assessment, rationale relating to accreditation standards, role of OEI • Review IRB protocol and informed consent • Assessments throughout their 4 years • Anonymous (code for tracking) • Reports in aggregate fashion
Student Participation • Curriculum Committee • Student Liaison Committee (SLC) • meets 3x per semester • Assessment Committee • Course • Performance Assessments • Other • Accreditation Visit, Program Review
Documenting Current Efforts Purpose • Determine what assessments were already taking place within the College • Develop an assessment inventory list and appropriate questions about student learning
Goal: Develop a Complete Student Profile • Self-assessment of outcome abilities • Attitudes towards peer/self assessment • Performance records from courses • CAPP results (integrated abilities) • Student portfolios • Yearly interviews • Post-graduation surveys
Three Phases • Phase 1: Needs Analysis • Phase 2: Recommendations and Plan Development • Phase 3: Align Assessment Plan, Outcome Indicators
Needs Analysis Research • Literature review was conducted to determine types of assessment data that might be useful to gather for the program’s assessment functions • Student records • Program evaluations • Student performance and competencies
Needs Analysis Data Audit • Class of 2002-2005: Student and educational outcomes records • Data obtained from various offices and files within the college: • Demographic • Entry-exam scores • Learning styles • GPAs
Needs Analysis Data Audit • Data portfolios for several student cohorts were examined • Faculty • Other sources: • Institutional Research Office • Boards of Pharmacy • State/local pharmacy associations
Needs Analysis Data Audit • Success • External IR officer requested to use the college audit form as a model for the campus • Data are being collected from all audiences in a wide variety of venues • Some data are being disseminated in meaningful way
Lessons Learned from Data Audit • Amount & breadth of information was considerable • Gaps/Duplication • Some areas require more or different data
Lessons Learned from Data Audit • Length and frequency of surveys • Faculty engagement • Incorporate findings into College assessment plan • Reassess
Phase 2 Recommendations about research questions to assessment committee • I-E-O data management model used to classify student and program data (Astin, 1993) I=inputs E=environments O=outcomes
Phase 2 A. Content Analysis B. Educational Impact Which educational experiences/environments maximize student learning during the first year of pharmacy school?
Phase 2 Ho: Structured Clinical Instructive Models (SCIMs), prior to students participating in EPPE, statistically improve student critical thinking skills. Ho: Videotaped counseling sessions statistically improve student communication skills during the first year of pharmacy school.