CREATING A CULTURE OF ASSESSMENT

1. CREATING A CULTURE OF ASSESSMENT Closing the Loop Dean Kenneth B. Roberts, Ph.D. Heidi M. Anderson, Ph.D. Eleanora Bird, M.S. Jeff Cain, M.S. Office of Education Innovation University of Kentucky College of Pharmacy

2. Outline • Literature “Creating a Culture of Assessment” • UKCOP Case Study • Created an office to handle the College assessment efforts • Faculty development and involvement • Student involvement • Documenting current assessment efforts • Future Directions at UKCOP

3. Creatinga Culture of Assessment 3

4. Culture of Assessment • Leadership must be ‘on board’ • Mission, resources, use results for improvement • Involve all stakeholders • Faculty • Students • Alumni

5. Culture of Assessment • Make assessment relevant • Course reviews • Surveys, rubrics • Focus on teaching and learning rather than assessment • Focus conversations not on assessment, but on helping students learn

6. Culture of Assessment • Provide opportunities to learn about assessment • Minimize the burden of assessment on faculty and students • Provide centralized leadership, coordination and support for assessment • Provide resources • Keep paperwork to a minimal

7. Culture of Assessment • Start small, Set up clear expectations and be flexible • Celebrate and reward assessment efforts • Give faculty and staff written recognition • Encourage leaders to commend publicly • Provide special funding • Honor faculty and staff who obtain external grants that will strengthen assessment

8. Culture of Assessment • Use assessment results appropriately • Keep faculty, students and staff informed • Don’t penalize faculty whose assessments results are less than positive • Don’t expect everyone to get on board

9. (UKCOP) 9

10. Vision • Office of Education Innovation (OEI) was developed to assist faculty • Teaching • Learning • ASSESSMENT • Technology • Scholarship of Teaching Today

11. Mission • The mission of the “Office of Education Innovation” (OEI) is to provide assistance to faculty in developing and evaluating the professional pharmacy and graduate programs within the University of Kentucky, College of Pharmacy (UKCOP).

12. Community • Dean’s Leadership Team • Office of Academic Affairs • Office of Education Innovation (4 FTEs) • Assistant Dean • Assessment Coordinator • Information Technology Officer • Administrative Assistant • Faculty • Students • Other Participants TEAM

13. www.mc.uky.edu/Pharmacy/oei

14. Who the office works with? • The “Office of Education Innovation” (OEI) works closely with: • UKCOP Course Coordinators/Directors • UKCOP Individual Faculty • UKCOP Departments • UKCOP Students • External Constituents to College

15. OEI Activities and Services • Curriculum • Assessment • Instructional Technology • Faculty Development (Enrichment) • Scholarship of Teaching Today

16. UKCOP Faculty Development: How, What and Why of Assessment? 16

17. Why do assessment? • Accountability • ACPE Accreditation • University Review • Curriculum committee requirements • Enhance student’s educational development • Decision-making

18. Accreditation • July 2000 accreditation standards require assessment activities to be aligned with mission and goals: • achievement indicators, • formative and summative formats, • outcomes assessment, and • systematic assessment plan.

19. Used for Improvement Ongoing Use Multiple Measures Assessment ProcessDriven by Mission Reassess

20. What is the Assessment Process? • “Systematic collection, review and use of information about educational programs undertaken for the purpose of improving student learning and development” • Palumbo and Banta, 1999, p. 3

21. Institutional Academic CURRICULUM COURSES Teaching-learning Methods Assessment Methods Individual STUDENTS: Learning Performance Attitudes Behaviors Skills Satisfaction Levels of Assessment

22. Institutional UNIV REVIEW ACCREDITATION Individual Faculty SELF Teaching Portfolio Peer Observation Scholarly Productivity Levels of Assessment

23. Student Student Learning Outcomes Performance Faculty Contribution to teaching Scholarly productivity National visibility Program Graduation rates Licensure pass/fail Curriculum Courses Teaching-Learning Methods What is Assessed?

24. Difference: Student Assessment vs. Program Assessment • Student Assessment is a teaching process designed to improve learning • Program Assessment is an institutional process designed to improve the program

25. Student Assessment • Carefully define your student learning outcomes • If you don’t know what you’re looking for, you will have a difficult time measuring it. • Outcome statements should describe student learning or behavior • Outcome: “the result”

26. Student Assessment • Start simple. Choose a limited number of outcomes to measure at first. • Begin modestly so that the data collected can help to define and refresh accomplishments

27. Faculty Involvement 27

28. Faculty Involvement • Faculty learning about assessment: • Website developed to keep them abreast of activities • Faculty participation on committees: • Assessment committee • Curriculum committee

29. Faculty Involvement • Self-study reports • Curricular purposes • Curricular Outcomes • Course Review (online submission) • Course Evaluation (online) • Survey development • PS Graduate Program • Requested peer evaluation of teaching

30. UKCOP General Outcomes • Foundation Knowledge • Critical Thinking • Information Management • Communications • Professionalism • Social Interaction, Citizenship, Leadership • Life-long Learning

31. UKCOP Professional Outcomes • Patient Care • Medication Distribution • Human Resources Management and Operations • Research • Teaching

32. Faculty Involvement • Focus on Teaching/Learning • Generate the appropriate assessment questions • Determine appropriate ways to measure outcomes • Decide on when to assess the students • Prepare and utilize the reports

33. Faculty Involvement • Developing Questions to Ask: • What do we want students to know, and do? • How will we know they are learning what we hope they will? • What is the effect of our curricula, courses, teaching, styles and educational tools? • Did our changes bring about the effect we desired?

34. Student Involvement 34

35. Student Involvement • Incoming orientation: • Explain the College’s assessment, rationale relating to accreditation standards, role of OEI • Review IRB protocol and informed consent • Assessments throughout their 4 years • Anonymous (code for tracking) • Reports in aggregate fashion

36. Student Participation • Curriculum Committee • Student Liaison Committee (SLC) • meets 3x per semester • Assessment Committee • Course • Performance Assessments • Other • Accreditation Visit, Program Review

37. UKCOPCase Study:Documenting Current Effort 37

38. Documenting Current Efforts Purpose • Determine what assessments were already taking place within the College • Develop an assessment inventory list and appropriate questions about student learning

39. Goal: Develop a Complete Student Profile • Self-assessment of outcome abilities • Attitudes towards peer/self assessment • Performance records from courses • CAPP results (integrated abilities) • Student portfolios • Yearly interviews • Post-graduation surveys

40. Three Phases • Phase 1: Needs Analysis • Phase 2: Recommendations and Plan Development • Phase 3: Align Assessment Plan, Outcome Indicators

41. Needs Analysis Research • Literature review was conducted to determine types of assessment data that might be useful to gather for the program’s assessment functions • Student records • Program evaluations • Student performance and competencies

42. Needs Analysis Data Audit • Class of 2002-2005: Student and educational outcomes records • Data obtained from various offices and files within the college: • Demographic • Entry-exam scores • Learning styles • GPAs

43. Needs Analysis Data Audit • Data portfolios for several student cohorts were examined • Faculty • Other sources: • Institutional Research Office • Boards of Pharmacy • State/local pharmacy associations

44. Data Audit Report

45. Needs Analysis Data Audit • Success • External IR officer requested to use the college audit form as a model for the campus • Data are being collected from all audiences in a wide variety of venues • Some data are being disseminated in meaningful way

46. Lessons Learned from Data Audit • Amount & breadth of information was considerable • Gaps/Duplication • Some areas require more or different data

47. Lessons Learned from Data Audit • Length and frequency of surveys • Faculty engagement • Incorporate findings into College assessment plan • Reassess

48. Phase 2 Recommendations about research questions to assessment committee • I-E-O data management model used to classify student and program data (Astin, 1993) I=inputs E=environments O=outcomes

49. Phase 2 A. Content Analysis B. Educational Impact Which educational experiences/environments maximize student learning during the first year of pharmacy school?

50. Phase 2 Ho: Structured Clinical Instructive Models (SCIMs), prior to students participating in EPPE, statistically improve student critical thinking skills. Ho: Videotaped counseling sessions statistically improve student communication skills during the first year of pharmacy school.