Developing an Assessment Culture: Strategic Approaches to Faculty Development in Assessment

1. Developing an Assessment Culture: Strategic Approaches to Faculty Development in Assessment Herb Amato, DA, ATC Megan Rodgers, MA KestonFulcher, PhD Kristen Smith, BS

2. Quick Survey • Where are you in the SACS Cycle? • Visit within this year • Writing report within the next 18 months • Approaching 5 year interim • 5-10 year period, nothing due

3. Workshop Agenda • Introduction: Assessment Improvement • Using Assessment Results for Improvement • Faculty Development • Group Activity • Break • Resource Sharing • Envisioning an Improved Institution • Discussion

4. Learning Outcomes As a result of this workshop, participants will: • Identify the six steps in the assessment model and explain how quality can vary at each step • Identify where faculty members stand in their assessment development progress and which resource is best-suited for a faculty member at that stage of development • Identify faculty development opportunities they can immediately offer • Explain how institutions can strategically enhance assessment culture through faculty development • Synthesize ideas about how an institution can use assessment to truly impact student learning.

5. Why practice Assessment? • Program Improvement • Helps program developers identify areas of improvement for the program • Shows program developers the actual impact the program has on students • Recruitment • Provides parents with evidence of the value a program has for their child • Gives prospective students evidence of why they should participate • Accountability • Meets University annual program reporting and program review requirements • May apply to other funding or regulatory agencies’ requirements • Addresses accrediting agency program evaluation requirements

6. Assessment is Cyclical • 3.3.1…. states that an institution… • “identifies expected outcomes, assesses the extent to which it achieves those outcomes, and provides evidence of improvement based on analysis of the results in …(3.3.1.1) educational programs, to include student learning outcomes.”

7. Establishing Objectives 1 • Objectives refer to expected or intended student outcomes • What are the specific knowledge, skills, or attitudes that students are expected to achieve as a function of your program? • Objectives should be: • Measurable • Specific • Student Centered

8. 2 • Aligning Objectives to Curriculum

9. 3 • After objectives established and learning opportunities are in place, we turn to measuring the objectives. There are two options: • Find a pre-existing instrument • Or, if existing instruments do not closely match the objectives, an instrument can be created • Measure should naturally follow your objectives • Define Research Design • Pre/post? • Embedded assessment? • Systematic Measurement of Objectives

10. 4 • Group Differences: Do we see expected differences in performances by different student groups? (Such as those in your program and those that aren’t.) • Relationships: Do we see relationships between performances and grades in relevant courses? • Growth: Do student performances change over time? • Competency: Do students meet performance expectations? • Analyze • Results

11. 5 • Assessment Report should be shared with others • Program faculty members • External stakeholders • e.g., students, advisory councils • Mode of communication should be clear • Share • Results

12. 6 • Use Results for Program Improvement • This is the stage where “closing the loop” begins • Involves feedback from program stakeholders and careful consideration of the assessment results • Target: make program improvements based upon assessment results • Successful “closing the loop” is confirmed only with new data collection showing impact and improvement in diagnosed problem areas

13. Assessment is Cyclical • Assessment Quality can vary in any of these areas

14. Assessment Improvement • Assessment is increasingly practiced • Expectations of quality assessment are less common • Why does quality assessment matter?

15. Example B.A. program in NFL History B.A. program in 80s Pop Culture Well defined objectives Objectives mapped to curriculum Uses at least one direct measure to assess all objectives (100% response rate) Makes curricular changes based on assessment results • Well defined objectives • Objectives mapped to curriculum • Uses an alumni survey to assess all objectives and one senior satisfaction survey (15% response rate on each) • Makes curricular changes based on assessment results What is the difference in assessment practice? Does it matter?

16. Assessment Quality • Enables one to “trust” inferences made from results • Is necessary if assessment is to lead to program improvement • Can we gauge assessment quality?

17. Measuring Assessment quality

18. “Meta-Assessment” • “Meta-Assessment”- assessing the assessment cycle (Ory, 1992) • Specifically, assessing the assessment quality

19. Meta-Assessment Research • Over 50 institutions practice “meta-assessment”(Fulcher, Swain, Orem, 2012) • Typically a rubric is used to evaluate an assessment report • Assessment coordinators who utilize resources and consultation improve assessment quality (Rodgers, Grays, Fulcher, & Jurich, 2012) • Programs who more frequently consult have higher assessment quality (Fulcher & Bashkov, 2012)

20. Advantages of Meta-Assessment • Identify individual program assessment strengths and weaknesses • Allows for a common reporting mechanism across the university • Allows institutional view of assessment quality • Via aggregate data

21. Using Results for Improvement Map Methods/Design Results Use Obj. Report Rubric

22. Misconceptions • 1) Quality assessment leads to improved learning • 2) The more resources you have the more improved learning will occur • 3) Simply making a change is an improvement

23. What happens after Assessment Quality is achieved? Wabash Link

24. What Obstacles Impede Use of Assessment Results for Improvement?

25. Variability in Faculty Assessment Knowledge • Faculty development opportunities can allow the institution to meet the faculty member responsible for assessment “where they are” • Generally, there are “Beginners”, “Intermediates”, and “Advanced”.

26. Resources • Resources are critical to assessment improvement • What kinds of resources are available to assessment practitioners?

27. Resource Possibilities Easily Accessible • Continuum of Resources and Experiences Involved Experiences

28. Books & Conferences Easily Accessible • Assessment Books • Assessment Conferences • SACSCOC • Assessment Institute • Association of Institutional Research • Community College conference on Learning Assessment • Many more! * Books and Conferences Informational Videos Reporting Expectations One on One Consultation Dedicated Time Fellowship Rating Experience A Provost Award for Assessment Certificate Program Involved Experiences

29. Assessment Videos Easily Accessible http://tinyurl.com/assessmentvid Books and Conferences Informational Videos Reporting Expectations One on One Consultation Dedicated Time Fellowship Rating Experience A Provost Award for Assessment Certificate Program * Involved Experiences

30. Assessment Report Exemplar Easily Accessible http://tinyurl.com/asmt-exemplar Books and Conferences Informational Videos Reporting Expectations One on One Consultation Dedicated Time Fellowship Rating Experience A Provost Award for Assessment Certificate Program * Involved Experiences

31. Consultation Easily Accessible • Consultation with an assessment expert • Most commonly identified support mechanism in qualitative study • Could be: • Institutional Effectiveness professional • Knowledgeable faculty members • Graduate students • Students in statistics courses • Could hire an external consultant Books and Conferences Informational Videos Reporting Expectations One on One Consultation Dedicated Time Fellowship Rating Experience A Provost Award for Assessment Certificate Program * Involved Experiences

32. “Assessment Lockdown” Easily Accessible Books and Conferences Informational Videos Reporting Expectations One on One Consultation Dedicated Time Fellowship Rating Experience A Provost Award for Assessment Certificate Program • Partnered with on-campus faculty professional development center • Hosted a 3 hour “Assessment Lockdown” • Faculty worked on their assessment reports • Experts “on-hand” • Faculty reported this to be a very beneficial experience * Involved Experiences

33. Assessment Fellows Easily Accessible Books and Conferences Informational Videos Reporting Expectations One on One Consultation Dedicated Time Fellowship Rating Experience A Provost Award for Assessment Certificate Program • Three week experience • Faculty work solely on an assessment project for their program • Residency in on-campus assessment consulting center • Work with assessment office staff * Involved Experiences

34. Faculty Assessment Raters Easily Accessible Books and Conferences Informational Videos Reporting Expectations One on One Consultation Dedicated Time Fellowship Rating Experience A Provost Award for Assessment Certificate Program • Faculty representatives from all Colleges • Two week experience • Two days of training • Worked with assessment office staff daily to evaluate assessment reports * Involved Experiences

35. Assessment Excellence Award Easily Accessible Books and Conferences Informational Videos Reporting Expectations One on One Consultation Dedicated Time Fellowship Rating Experience A Provost Award for Assessment Certificate Program • Annual award given through Provost’s office • Program receives recognition plaque and small honorarium * Involved Experiences

36. Assessment Certificate Easily Accessible http://www.jmu.edu/outreach/assessment.shtml Books and Conferences Informational Videos Reporting Expectations One on One Consultation Dedicated Time Fellowship Rating Experience A Provost Award for Assessment Certificate Program • Professional Development program • Online • Four courses taught by assessment professionals • Two Year commitment * Involved Experiences

37. Targeted Audience Easily Accessible Beginner Intermediate Advanced Books and Conferences Informational Videos Reporting Expectations One on One Consultation Dedicated Time Fellowship Rating Experience A Provost Award for Assessment Certificate Program Involved Experiences

38. Assessment Culture Beginner Intermediate Advanced • Fearless • Empowerment • Resource to • Others

39. Group Activity • Work with your table to brainstorm resources and faculty development opportunities for your hypothetical faculty member.

40. SHARE • Each Scenario will report their proposed solutions

41. Envision • Institution had to focus on having better writers • What would you do to ensure writing quality improved? • How would you use assessment to capture writing improvement?

42. Take Aways • XX% of institutions have recommendations about 3.3.1.1 • Think of where your university is now and where it could be in 10 years • How could you get there? • Presently, assessment is the current issues, but ultimately we want to be a place where learning is the bottom line

43. Action Plan Worksheet • Fill out individually • Discuss with Table

44. References • Fulcher, K.H. & Bashkov, B. M. (2012, November-December). Do we practice what we preach? The accountability of an assessment office. Assessment Update. 24(6). 5-7, 14. • Fulcher, K.H., Swain, M., & Orem, C.D., (2012). Expectations for assessment reports: A descriptive analysis. Assessment Update 24(1), 1-2, 14-16. doi:10.1002/au.241 • Orem, C. (2012) Demonstrating validity evidence of meta-assessment scores using generalizability theory (Unpublished doctoral dissertation). James Madison University, Harrisonburg, VA. • Ory, J.C., (1992). Meta-assessment: Evaluating assessment activities. Research in Higher Education, 33(4). 467-481. • Rodgers, M., Grays, M. P., Fulcher, K. H., & Jurich, D. P. (2012). Improving academic program assessment: a mixed methods studyc. Innovative Higher Education, 38(5), doi: http://dx.doi.org/10.1007/s10755-012-9245-9