ASSESSMENT for Administrative Departments and Educational Support Units September/October 2002

1. ASSESSMENTfor Administrative Departments and Educational Support UnitsSeptember/October 2002

2. Overview • What’s New? • Latest developments in assessment • Review new deadlines • What’s Next? • Finish old business (complete 01-02 reports) • Prepare for next assessment cycle • Questions / Resources

3. Timing of Assessment Cycle • Fall too busy – move cycle to spring • Finish current (01-02) assessment cycle now • Columns/steps 4 & 5 of current assessment reports due Oct. 15, 2002 • Slight reprieve • Delay start of next cycle (03-04) until spring • Steps 1-3 due spring 2003 • Steps 4 & 5 due spring 2004

4. Planning and Assessment Portal • Created a single point of entry for strategic planning and assessment activities • Access through ERAU Online / Blackboard • Overview of planning and assessment processes • Directory of assessment and strategic planning units • University Planning and Assessment Policy (APPM 4.3)

5. Other Changes • “Best practices” adopted from institutions with successful assessment process: • Use committees to guide ongoing assessment • Set up “peer review” process for unit reports • Annual summary of university-wide assessment • Incentives for doing assessment well • Establish centralized support services

6. Assessment Committees • Committee structure pushes ownership of assessment process further down into ERAU’s “foundation” (faculty/staff) • 4 Campus Assessment Committees (CAC) DB, EC, Prescott, and University Admin (UA) • Faculty & staff on each CAC (UA – staff only) CAC co-chaired by a faculty and a staff member • CAC co-chairs = University Assessment Committee (UAC)

7. CAC Duties • In broad terms, CAC will… • Work with individual units to develop effective assessment reports • Facilitate communication across departments and programs • Be a vehicle for feedback to / from Chancellor / VPs • Nominate exemplary assessment reports

8. CAC Duties • Work with individual units to develop effective assessment reports • Guidance on 5-Step model / assessment process • Peer review assessment reports at 2 points in time using a checklist of assessment guidelines • Upon draft submission of Steps 1-3 at start of assessment cycle • Submission of Steps 4 & 5 at the end of the assessment cycle • Peer review is to ensure the PROCESS; interpretation of results and the decision about what to do with them is YOUR call - this is NOT a prescriptive review!

9. CAC Duties • Facilitate communication across departments and programs about assessment and continuous improvement • Encourage sharing of best practices and useful assessment measures • Avoid duplication of efforts • Summarize campus assessment activities

10. CAC Duties • Nominate exemplary assessment reports as part of incentive • Submit “best assessment report” nominees to UAC • UAC will pick “best reports” • “Best reports” selected by UAC will receive $500 to use toward assessment-related activities

11. CAC Members CAC-UACAC-DB Susan Erdman Tim Brady Allison Kish Rich Clarke Shirley Waterhouse Tina Frederick Kathy Welch Tom Hilburn Jim Libbey Linda Manning Terry Mularkey Notis Pagiavlas Paula Reed

12. UAC Duties • Unify campus-level assessment activities • Facilitate communication re: assessment and continuous improvement across campuses and UA; encourage sharing of best practices • Annual summary of assessment activities at the university level • Vehicle for feedback to / from Cabinet • Vote on “best assessment reports” nominated by CACs

13. Institutional Research Duties • Institutional Research (IR) moves to a supporting role • Assessment coordinator available to assist with • development of assessment measures • administration of surveys for assessment • id existing sources of data (IR and external) • help do research for assessment techniques • IR website has survey data and project calendar • IR houses archived assessment reports

14. Complete Current Assessment • Close out 01-02 assessment report • Download current report from assessment website • Steps 1-3 were submitted last October • Word format this cycle; web-based next cycle • Complete Steps 4 & 5 • Summary of Data Collected • Use of Results • Submit completed reports to Assessment Coordinator by Oct. 15, 2002

15. Complete Current Assessment • Steps 4 & 5 are straightforward IF…. • Assessment data were actually gathered • Data provided information to determine if outcomes were actually met • Thought was put into how various results might be used

16. Possible Scenarios • “Winning” Scenarios • Criteria for success were met • Criteria for success were NOT met and results were used to make improvements (even better?) • “Problematic” Scenarios • No use of results are shown • Insufficient data without offering a “fix”

17. Winning Scenarios • Criteria for success were met • Step 4: Summarize assessment data collected • Step 5: State that criteria were met and indicate future of intended outcome • no further action required and retire outcome from next assessment cycle • re-assess next cycle using different criteria / measures

18. Winning Scenarios • Criteria were NOT met / results were used • Step 4: Summarize assessment data collected • Step 5: State that criteria were not met and explain how results have been used to make improvements • Changes made to program • Change criteria (criteria too strict?) • Use different assessment method (corroborate) • Step 5: Indicate future of outcome • No further action required (?) • Assess again using different criteria / assessment method? • Sparked new initiative for strategic plan / re-assess at later date?

19. Problematic Scenario Solution • Potential Problem: Haven’t yet used results; can’t “close the loop” by end of assessment cycle • Solution: • Not a problem IF new / strategic initiatives must be taken in order to make improvements – write these into strategic plan and reference assessment report • Otherwise, need careful wording to put “will” into past tense. Hold meetings / make plans prior to submission of report so that decision actions may be stated in the past tense.

20. Problematic “We are planning a retreat to discuss results” …or… “We will…” Preferred “Assessment results revealed insufficient student access to the internet. See new initiative regarding additional workstations in 03-04 strategic plan “ …or… “We have met and have agreed that these are the actions to take… (outline a plan)” Problematic Scenario Solution

21. Problematic Scenario Solution • Potential Problem: Insufficient data • Solution: Explain why (be specific about nature of the problem) and state what is to be done differently next time to obtain data

22. Problematic “No data available” … or … “Sample size too small” Preferred “Survey administration was delayed; no data collected. Same outcome/criteria will be carried over to next cycle when survey is to be administered” … or … “Sample size (n=3) was too small to determine whether criteria for success was met. Outcome carried over to next assessment cycle; will combine three years of survey data to ensure sufficient sample size” Problematic Scenario Solution

23. Use of Results • Responses to unmet outcomes / objectives • Academic Programs • “What is taught” • Closer alignment of coursework with “world of work” • Change in sequence of courses • Additional courses required for degree completion • “How it is taught” • Methodology / technology • Active participation • Administrative Departments • Varies widely - improvements in services or processes

24. Preparing for Next Cycle • There is an expectation that the assessment process will evolve and mature • As you close out current cycle (Oct. 15, 2002) • complete Steps 4 & 5 – no need for re-writes now • use troubleshooting tips if useful • start thinking about Steps 1-3 that will guide your assessment activities in the next cycle • Some areas that could use improvement…

25. Preparing for Next Cycle • Step 1: Expanded Statement of Institutional Purpose • Check your mission statement – it may need to be revised to reflect changing services, new clients, etc. • Step 2: Intended Administrative Objectives • Objectives should have broad focus • Step 3: Criteria for Success & Means of Assessment • State specific criteria (set a target) • Detailed description of assessment method • Include sub-scores

26. Common Step 2 Problems • Administrative objective is too specific • Example:“Increase student satisfaction with service X by 10%” • Problems: • Criterion in Step 3 is made redundant • : “The increase in students that are satisfied with service X will be 10% greater than the last administration” • Can’t really set more than one criterion for success • Instead: “Students will be satisfied with service X”

27. Common Step 3 Problems • Criteria for success are not specific enough • Examples:“turnaround time” or “data accuracy” • Problems: • No targets set to indicate whether criteria are met successfully or not • Little chance of using results to “close the loop” • Instead: “An audit of 100 randomly selected files will reveal no more than 5% with missing data”

28. Common Step 3 Problems • No reference to assessment method(s) • Problems: • No guidance for assessment activities • No documentation for how data were gathered • Instead: “A survey of web users will be administered in the spring by Institutional Research”

29. Suggestion • Consider using subscale scores • Easier to formulate a specific response for use of results using subscale scores than using overall scores only. • Example of subscale use: “Overall, at least 75% of students responding to the Student Satisfaction Survey agree or strongly agree that services of Dept. X are satisfactory and on none of the 4 specific services mentioned will 25% of students or more give ratings of poor or fair.

30. Suggestion • Consider qualitative assessment (focus groups, etc.) • Instead of survey • To clarify survey results

31. Review • This is a learning process • Incorporate existing assessment; don’t duplicate efforts (IR, ABET, CAA, ACBSP, grants) • Establish non-threatening, non-accusatory environment; use results only for improvement • USE results

32. Resources • Campus Assessment Committee • Institutional Research • Assessment support office • Provides logistical means for conducting and processing surveys • Website contains survey data and calendar of projects • URL: http://irweb.erau.edu • Assessment coordinator, Tiffany Phagan – contact via phone 386-226-6224 or via email phagant@erau.edu • Assessment Website • Log onto ERAU Online->Faculty/Staff->Strategic Planning/ Assessment • Forms, training materials • Archived assessment reports

33. Questions and Discussion