Educating Ourselves through Program Assessment – Continuously Improving our Service to Students

1. Educating Ourselves through Program Assessment – Continuously Improving our Service to Students Dr. Ann Groves Lloyd Associate Dean Student Academic Affairs College of Letters & Science University of Wisconsin-Madison

2. Snapshot of Student Academic Affairs

3. Letters & Science Student Academic Affairs Demographics • Approximately 18 units • Size varies from 15 staff members to one • Some units have assigned populations • Some units have a campus-wide mission • College encompasses nearly 17,000 students • Familiarity and use of tracking and evaluations tools varied widely

4. SAA – Vision for Assessment • Each unit would assess program quality, productivity, need, and demand • Continuously improve quality of programs offered by capturing voices of students and other constituents • Enhance program planning and budgeting • Establish baseline data to explore longitudinal trends and patterns • Satisfy institutional/state/national reporting and review requirements

5. Charge to Assessment Committee • Develop instruments to measure student perceptions and overall quality of student experience • Develop standardized expectations for unit directors/coordinators • Provide staff with a practical, helpful tool • Begin systematizing data collection to acquire retention and longitudinal data • Pilot project in place for 2002-03 academic year

6. L&S SAA Assessment Committee • Made up of a cross-section of staff in SAA • Determined that a handbook would be the best route to go • Developed timeline and content areas • Split into teams to work on each section

7. Factors for Success • Involved those who will implement and use the Handbook • Process involved on-going negotiation and compromise • Wanted to take the assessment picture down to very small and practical increments • Took existing systems into account • Developed flexible, adaptable plan • Encourage risk-taking

8. Foundation • Differentiates between tracking for annual reporting and collecting data for program improvements (Service Quality versus Student Learning Outcomes) • Does not ask unit directors/coordinators to get into determining retention rates • Next steps will be defining and measuring learning outcomes

9. Handbook Introduction • Two-hour session with unit coordinators/directors in May 2002 • Associate Dean discussed her vision for assessment for our division of student academic affairs • Assessment Team walked everyone through the contents of the Handbook

10. Handbook Contents • Associate Dean’s vision for Assessment in Student Academic Affairs • Ethics in assessment • Expectations for unit directors regarding collecting usage statistics and program improvement data, developing annual reports, and conducting self-studies • Primer on the basic tools of assessment • A resource section

11. Ethics in Evaluation • Ensure ethical treatment of research participants • Avoid harm • Informed consent • Freedom from coercion • Ensure ethical collection of data • Ensure ethical interpretation and presentation of data

12. Primer on Statistical Concepts • Includes brief discussion and cautions regarding interpretation of data • Also addresses managing data, including issues of confidentiality and security • Has changed significantly since evolution of on-line survey tools such as Zoomerang.com and Survey Monkey

13. On-Going Assessments • Capturing student names/IDs in ISIS and/or stand-alone databases (discouraged) • Collect demographic information about program participants • Capture information about advising appointments • Establish systems to regularly assess student satisfaction with services • Obtain feedback for program improvement • Systematically gather data to assess retention and persistence (done at the divisional and college level)

14. Ways to Collect On-Going Data • Phone calls • Emails • Advising contacts • Resource use • Workshop attendance • Events in general • Website hits • Number of students turned away from services

15. Annual Reports • Each SAA unit will submit an assessment plan in September each year. • Each unit will submit an annual report due in January each year. • Responsibility of unit director/coordinator • Snapshot of previous year’s accomplishments and a look toward future initiatives and challenges

16. Contents of Annual Report • Table of Contents • Executive Summary • Mission Statement and Core Values • Strategic Plan if applicable • Overview of past year • Future goals and initiatives

17. Self Studies • Ten-year cycle for self studies • Dean’s office has determined expectations for self studies • Additional details are outlined in the handbook

18. Assessment 101 • Selecting Assessment Tools • Statistical Concepts • Interpretation • Managing Data • Surveys • Interviews

19. Selecting Assessment Tools • Quantitative • Quantitative • Advantages and disadvantages of each • Encourages multiple measures and methods

20. Surveys • Gathering demographic information • Types of questions • Ways to gather survey data • Tips • Survey Kit by Arlene Fink (Sage Publications)

21. Interviews • Role of human subjects committee • FAQ prior to interviews • Focus groups versus in-depth individual interviews • Analyzing interviews

22. Resource Section • List of campus resources • List of web-based resources • Books on assessment • Article by Upcraft & Schuh from About Campus (1998) • AAHE – 9 Principles of Good Practice for Assessing Student Learning • Handbook can be found at: http://wiscinfo.doit.wisc.edu/obpa/Assessment/SAA_Assessment.htm

23. Where are we now? • Got the tracking and evaluation systems established in all units • Have had two sets of assessment plans and annual reports turned in • Obtained feedback from unit directors/coordinators and began updating handbook • SAA has spend the past nine months going through its own assessment and evaluation process

24. Final Thoughts • Assessment can be intimidating, but knowing where you are is essential to determining where you want to go! • Just pick someplace to begin – don’t get overwhelmed by the big picture • Develop a culture that embraces continuous improvement and inclusion of student voices in the planning

25. Stretch and Take a Break! Discuss with your colleagues your biggest challenge regarding assessment

26. Self-Evaluation of Assessment Practices • Tool for evaluating “where you’re at” regarding assessment in your unit/department • Identify strengths upon which to build • Identify aspects to improve or enhance • Catalyst for important conversation and action