Assessment on the ground

1. What you need to know when conducting assessment or working with an assessment professional so that the results help you do your best work! Assessment on the ground

2. SARA • Mission Student Affairs Research and Assessment (SARA) provides leadership and service to the Division of Student Affairs in the areas of assessment, learning outcomes, educational programming, and strategic planning. SARA collects and disseminates data about Penn State students, their experiences and learning, and their perceptions of the campus environment. SARA enhances students’ educational experiences through the design, implementation, and evaluation of programs that support the University’s Cocurricular Learning Outcomes. • http://studentaffairs.psu.edu/hr/pdf/intorgchart.pdf

3. Why is assessment important? • It will help you do a better job • It will help ensure that you have the resources to do that job • It can help you prioritize your efforts • It can contribute to the broader conversation and understanding

4. What do you need to know? • Not everyone has to be an assessment expert • But you do need to know how to make informed decisions • To conduct an assessment • To use the findings effectively • Plan, Do, Check, Act (PDCA)

5. You are the content expert • Things to consider: • The parameters of the activity you are assessing • The conceptual model that underlies the activity • The desired objectives/outcomes (learning or other) • Past assessment efforts and findings • How this activity/assessment relates to the strategic goals of your unit and student affairs • What resources are available • How you will use the results • Where/how will you disseminate the results

6. Basic Approaches (Schuh & Upcraft, 2001)

7. Key steps - PDCA • Identify objectives • Identify appropriate measures • Choose an appropriate assessment method • Choose an appropriate project design • Collect the data • Analyze the data • Disseminate the findings • Take action • Start all over again!

8. Identify & articulate objectives • You should always be able to tie what you are doing (programming, advising, etc.) to a specificobjective • Bloom’s taxonomy or similar learning constructs may be useful ifyou’re seeking to measure learning • The focus of a learning objective is not on what you will teach or provide, but on what students will learn or gain

9. One helpful approach to writing learning outcomes is ABC(D): • Audience: Who will accomplish the objective (e.g., students) • Behavior: What is expected; most helpful when overt and observable • Condition: Under what circumstances the behavior is expected • (Degree: The acceptable performance level - often unstated)

10. Avoid the “weasel words” • Learn • Know • Be aware of • Be familiar with • Have a firm grasp of • Understand • Appreciate

11. Example using Bloom • Remember – After attending the Women's’ Resource Center open house, students will be able to describe the resources available at the Center. • Understand – After completing their first-year seminar, participantswill be able to interpret university policies related to academic integrity. • Apply – By the completion of the Study Skills series, participantswill establish a realistic time management plan. • Analyze – By the end of their first elected term, student government participants can find and provide credible evidence to support a planned initiative. • Evaluate – After completing Leadership 101, class members will be able to debate the merits of various leadership styles and their application in different situations. • Create – By the end of their first elected term, student leaders will be able to develop and present detailed plans for proposed projects. AudienceBehavior Condition

12. Determine appropriate measures • Attitudes? • Experiences? • Knowledge? • Abilities? • Persistence? • Graduation? • Employment?

13. More on measures • Norm-referenced = report student performance relative to other students • Criterion-referenced = compares to an absolute standard of achievement (e.g., you pass or you fail) • Self-referenced = compares different scores/ratings from the same student

14. What kind of data? • Quantitative • Can be economical, generalizable, reliable, valid • Qualitative • Can provide rich data – can ask about relationships rather than infer • Mixed methods

15. Consider various types of data collection approaches • Test of abilities or knowledge • Survey • Interview • Focus group • Other types

16. Tests • Can be used to measure a specific skill or knowledge base • “Objective” • Feasible? • Surveys • Attitudes • Experiences • Self-reported abilities & knowledge • Can be confidential/anonymous • Can reach large numbers of people

17. Interviews • Rich & detailed information • Body language can be helpful • Respondents may feel inhibited • Repeated interviews can build trust/comfort • Time-consuming • Focus groups • Many of the same benefits as interviews, but not as in-depth • Participants “feed off of” each other • Can reach more people than interviews

18. Other ways to collect data • Student self-assessments • Portfolios • Reflective journals • Observation-based What others can you think of?

19. Project design • Go back to your assessment question(s) • What do you want to know? • What are the limitations (e.g., time, resources, staff)? • How will the assessment be used? • The answer to these questions help you figure out: • What your purpose/question is • What approach to use • What type of information to collect • How to analyze your data

20. Data collection

21. Data analysis • Make sure that your analysis is appropriate to your question. • Does your data meet the basic assumptions of the statistical test you are using? • Is your analyst free coding your transcripts when your goal was to analyze your data using a pre-assigned coding system based on the theoretical framework underlying your program? • Make sure that you understand the assumptions and limitations of any analyses you rely on or report, whether you are the one who conducted them or not

22. Reporting & Dissemination • Think about your audience • Administrators don’t have time to read a dissertation or even a journal-length article • Executive summary of the executive summary! • Put key points into easily digested tables, charts, graphs, and bullets • Don’t get carried away!

23. Dissemination continued • DO NOT FORGET to spell out the implications of your work • How do the findings improve our understanding? • What should we change? • What else do we need to know? • Make sure the information gets to key decision makers – understanding the structure and politics of your institution is very important (Terenzini’s “contextual intelligence”)

24. You’re not done yet!Time for action. • What are you going to do with what you learned? • Plan, Do, Check, Act, Repeat

25. Theory to practice • In practice, you will not always have the resources to do assessment the way it is outlined in books and papers • What’s important? • Understand the impact of the compromises you will make • When you act, you are doing so based on the best available information

26. Questions?

27. Your turn • Get out a piece of paper • Choose a student affairs area to represent • Write at least one learning outcome for your unit • How would you assess it?

28. Think about • Type of approach (needs assessment, learning outcomes, satisfaction, etc.) • Type of measure(s) (attitude, knowledge, etc.) • Type of data (quant, qual, mixed) • How will you collect it? (focus group, survey, etc.) • What is your design? (pre-/post-test, longitudinal, etc.) • How will you analyze it? (thematic analysis, mean comparisons, etc.)

29. Betty Harper BHarper@psu.edu