Learning Outcomes: Critical to Assessment

1. Learning Outcomes: Critical to Assessment Prepared for Office of Dean of Students Assessment Training January, 2011 Gail S. Rooney, Ph.D. Director, The Career Center University of Illinois at Urbana-Champaign

2. Session Purposes • Introduce a learning framework for student affairs • Gain experience writing learning outcomes tailored to your unique offices • Share sample ideas for engaging in the process of defining outcomes • Share sample evaluation project using learning outcomes • Engage in hands-on learning that you can take back to your offices

3. Session Outline • Learning as a framework for student development programs and services • Share a process for writing effective learning outcomes • Defining context, brainstorming outcomes, writing outcomes statements, and considering assessment processes • Considerations of a learning outcomes approach for your office

4. Warm Up Activity • Write a learning outcome for a program or service that you might offer. • Rotate, discuss, and rate outcomes • Scale of 1 (not good) to 8 (excellent) • Add the rating scores for a total score

5. Part 1: A Learning Framework for Student Affairs

6. Learning Framework • To be a part of the core functions of our institution, we need to think of ourselves as learning partners • In an educational environment, learning is the name of the game • Student development programming is education • As student affairs practitioners, we are facilitating the development of lifelong and decision-making skills • Learning can be a paradigm for describing the value of our work

7. What is Learning? * • Learning is a complex, holistic, multicentric activity that occurs throughout the lifespan • Learning is more than information transfer – it is identity development (transformation), giving meaning, reflection and application in the context of the individual’s life * Learning Reconsidered, 2004

8. What are Learning Outcomes? * • Learning outcomes focus on intentional student experiences (activities, courses, programs, services, resources) • What clients should be able to know, do, demonstrate or feel as a result of engaging in a learning experience • Outcomes help us define how clients have changed; they tell us how we are making a difference * Keeling & Associates, 2007

9. Why Learning Outcomes? * • Outcomes define impact – how the client has changed • Focusing on learning moves beyond “bodies in the door” and “smiles on the way out” • The achievement of learning outcomes (individual success) measures institutional or intervention effectiveness * Keeling & Associates, 2007

10. How do Learning Outcomes Shift Our Focus? * Outcome How the client will change as a result of what we do How will clients be different; how will they learn, change, grow? Documenting change; showing results Process Planning activities and things to “do” “What are we going to do?” Counting and asking * Keeling & Associates, 2007

11. TCC’s Office-Wide Student Learning Outcomes • Posed two questions: • What do we want undergraduate students to know and be able to do upon graduation regarding personal and professional career development? • When a student graduates from the University of Illinois, what difference should The Career Center have made in their lives? • Gathered input from: • All office staff – counselors, support staff, graduate assistants, undergraduate outreach presenters, undergraduate office assistants • TCC Advisory Committee members from around campus Questions back in Spring 2005

12. Application of Theories, Models, & Standards – TCC’s Office-Wide Student Learning Outcomes • Brainstormed responses, which you see in your handouts

13. TCC’s Office-Wide Student Learning Outcomes • There are no “right” answers… but you may be wondering: What did TCC come up with? • Employed a counseling model • Affective, Behavioral, and Cognitive outcomes • Followed learning domains from professional standards – • Explore self and options, Manage education, Use resources, Gain experience, Communicate accomplishments, Conduct a search • Recognized career development theory • Self, options, decision-making, life-long process • Affective components of encouragement, hopefulness, etc (See “Samples” Packet, page 2)

14. TCC’s Office-Wide Outcomes: Practice (See “Samples” Packet, pages 3 - 4)

15. Part 2: A Process for Writing Effective Learning Outcomes

16. Learning Outcomes Cycle Defining Context Brainstorming Outcomes Reconsidering Outcomes, Evaluations, Interventions Writing Outcome Statements Evaluating Interventions Connecting Theories / Models / Standards Prioritizing Focus Areas

17. Demonstrating a Process • Steps addressed in learning outcomes activities: • Defining context • Brainstorming outcomes • Writing outcomes statements • Connecting theories, models, & professional standards • Additional steps: • Prioritizing focus areas • Evaluating interventions • Re-considering outcomes, evaluations, and interventions

18. Defining Context • What is your program / service? This could be… • Small – a single workshop or activity • Medium – multi-session interactions • Large – all interactions between students and your office • Who are your intended learners? This could be… • Specific – first-year students in a particular minority group • Broad – all community members who may seek services

19. Defining Context – Example 1 • Resume Review Services • Intervention:Daily resume review services provided on a drop-in basis to clients who are conducting job and internship searches, as well as by students who need to prepare a resume as a part of an academic class. • Intended Learners:Current students and alumni at the University of Illinois • (See handouts for more detail)

20. Defining Context – Example 2 • Exploring Pathways in Career Success • Intervention:EPICS is an interactive, online, modular program that engages students in thinking about their career decision-making early in their college experience. • Intended Learners:Current undergraduate students at the University of Illinois at Urbana-Champaign • (See handouts for more detail)

21. Defining Context – Example 3 • Health and Graduate School Fair • Intervention:Health and Grad School Fair is held each fall in a large ballroom. It is a time of information networking and sharing regarding educational opportunities for college graduates. • Intended Learners:Undergraduate students who may or may not have decided to attend grad school. • (See handouts for more detail)

22. Defining Context – Your Turn! • Review the tips, examples, and prompts on pages 2 and 3 of your worksheet packet • Share with your partner: • the program or service that you will work with today • the intended learners for that program or service • Record your responses on page 4 in your packet

23. Brainstorming Outcomes • What should (intended learners)know, be able to do, be able to demonstrate, value, or feelwhen they have completed (program / service)?Or, stated another way… • What difference should (program / service) have made for (intended learners) who participate?

24. Brainstorming Outcomes – Resume Review Services Example • What should undergraduate studentsknow, be able to do, be able to demonstrate, or feel when they have completed a drop-in resume review? • Know the characteristics of a well formatted resume (Know) • Demonstrate accomplishments in their descriptions of past work experiences (Do / Demonstrate) • Feel confident in their ability to create a resume that highlights their strengths and abilities (Feel) • (See handouts for more detail)

25. Brainstorming Outcomes – EPICS Example • What should undergraduate studentsknow, be able to do, be able to demonstrate, or feel when they have completed online modules in the EPICS program? • Understand how exploring their own personal interests is helpful for narrowing down potential major choices (Know) • Identify skills that they would like to use in their major (Do / Demonstrate) • Feel encouraged – that they can choose a major that will be a good fit for them (Feel) • (See handouts for more detail)

26. Brainstorming Outcomes – Health and Grad Fair Example • What should undergraduate studentsknow, be able to do, be able to demonstrate, or feel when they have attended the Health and Graduate School fair? • Understand the requirements for health and graduate school programs (Know) • Identify and implement next steps to prepare for graduate school (Do / Demonstrate) • Feel confident talking with health and graduate school representatives and exploring options (Feel) • (See handouts for more detail)

27. Brainstorming Outcomes – Your Turn! • Return to working with your partner • Review the tips, examples, and prompts on page 5 of your worksheet packet • Try brainstorming potential learning outcomes for the intervention and intended learners that you selected • Record your responses on page 6 in your packet

28. Writing Outcomes Statements – First Round • (Intended learners) who(intervention) will be able to (intended outcome)? { } Engage in Participate in Complete { } Identify Summarize List Discuss Describe Explain Demonstrate * Keeling & Associates, 2007

29. Writing Outcomes Statements – Resume Review Services Example • Undergraduate studentswho engage indrop-in resume review services will be able to articulate a clear target or purpose for their resume.

30. Writing Outcomes Statements – EPICS Example • Undergraduate studentswho completeEPICS Competency Builder will be able to use resources to strengthen communication skills needed in their potential work environments.

31. Writing Outcomes Statements – Health and Grad Fair Example • Juniors and Seniorswho attendthe Health and Graduate School fairwill be able to identifyat least 3 educational options beyond their undergraduate degree.

32. SMART Approach to Writing Outcomes • S Specific • M Measurable • A Achievable • R Relevant • T Time-based

33. Writing Outcome Statements – Your Turn! • Return to working with your partner • Review the tips, examples, and prompts on pages 7 and 8 of your worksheet packet • Try writing some outcome statements that connect intervention, intended learners, and intended outcomes that you have brainstormed • Record your responses on page 9 in your packet

34. Writing Outcomes Statements – Second Round Bloom’s Taxonomy*Different Levels of Student Understanding & Performance Evaluation Synthesis Higher Cognitive Levels Analysis Application Comprehension Lower Cognitive Levels Knowledge * Bloom, 1956

35. Writing Outcomes Statements with Bloom – Example • Undergraduate students who engage in drop-in resume review services will be able to: • list appropriate content for the education section of a resume (Knowledge) • use strong, varied action verbs to describe their experiences (Application) • integrateaccomplishments / results into descriptions of their experiences (Synthesis)

36. Writing Outcome Statements with Bloom – Your Turn! • Return to working with your partner • Review the tips, examples, and prompts on pages 10 and 11 of your worksheet packet • Try writing some outcome statements that connect activity, intended learners, and intended outcomes that you have brainstormed • Record your responses on pages 12 and 13 in your packet

37. Theories / Models Cognitive Information Processing Krumboltz’s Learning Theory Person-Environment correspondence (e.g., Holland’s typology) Social Cognitive Career Theory Super’s life-span, life-space Tiedeman’s decision making model and many more… Professional Standards National Career Development Guidelines (NCDG; 1996, 2004) National Association of Colleges and Employers (NACE) Principles (2006) Council for the Advancement of Standards in Higher Education (CAS; 2003) American School Counselor Association’s National Standards for Students (ASCA; 2003) and many more… Connecting Theories, Models, & Standards

38. Benefits Helps us understand our assumptions Provides recognized support for program and service decisions Provides insights into: important program and service components target learners potential learning outcomes evaluation strategies Drawbacks Can create “blind spots” May not adequately translate to our unique populations and contexts At times, challenging to figure out how to translate theory into practice Theories & Models – Benefits & Drawbacks

39. Benefits Provides recognized practices for program and service decisions Provides a framework, each through a different lens CAS → system / program level NCDG & ASCA → individuals NACE → partnerships between career practitioners and employers Drawbacks Standards speak broadly Need to tailor outcomes to our unique learners / environments Can be overwhelming Must prioritize to make it manageable Professional Standards – Benefits & Drawbacks

40. Connecting Theories, Models, & Standards– Homework! • Review the tips, examples, and prompts on page 14 of your worksheet packet • Brainstorm the theories, models, and standards that influence your practice. • Record your responses on page 15 in your packet

41. Connecting Theories, Models, & Standards – Homework • A “take-home” exercise! • How do your learning outcomes currently speak to your theory/model/standards? • Would you change or re-word any of your learning outcomes based on your theory/etc.? • What learning outcomes might you add based on your theory/etc.? • How might your preferred theory/etc.help justify or support your choice of learning outcomes? • We’ll still share examples

42. Part 3: A Sample Learning Outcomes Evaluation

43. Learning Outcomes Cycle Defining Context Brainstorming Outcomes Reconsidering Outcomes, Evaluations, Interventions Writing Outcome Statements Evaluating Interventions Connecting Theories / Models / Standards Prioritizing Focus Areas

44. Setting Evaluation Priorities • Learning Reconsidered Framework, with activities to engage staff in thoughtful discussion • Write learning outcomes for specific programs and services • Discuss program evaluation possibilities for the next year (See “Samples” Packet, pages 5 - 9)

45. Setting Evaluation Priorities, Con’t • Selected three programs for “critical incident” evaluations • Exploring Pathways in Career Success (EPICS) • Resume Review Services • Mock Interview Program • Imagine building a “house of evidence” and each program evaluation is a brick

46. Evaluation Example: Resume Review Services • Investigates: • Student learning outcome of effectively communicating accomplishments, as demonstrated through resume writing abilities • Three experimental groups • Participated ONLY in resume review services • Participated in resume review services AND other programs/services • Have NOT used programs/services (control group)

47. Evaluation Example: Resume Review Services • Sample consisted of Juniors and Seniors in the College of Liberal Arts & Sciences who had posted resumes on our online job posting system • Blind review of “self-selected, final product” resumes with pre-determined rubric (See handout) • Compared with ANOVA analysis and Dunnett’s Test of Multiple Comparisons • Total rubric score • Eight individual components of effective resume writing

48. Evaluation Example: Resume Review Sample Findings • Overall resume quality • In comparison to “No TCC” group: • “Resume services and more” group found to have significantly higher quality resumes (α<.05) • “Resume services only” group found to have significantly higher quality resumes (α<.10)

49. Evaluation Example: Resume Review Sample Findings • Resume components • Significant differences found between control and experimental groups on three components: education, additional resume sections, clear purpose/target.

50. Overall highest ratings found on resume sections with “fairly standard” content Name / contact information (3.82/4.00) Spelling / grammar (3.47/4.00) Education (3.22/4.00) Modest overall means on areas that have general tips and guidelines, yet leave ultimate decision-making up to the resume writer. General format (2.90/4.00) Additional resume sections (2.88/4.00) Positive / professional tone (2.87/4.00) Primary experience (2.82/4.00) Evaluation Example: Resume Review Discussion Items