Moving from Concept to Meaningful Assessment :

1. Moving from Concept to Meaningful Assessment: Using Logic Models to Maximize Efforts & Clarify Results Ashley Finley, Ph.D Senior Director of Assessment & Research, AAC&U National Evaluator, Bringing Theory to Practice Bringing Theory to Practice Well-Being Initiative Assessment Webinar March 5, 2014

2. The problem with program assessment… Assessment Curriculum Outcomes Co –Curriculum

3. Assessment Challenges • What are the long-term impact goals (the vision for change) vs. more immediate outcomes? • How is what we propose to do actually connected to the changes (outcomes) we propose to get? • What types of evidence are needed to say something meaningful about outcomes? • How do we think holistically beyond grant resources to other resources on campus that can help facilitate change toward outcomes?

4. Defining Impact from Outcomes… Would LIKE to see Would LOVE to see EXPECT to see Programmatic control over outcomes decreases as move toward love…because you need people beyond the program

5. Resources needed to start or keep going: Actions needed to produce outputs: OUTPUTS (What counts as good evidence?) ACTIVITIES (What will students be asked to do?) OUTCOMES (What should improve as a direct result of efforts that contribute to the long-term vision?) INPUTS (What is needed for the process?) Impact Goals (What is the hope for the future for students, for faculty, for the institution?) Products needed to assess outcomes, “countables”: Expected Changes: short, inter-mediate Long-term vision for change

6. Resources needed to start or keep going: Actions needed to produce outputs: OUTCOMES ACTIVITIES (What will students be asked to do?) OUTPUTS Impact INPUTS (What is needed for the process?) • Flourishing • Self-esteem • Resilience • Persistence • Self-authorship • Satisfaction with life • Hope • [Resource: University of Pennsylvania Positive Psychology Center] • Individual/ group Reflection • Summary of group dialogue • Critical application of well-being to course material • Multimedia • Build culture of well-being at institution • Increase institutional commitment to whole student development • Increase commitment to “flourishing” across campus community • General education • Teaching and learning center • Counseling center • Student affairs • Student orientation • Career center • Alumni center • Group dialogue • Small group problem-solving orapplications • Engagement with contemporary issues, real-life questions Products needed to assess outcomes: Expected Changes: short, intermediate:

7. Logic Models: Basic Principles • Should be developed collaboratively • Intended to simplify plan for development and function as communication tools • Attentive to maturation of projects over time • Start with outcomes, not inputs • Assumptions underlie all dimensions

8. Case Study: A BTtoP Project Campus • The [community engagement program participants] are a trained group of students who work with professors and students on cultural sensitivity and inclusive language. Their work will help the college community become more aware of the politics of power, the root historical causes for social issues, and best practices for determining community needs in an unbiased manner. • This awareness and attention to understanding the complex web of attitudes and linguistic choices that make people culturally sensitive, will begin to make us conscious of alternative choices in our engagement with the full diversity of our community. • Our basic premise is that phobias and “isms”, are socially conditioned and constructed and can be unlearned and deconstructed…[R]ecognizingand challenging inappropriate discriminatory language is key. The work of the [program participants] and supporting faculty members will create a more whole campus community that is better prepared for wider engagement.

9. Moving from Goals to Expected Outcomes Would LOVE to see EXPECT to see Would LIKE to see • College community becomes more aware of politics of power, root historical causes for social issues, best practices for determining community needs • Consciousness of alternative choices in engagement w/ full diversity of community • Create more whole campus community better prepared for wider engagement Expected changes through community engagement program participation that lead to… Programmatic control: Who needs to be involved beyond & in coordination with the community engagement program?

10. Outcomes Impact Goals OUTPUTS INPUTS ACTIVITIES • Community engagement program • Faculty advisory board for program • Center for public service • Common book program • Comm. Engage. Diversity • Inclusion Excellence Program • Unlearn hurtful language &unconscious oppression • Develop sensitivity to &awareness of unique history & culture of local comm. • Gain particular knowledge & awareness of major “isms” and phobias: sexism, homophobia, religious bias, racism, xenophobia, classism, able-ism, and ageism. • Proposed Evaluation plan: • Program Evaluations • Flourishing scale & civic attitudes scale to internship seminar students led by participants+ focus groups with seminar students (compare w/ control grps of students not working w/ participants) • NSSE & CIRP (to look at community engagement & well-being) • 3 presentations or panels and related discussion • Program training, comm. tour, grpdisc. • Film Screening & Panel • Collaborative projects • Program training sessions and workshops • (None specified) • [What are student participants asked to do as result of training that encourages them to engage or synthesize learning to meet intended outcomes? What are students asked to do as a result of attending panels or presentations that encourages them to engage or synthesize learning to meet intended outcomes?]

11. Mapping Outcomes Beyond the Student Level INPUTS ACTIVITIES OUTPUTS OUTCOMES • Student • Program components • SA programs • First-yr seminars • Student • connection of well-being to coursework • Group work • Activities in co-curr • Student • Reflection papers • Summary pts from group discussion • Collaborative project • Student • Increased flourishing • Increased self-esteem • Resilience • Faculty • No. of faculty allies • Faculty dev. $ • Faculty • Faculty orientation • Faculty develop. Workshops • Faculty • # of courses w/ best practices • # of comm. contact hrs/integration with course material • Faculty dev. hrs • Faculty • Use of engaged pedagogies • Awareness of Well-being • Bldg. Communities of Practice • Institutional • Current assess. • Inst. Research • Student Affairs • Counseling center • Teaching Center • Institutional • Assessment workshops • Marketing discussions • Institutional • increase in assign/courses focused on well-being • Posters/banners on well-being • recognition event • Institutional • Evidence of connection of well-being & learning & CE • Retention • Campus awareness

12. Case Study: Dartmouth Mentoring with Purpose IMPACT • Mentoring With Purpose =collaborative project to foster a world view of looking out for others; supporting the recently-launched Dartmouth By-Stander Initiative (against sexual assault) by engaging students as allies; and assessing the effects of cascade mentoring on the campus, community, and participants. • A coordinated mentoring reflection program will allow students—and faculty and staff members--to work in a "cascade fashion" with people above them, beside them, and below them. • Through an annual mentoring summit, dialogue groups, classroom discussions, and their own mentoring of others, students--with faculty, staff, community leaders--will learn more about themselves; about those whom they mentor; &about the possibilities &need for social change. They will address significant life questions while mentoring, helping them to flourish and to nurture others in flourishing as well. • Aim of the program is to involve 200 students in first mentoring summit &100 students in early discussion groups with faculty &staff. Then expand to include larger numbers of students &mentees and in surrounding communities, with eventual aim of constituting a national civic engagement model. Outcomes Activities Resources IMPACT

13. Outcomes Impact Goals OUTPUTS INPUTS ACTIVITIES • New pres. w/ strong commitment to experiential educ. • Critical # of committed fac., staff & students • Coord. efforts of Tucker Foundation, Council on Service & Eng. & Dean of College • Existing mentoring programs • Relevant academic courses (CBL) • Far-reaching awareness of mindfulness • Survey data on mentoring programs’ success vs. past success • Pre/post survey on participants’ understanding of personal values & goals • Pre/post survey on ability to connect learning outside of class • # of students, fac., staff, CLs as mentors • # of cross-campus & comm. Partner collaborations • # of younger people/peers mentored • Media coverage of mentoring activities • For all participants: • articulate value of mentoring & rel. bet. mentoring theory & practice • Students will: • Dem. under. of CE & social change through mentoring • Increased awareness of potential for personal growth • Dem. know. of mentoring skills • Apply curr. Learning to exp, outside classroom • More committed to create just society • Faculty/Staff/CL • Dem. Increased know. of student lives outside classroom • Involved students better understand who they are & who they might become • Dartmouth is known as a “mentoring” college • More connection bet. academic theory & learning outside classroom • Stronger community on campus across faculty, staff & comm. Members • Increase # students committed to helping create a just society • Plan overall program w/ stakeholders • Glean learning from guest lecturers • Plan summit • Create recruitment training, implementation, debriefing, assess. plan for dialogue grps • Train presenters & facilitators • Annual mentoring summit • Annual mentoring dialogue grps. • Assessment • Write summary report

14. Summary Considerations for Working with Logic Models • Have you separated long-term impact goals from intermediate and short-term outcomes • What are the activities of the program? What is happening to effect change in students? What about faculty? At the institutional level? • What are students doing as a result of their participation that can be counted as evidence? What other sources of evidence will be used to assess outcomes? Ditto at faculty and institutional levels. • What resources exist that contribute to project goals and outcomes?

15. Additional Resources • Logic Models: http://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide • Bringing Theory to Practice Toolkit Instrument & Overview document (includes the flourishing scale): http://www.aacu.org/bringing_theory/assessmenttools.cfm#Tlkt • University of Pennsylvania Positive Psychology Center: http://www.positivepsychology.org/ppquestionnaires.htm • AAC&U VALUE Rubrics (direct assessment of learning outcomes, including civic engagement & intercultural competence): http://www.aacu.org/value/index.cfm • Wabash Center for Inquiry in the Liberal Arts: http://www.liberalarts.wabash.edu/study-instruments/