Evaluating Educational Programs : Framing and Pinpointing the Guiding Questions

1. Evaluating Educational Programs:Framing and Pinpointing the Guiding Questions Tim Burrows, Chloe Ruff, Lauren Bryant, Virginia Tech February 2012, Conference on Higher Education Pedagogy

2. Agenda for This Session • Definitions • Help you understand the need for focus, guiding questions, and logic models in educational evaluations. • Discussion • Discuss developing guiding questions and a logic model helped to narrow the focus of an evaluation proposal. • Logic Model Jigsaw

3. Focus • Effective evaluations are evaluations with a clear focus. • Even small programs/ projects may have multiple facets that make an overall evaluation difficult. • The Common Book Project at Virginia Tech • Common book is given to all incoming students • Resident Advisors hand out the books to their residents and are encouraged to deliver an activity related to the book. • All faculty and staff interacting with first year students are encouraged to incorporate the book. • First Year Experience courses are required to include the common book in their courses. • Working with stakeholders to develop guiding questions and a logic model can help to narrow the focus of the evaluation.

4. Guiding Questions • Guide the process of an evaluation • Allow evaluators to articulate the alignment between the goals/objectives of the evaluation and the methodology/design of the evaluation. • Specific, actionable, and co-constructed with the stakeholder requesting the evaluation. • For example: • How are FYE course faculty members using the Common Book in their courses?

5. Logic Model • A graphic presentation of a program that links inputs and outputs with expected results. • Inputsare the investments that are made in a program. These resources can include time, material, staff, building space, etc. • Outputs are the activities, participation, and products that are the result of the investment of resources in a program. • Outcomesare the short and long-term results of the program, which include gain in knowledge or changes in behavior. This describes both intended and unintended outcomes, as well as positive and negative outcomes.

6. Basic Logic Model

7. Proposed Common Book Logic Model

8. Logic Model Jigsaw • Break into assigned group by the numbers on your card. • Read the scenario provided to you located in your instructional handout. • Fill in your assigned portion of the logic model guided by the information provided in the scenario. • Use our logic model as an example to help guide you along. • If you get stuck or have any questions, please raise your hand and we will assist you.

9. Questions? • Contact information: • Tim Burrows: tburrows@vt.edu • Chloe Ruff: cruff7@vt.edu • Lauren Bryant: labryant@vt.edu