Aaron J. Scott Research Associate October 13 th , 2009

1. Building for assessment practice: Establishing a culture of evidenceStage 1: Planning the process Aaron J. Scott Research Associate October 13th, 2009

2. Objectives of this session • To briefly revisit the culture of evidence definition and model and its associated stages. • To reconsider Stage 1 and expand our notion of planning. • To discuss specific items to consider during the planning process – i.e. going beyond the objective statements. • To provide examples from our own department that are in or have recently gone through this stage.

3. What is a culture of evidence? • A culture of evidence is defined by Lakos & Phipps (2004) as: “An organizational environment in which decisions are based on facts, research, and analysis, and where services are planned and delivered in ways that maximize positive outcomes and impacts for customers and stakeholders” (p. 352).

4. Why a culture of evidence? • Transparent Decision-Making • Charting progress toward goals • Documenting implementation and improvement • Data-Informed decisions through assessment • Organizational/Departmental Effectiveness • Integrating goals and working together • Reporting outcomes in a consistent format • Accountability • To the Department • To the University • To External Stakeholders • The State • Parents • Funding Sources

5. Modeling a culture of evidence Stage 1 Stage2 or Processes Stage4 Stage3

6. Stage 1: Planning 3 Steps in Affirming and Building Upon Our Purpose(s) • Through development/review of Mission • Through development of Goals • Through development of Objectives

7. Purpose Expressed in Objectives • Action statements • Clear, specific and concrete • Measurable • Time limited • Realistic • Hierarchical (ultimate, intermediate, immediate) • Build on strengths and reduce need • Focus on outcome or process Now What?

8. Toward Good Assessment • Securing the Planning & Budgeting at the beginning sets the foundation for a clear course of implementation and assessment. • This first stage has only begun, however. You must determine how you will go about implementing the program or services, defining and measuring their degree of success, who will be involved, the resources it will take to work toward accomplishing your objectives, and how long the process will take, to name just a few. And this often requires much more work and time!

9. Planning “Reconsidered”

10. Revisiting the definition “An organizational environment in which (1) decisions are based on facts, research, and analysis, and (2) where services are planned and delivered in ways that maximize positive outcomes and impacts for customers and stakeholders”

11. Applying the defintion • The quality of a program or service, and decisions made on its behalf, is dependent, in part, upon the quality and utility of the data informing it. • But the quality of the data, and therefore evidence, is proportional to the degree of planning and rigor invested in developing the clarity of purpose, the design, and the implementation of a program or service, and its associated assessment.

12. Housing Example • Example: Living on campus during a student’s first year is known from a plethora of previous research to have a positive impact on his/her retention and overall academic success. But less certainty exists about how this takes place(Braxton, 2007). • To what degree is this true on our campus and how do we know? Is it true for all sub-groups? Are there ways we can maximize our impact? What data are there to support what we know? What data do we still need?

13. Stage I Flow Chart

14. Evidence & Decisions • Carefully choose your evidence and it’s appropriateness (and timeliness) for the program or service. • Choose how to collect your evidence • (interviews, focus groups, brief questionnaires, in-depth surveys, paper form, web-based, email-based). • Choose how to frame your evidence by providing a context for the results. • Determine to whom will you be providing this evidence.

15. Important Questions to Ask • What is my potential available budget? • What is my timeline? How realistic is it? • What are my analysis capabilities? • Who needs to see these data? How long will it take to get the data in presentable format? • How easily can I fit this method into my annual responsibilities? • Who needs to make decisions with these data? • Will this kind of evidence help me make the decisions I need to make? How? • How will I document the evidence and the decisions made from that evidence?

16. Benchmarks of Stage I Completion • Purpose of program/service and assessment is stated clearly, concisely, and completely in terms of outcomes or processes. Often times this is stated in terms of a problem or need. • Members involved have been consulted early on and have agreed to participate. • How purpose will be carried out (i.e. implementation) is spelled out clearly. • Expectations are clearly stated. • A timeline has been through at least two drafts and has been peer reviewed. • A method for measuring outcomes has been secured (pre-fab or home-made) and either has well-established reliability and validity or has been planned to be piloted. • IRB process has been completed, if required. • Anticipated costs have been outlined.

17. Criteria for Outcomes • Is it measurable? • Is it meaningful? • Is it manageable? • Who is the target audience of my outcome? • How will I know if it has been met? • Will it provide me with evidence that will lead me to make a decision for continuous improvement?

18. Final Considerations • A culture of evidence is a great motto, but what kind of evidence? How good is the evidence and how do you know? Only through rigorous planning and implementation can we begin to answer these and other important assessment questions.