Ross Conner University of California Irvine USA and

1. Blending the Theoretical and the Empirical in Evaluation Science: A Case Example from Cancer Control/Prevention and a General Discussion Ross Conner University of California Irvine USA and Senior Adviser and Past President – International Organisation for Cooperation in Evaluation – IOCE

2. Workshop Outline • Evaluation Overview • The Theoretical Component • The Empirical Component • Case Example: Chinese-Korean Cancer Prevention/Control Program • Conclusions • Questions & Answers; General Discussion

3. 1. Evaluation Overview Inputs  Processes  Outputs  Outcomes  Impacts • A simple example: program = a mathematics class • Inputs: students, teacher, lesson texts, paper and pencils, classroom setting • Processes: lessons and exercises taught by teacher • Outputs: exercises completed by students • Outcomes (short-term): Students learn new concepts, have new understandings (measures: tests) • Impacts (long-term): Students apply new concepts and understandings, students advance to next level of learning • For simplicity, this is set out as a linear track of 5 stages; in practice, it is often more complex with multiple tracks because of different components of the program.

4. 2. The Theoretical Component • Theory from Evaluation Science • Evaluation Science: bringing the tools of science and discovery to the assessment and evaluation of programs or policies. • Tracking and monitoring inputs and processes; identifying and documenting outputs; measuring outcomes; tracking and measuring impacts. • Theory from the Program or Policy Area • Program or Policy Area: bringing the past learning from the area to advance the area • Building upon past experimental and theoretical work in the area, developing program components based on past research and practice. • Using both types of theories, the program components and the evaluation components are developed.

5. 3. The Empirical Component Program Design Specify planned inputs and planned processes Specify expected outputs, outcomes and impacts Specify evaluation questions Evaluation Design Specify evaluation design(s) to answer the evaluation questions. Design = the plan for data collection Specify evaluation measures to obtain necessary data

6. 4. Case Example • Area: Health – cancer control and prevention • Populations: Chinese and Koreans in California, USA • Program focus: women’s cancers initially, later men’s cancers • Program aim: disease destigmatization

7. Overview:Chinese Program Components Materials in Chinese language Large free luncheons with expert lecturers Free screenings for cancers Cancer survivors involved in the programs High visibility in community

8. Overview: Korean Program Components Materials in Korean language Used networks of Korean Christian ministers to destigmatize; cancer survivors featured Free screenings for cancers Prominent role at public events and fairs

9. The Program Inputs  Processes  Outputs  Outcomes  Impacts Planned Inputs: materials in native language, large gatherings of community members, religious leaders (Korean), cancer experts (Chinese) Planned processes: dissemination of materials and information to 4,000 people, cancer screenings, discussions among community members Planned outputs: increased knowledge about cancer, increased understanding of consequences of disease stigmatization Expected outcomes: increased (by 10%) cancer screenings, decreased disease stigmatization Expected Impacts: 100 community cancer survivors serving as volunteers and outreach workers for the program

10. The Evaluation Inputs  Processes  Outputs  Outcomes  Impacts Examples of evaluation components: Inputs: tracked numbers of attendees at events Processes: observed community events – materials distributed, lectures given, discussions held Outputs: assessed knowledge about cancer Outcomes: tracked cancer screenings, tracked decreased disease stigmatization Impacts: tracked cancer survivors serving as volunteers for the program

11. Selected Evaluation Results * Estimated duplicate counts: 40% ** Ministers + Includes breast, uterus, liver, prostate and colon cancer screenings

12. 5. Conclusions • The program and the evaluation blended theoretical components from evaluation science and from the area of health education/disease prevention. • The program set measurable objectives. These anchors provided a basis to judge achievements. • The evaluation involved empirical assessments of inputs, processes, outputs, outcomes and impacts. • The program planners/implementers from the Chinese and Korean communities were partners with the evaluation team, from start to finish. • The programs continue today, with a smaller funding base

13. 6. Audience Inputs and Comments Questions about the program Questions about the evaluation Questions about the results General questions about evaluation Any other general questions for discussion

14. Evaluation: Next Steps New book available soon: Оценка программ: методология и практика Редакторы: А.Кузьмин, Р.О’Салливан, Н.Кошелева