1 / 23

Evaluating Workplace Health and Wellbeing Interventions

Learn how to evaluate workplace health interventions effectively with theoretical models, design methods, and outcome evaluation criteria to measure impact on employees' well-being and productivity.

rgilliam
Download Presentation

Evaluating Workplace Health and Wellbeing Interventions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde RTI International

  2. Steps for Conducting Effective Evaluations Centers for Disease Control 2

  3. Types of Evaluations Adapted from: Norland, E. (2004, Sept). From education theory.. to conservation practice Presented at the Annual Meeting of the International Association for Fish & Wildlife Agencies, Atlantic City, New Jersey.

  4. Theoretical/Logic Model • A diagram of the theory of how a program is supposed to work—a graphic depiction of relationships between activities and results (conceptual/theoretical model) • A logical chain of connections showing what a program intends to accomplish • Increases intentionality and purpose • Guides prioritization and allocation of resources • Helps to identify important variables to measure; use evaluation resources wisely • Supports replication

  5. WFHN Evaluation Design • Single protocol • Work redesign, supervisor training and self-monitoring • Multiple industries and worksites • Healthcare (blue collar) • Telecommunications (white collar) • Group randomized field experiment • Intervention and comparison group assignment at the worksite or work group level • Adaptive randomization to balance covariates

  6. Outcome Evaluation Design • Nested cohort design • Worksite partners are randomized within industry to intervention or control group • Outcomes evaluated at multiple levels, employees, workgroups and worksites • Allows for multiple levels of clustering (e.g., workgroups within worksites), and a variety of outcomes (e.g., discrete, continuous, count)

  7. Formative(Process) Evaluation Framework Implementation Outcomes Acceptability Adoption Appropriateness Cost Feasibility Fidelity Penetration Sustainability Service Outcomes* Efficiency Safety Effectiveness Equity Patient- centeredness Timeliness Individual Outcomes Satisfaction Function Health status/ symptoms How? What? Implementation QIs - Strategies ESTs *IOM Standards of Care Implementation Research Methods Proctor’s Model of Implementation Outcomes

  8. Process Evaluation Data • Document Reviews – provide information to build an “a priori” understanding of the program content, operations, context, and program stakeholders • Review reports, instruments, protocols, promotional materials, patient materials, resource lists, organizational documents (org charts, flow charts, operation manuals), web sites. • Observational Studies – provide empirical evidence to assess program fidelity, and generate service flow and timing data to inform outcome and cost analysis • Key Informant Interviews – provide contextual information related to program utility, contextual factors that influence program implementation, service delivery, dispersion, and sustainability • Practioner and Consumer Surveys -- provide systematic data related to service delivery experiences and program related perceptions

  9. Qualitative Analysis Methods • Recursive abstraction (Document Summaries) • Iterative process that generates summaries, classifications, lists, rates, or groupings • Deductive and Inductive Analysis (data coding) • Deductive “a priori” framework • Inductive “grounded theory” analysis • Results in taxonomies, themes, categories, orders • Comparative analysis • Documented vs observed processes, behaviors or outputs • Document changes over time (e.g., model migration) • Variation between processes or outputs • Mixed Methods (triangulation and convergence)

  10. Process Evaluation Results • Comprehensive description of program components • Performance indicators and proficiency scores for program delivery • Common barriers and facilitators across worksites • Descriptive taxonomy of program settings • Construction of moderator variables for use in outcome and economic analyses • Program delivery protocols that are used at each worksite site

  11. Outcome Evaluation Questions • Program Outputs • To what extent is the program actually performed as measured by (e.g. the number of health risk appraisals completed, percentage of employees participating in workshops, number of follow-up services delivered) • Proximal and Distal Outcomes • What is the effectiveness of the program on outcomes of interest? (changes in sleep quality, changes in bio-measures)? • What stakeholder and employee-level characteristics moderate the willingness and ability of performance sites and practitioners to adopt the program

  12. WFHN Study Data

  13. Outcome Evaluation Methods • Descriptive statistics generating classifications or groupings of the participating worksites, provider characteristics, employee characteristics • Multivariate regression examining the relationship between groups • Statistical modeling measuring changes on outcomes over time

  14. Outcome Evaluation Results • Rates and frequencies of program participation • Performance site descriptive characteristics (including staff characteristics) that are associated with outcomes of interest • Within group pre to post changes in health and wellness outcomes • Between group comparative changes in health and wellness outcomes at multiple time points

  15. Intervention Effect on Work Environment FSWE Hypotheses • H1. There are changes in work environment for the • H1a. TOMO intervention group • H1b. TOMO control group • H1c. LEEF intervention group • H1d. LEEF control group • H2. There are baseline differences in environment between intervention and control groups for the • H2a. TOMO industry • H2b. LEEF industry • H3. There are differences in the rate of change in climate between intervention and control groups for the • H3a. TOMO industry • H3b. LEEF industry

  16. FSWE Factor Analysis

  17. Growth Model Results by Industry

  18. Economic Evaluation Questions • What is the program cost to worksites and to other stakeholders? • What is the cost-effectiveness of the program? • What characteristics moderate the cost-effectiveness of the program?

  19. Translational Study • An effective intervention is only useful if it is communicated to and adopted by workplaces • Complementary methods: • Use participant feedback to inform post-study messaging • Analyze process data to study employee perception for developing portrayals of the intervention • Assess potential dissemination channels • Stimulate market demand for effective intervention

  20. Thank You For more information on the Work, Family, and Health Network Study http://projects.iq.harvard.edu/wfhn/people

More Related