The California Tobacco Program What has this to do with medical care improvement?

1. The California Tobacco ProgramWhat has this to do with medical care improvement? Program characteristics • Attempted to change “system” and culture of tobacco acquisition and use • Multi-component • Multi-level – policy, institutional change, services to individuals • Local adaptation expected and encouraged • Evaluation depended on ongoing surveillance data supplemented by measures targeted at specific intervention components

2. The California Tobacco ProgramWhat has this to do with medical care improvement? Same general evaluation questions • Does the QI intervention work? • If so, what is the contribution of the individual components? • If not, were there promising components whose effect was too limited to impact overall outcome?

3. Testing a standardized, simple unchanging treatment over a relatively short time span is irrelevant for many medical care and public health interventions. • Evidence and theory often suggest that comprehensive system change is indicated Randomization is not the issue; it is often possible and desirable • Standardization of an intervention across sites and over time precludes learning and improvement, and fails to account for site differences

4. Comprehensive system change is often necessary • “Systems are perfectly designed to get the results they achieve.” • Service systems are by definition complex and adaptive – not simple like a pill or machine • Improving outcomes often means changing multiple interacting components of systems – making tobacco less attractive to buy, more difficult/costly to use, and easier to quit

5. Interventions are multi-level • The most potent public health or QI interventions like the CTCP often act at multiple levels – at the level of individual people, at the organizational level, and at the policy or environmental level • Sequential testing and factorial designs may not be feasible or possible • But, understanding the contributions of changes at all levels is important

6. Multi-level, multi-component system change is often optimal, but is it evaluable? Small theory of treatment (Lipsey) • Posit how the various elements of the intervention impact which intermediate or outcome variables in which subpopulations at which time points • E.g., leadership influences QI team effectiveness which influences depth of system change which impacts outcome • E.g., QI facilitation increases involvement of non-physician staff in prevention; EMR with registry function leads to proactive visits to meet prevention guidelines; nurse-involved preventive visits improve guideline adherence rate.

7. Lessons • Maximize design within constraints • Maximize learning from variation within the intervention • Validate then use existing surveillance measures • Use multiple measures of same phenemenon