Advanced Methods in Delivery System Research Webinar: Fuzzy Set Analysis

1. Advanced Methods in Delivery System Research –Planning, Executing, Analyzing, and Reporting Research on Delivery System Improvement Webinar #1: Fuzzy Set Analysis Presenter: Marcus Thygeson, MD Discussant: Jodi Holtrop, PhD, Moderator: Michael I. Harrison, PhD Sponsored by AHRQ’s Delivery System Initiative in partnership with the AHRQ PCMH program April 15, 2013

2. Speaker Introductions N. Marcus Thygeson, MD is Senior Vice President and Chief Health Officer, Blue Shield of California, where he leads the healthcare services team. Dr. Thygeson’s presentation today will draw on his paper with Deborah Peikes and Aparajita Zutshi on Fuzzy-Set Qualitative Comparative Analysis and Configurational Comparative Methods. This AHRQ PCMH Research Methods Brief is posted on the AHRQ PCMH website. Details will be provided at the end of this webinar. Jodi Holtrop, PhD has been an Associate Professor in the Dept. of Family Medicine at the Clinical Center of Michigan State University. In July she will become a faculty member of the Medical School at the University of Colorado at Denver. Dr. Holtrop is currently the principle investigator on several funded grants including an AHRQ-funded demonstration grant on “A Comparison of Provider versus Health Plan Delivered Care Management in Michigan.” Her presentation today is based on one part of the research conducted under that grant.

3. Fuzzy Set Qualitative Comparative Analysis (fsQCA): Application to PCMH Research An AHRQ Advanced Methods Webinar Presented by: Marcus Thygeson, MD MPH April 15, 2013 Reference: Thygeson NM, Peikes D, Zutshi A. Fuzzy Set Qualitative Comparative Analysis and Configurational Comparative Methods: Powerful Methods to Study and Refine Patient-Centered Medical Home Models. Rockville, MD: Agency for Healthcare Research and Quality.

4. Acknowledgments • Apu Zutshi (Mathematica Policy Research) for early input to this presentation • Apu and Deborah Peikes (also MPR), my co-authors on the AHRQ methods brief from which this presentation is derived • Leif Solberg (HealthPartners, MN), who has been a great mentor and gave me the opportunity to use fsQCA to study the relationship between quality outcomes and PCMH characteristics • Charles Ragin (UC, Irvine), who originated the fsQCA method, and has generously provided guidance and encouragement to countless investigators, myself included.

5. Intended Results for Today • Understand what fuzzy-set Qualitative Comparative Analysis is, and how it is different from more traditional biostatistical analysis • Understand the kind of studies or questions for which fsQCA is well-suited • Understand some practical considerations to keep in mind if you are going to try using it.

6. Presentation Outline (Part I) • Context setting: • fsQCA as a set-theoretic method for analysis of social phenomena • Comparison between set-theoretic configurational methods and methods based on correlation and regression • Performing a fsQCA study • Study design and set up • Performing the analysis • Sample results from a PCMH study • Advantages • Limitations • (Other applications, useful references)

7. fsQCA—the method • Set-theoretic method that matches the ubiquity of set relationships in social and natural phenomena: • “All PCMHs are primary care practices but not all primary care practices are PCMHs”  “PCMHs are a subset of primary care practices” • Developed in the social sciences for comparative analysis of small-N (“5-50”) case studies • Care delivery systems are social phenomena; each primary care practice is a potential “PCMH” case; most PCMH case series are small • To study phenomena characterized by complex causality • Equifinality: multiple paths to the same outcome • Conjunctural causation: impact of one factor contingent on presence or absence of other factors • Causal and conceptual asymmetry: A  B does not imply not-A  not-B, and the absence of A does not necessarily imply not-A

8. Contrast with Regression A Middle Path

9. fsQCA—based on (fuzzy) set theory • The basic unit of analysis: sets and subsets • Researcher assigns each case (unit of observation) membership in sets representing different explanatory variables and outcomes. • For example, Practice A is in the set of “good team-care” (1) and Practice B is not (0) [This is a “crisp set” binary membership assignment.] • “Fuzzy set” QCA allows cases to have partial (or ‘fuzzy’) membership, between 0 and 1. For example, the fuzzy set of practices with NCQA PCMH certification may look like:

10. Calibration • Assignment of set membership requires “calibration”—an explicit decision about the substantive meaning of observed variation • For instance, temperature is calibrated against the physical properties of water: water freezes at 0 degrees Celsius and boils at 100 degrees Celsius • Water at 37 degrees is “neither in nor out” of the set “warm water” • The calibration process forces the investigator to think carefully about the concepts being investigated • A practice that should be routine in regression-based analysis, but is often overlooked • An example—calibrating “access”

11. Necessity and Sufficiency • Necessary relationships occur when a characteristic, or combination of characteristics, is observed in all practices with the outcome. The outcome is only observed when the characteristic is present. That is, the set of cases with the outcome is a subset of the cases with the characteristic. • Sufficient relationships occur when a characteristic, or combination of characteristics, is observed in a subset of cases with the outcome. The outcome is invariably observed if the characteristic is present. • Correlations are symmetric “necessary and sufficient” relationships: presence of the characteristic implies the presence of the outcome, and vice versa. Cause Outcome Outcome Cause

12. Recognizing fuzzy set relationships in dataNecessity (the lower right triangle plot) Actual output from fs/QCA graph capability demonstrating that higher socioeconomic status is a necessary but not sufficient condition for achieving better optimal diabetes care. In other words, ODC is a fuzzy subset of ~LOWSES (not low socioeconomic status). The consistency of the set relationship is 0.96; and the coverage is 0.62. Thygeson et al., HSR, January 2012

13. Sufficiency (upper left triangle): Serum methylmalonic acid (MMA) and homocysteine (HC)levels in 491 episodes of B12 deficiency. (NEJM 368;2:158) In patients with B12 deficiency, elevated HC is sufficient for elevated MMA. Cases with elevated HC are a subset of cases with elevated MMA. MMA is a more sensitive test for B12 deficiency.

14. Set relationships in B12 deficiency Elevated homocysteine Elevated MMA Elevated vitamin B12

15. Consistency and Coverage • Consistency: • Measures the degree to which one set is a perfect subset of another. Varies between 0 and 1. • Generally require consistency > 0.8 to establish a set relationship. • Analogous to correlation coefficient; measures the strength of the relationship. • Coverage: • Measures the degree to which a subset condition “covers” or “explains” an outcome. Varies between 0 and 1. • Analogous to “relevance”. A highly consistent condition with low coverage has low empirical importance. Low consistency, Low coverage High consistency, Moderate coverage High consistency, Low coverage

16. fsQCA enables configurational analysis • Exploring the relationship between combinations of characteristics and an outcome. • In a study of 21 Level 3 PCMH clinics, low preventive service use (PSUTD) was found to be related to a variety of factors, including average patient BMI, poor communication (~C), caring for women or seniors (WS), and weak practice systems for promoting preventive services (~PS). The findings: BMI or (~C and WS) or (C and ~PS and ~WS)  low PSUTD • Prior statistical analysis had overlooked these relationships. • Importance of practice systems contingent on patient population. • Importance of patient population characteristics obscured when treated as control variables • IMPLICATIONS: work on communication in WS practices, practice systems in non WS clinics Thygeson et al., 2012

17. High-level process overview: • Step 0: Study design, case selection and data collection • Step 1: create a data set of cases with key characteristics and outcome of interest • Step 2: Choose membership function approach and assign fuzzy set membership in each characteristic for all cases. Calibrate based on substantive or theoretical knowledge. • Step 3: Identify necessary and sufficient relationships between possible explanatory conditions and the outcome of interest • Step 4: Use findings from Step 3 to develop or refine logic models ; then use fuzzy set Truth Table Analysis to identify sufficient associations between combinations of conditions (configurations) and outcomes

18. Truth Table Analysis Results • Another example: factors associated with low performance on Optimal Diabetes Care (ODC) in 21 Level III PCMH clinics (Thygeson et al., 2012): • Patients with low socioeconomic status, OR • Women or senior patients, OR • Weak team care capabilities for diabetic patients. LOWSES + (W+S) + ~DTC  low ODC scores consistency = 0.83, coverage = 0.74

19. Advantages of fsQCA • Well-suited to exploratory analyses and theory development • Useful for small-N studies (and also large-N) • Provides systematic approach to conceptualizing and analyzing qualitative data • Identifies equifinality—multiple paths to the same outcome • Useful with non-linear, non-normal data • Can be combined with (complements) statistical methods • Identifies interactions between contextual and instrumental factors • Supports linkage between implementation and impact factors

20. Limitations of fsQCA • Sensitivity to case selection (especially with small N) • Risk of bias due to subjective decision making (e.g., choice of membership function and calibration criteria) • Limits to explanatory power • Case-to-condition ratio limits prevent over-fitting of results • Limits to causal analysis (like regression, association does not imply causality) • Analysis requires considerable time and effort (iteration, and lack of fully featured software tools) • Lack of familiarity means hard to find knowledgeable reviewers for grants and papers

21. Transition to Jodi Holtrop • Appendix slides address more detailed description of the fsQCA process, a list of published health services research articles using the method; and some useful references.

22. fsQCA applications in health services • Examine relationship between medical “home-ness” and quality of care (Thygeson, Solberg, Asche, et al., 2012) • Identify high-performance work practices associated with high front-line health care worker job satisfaction and perceived quality of care (Chuang, Dill, Morgan, et al., 2012 ) • Identify causal configurations of antecedents associated with heavy drinking (Eng and Woodside, 2012) • Organizational designs for achieving high treatment trial enrollment (Weiner, Jacobs, Minasian, and Good, 2012)

23. Useful references • The Comparative Method, Charles Ragin, 1987 • Redesigning Social Inquiry: Fuzzy Sets and Beyond, Charles Ragin, 2008 • Configurational Comparative Methods, Benoit Rihoux and Charles Ragin, 2009 • Set-Theoretic Methods for the Social Sciences, A Guide to Qualitative Comparative Analysis, Carsten Schneider and Claudius Wagemann, 2012 • Thygeson NM, Peikes D, Zutshi A. Fuzzy Set Qualitative Comparative Analysis and Configurational Comparative Methods: Powerful Methods to Study and Refine Patient-Centered Medical Home Models. Rockville, MD: Agency for Healthcare Research and Quality. • http://www.u.arizona.edu/~cragin/fsQCA/

24. Performing the Analysis • Step 0: Study design, case selection and data collection • fsQCA is both a research approach and an analytic method • Often select cases that are exemplars (selection on the outcome), a “no-no” with traditional analysis • Supports “solution” oriented analyses and understanding why something didn’t work that was expected to • Specify concepts and collect data so as to facilitate calibration and assignment of set membership • Develop a logic model for expected relationships • Expect to iterate based on preliminary findings—case selection, data collection, calibration of measures

25. Performing the Analysis (cont’d) • Step 1: create a data set of cases with key characteristics and outcome of interest

26. Performing the Analysis (cont’d) • Step 2: Choose membership function approach and assign fuzzy set membership in each characteristic for all cases. Calibrate based on substantive or theoretical knowledge.

27. Performing the Analysis (cont’d) • Step 3: Identify necessary and sufficient relationships between possible explanatory conditions and the outcome of interest • Necessary conditions are present in all cases with outcome Fuzzy membership in outcome < membership in condition • Sufficient conditions are present in a subset of cases with outcome Fuzzy membership in outcome > membership in condition • Step 4: Use findings from Step 3 to develop or refine logic models ; then use fuzzy set Truth Table Analysis to identify sufficient associations between combinations of conditions (configurations) and outcomes

28. Confessions of a First Time fsQCA User:Using fsQCA to Understand Determinants of Care Management Program Quality An AHRQ Advanced Methods Webinar Presented by: Jodi Summers Holtrop, PhD, April 15, 2013 Co-Authors: Lee A. Green, MD, MPH; Georges Potworowski, PhD

29. Disclaimer • I am NOT an expert on fsQCA; instead I share my experiences using this method; also I am but one of a three person core team. • The study referenced is in process and the results presented are not final; at this point they are only shown for an example of the method.

30. Provider-Delivered Care Management Pilot (BCBSM) and Grant (AHRQ Study) • Studied BCBSM pilot project to examine delegation of care management from health plan to physician organizations and their practices. Aims: • Compare patient outcomes among those offered PDCM versus health plan delivered care management (HPDCM) • Describe the implementation of PDCM • Important: Describe practice environments and contexts of chronic care delivery, and identify implementation, PO, and practice features associated with improved outcomes

31. Focus for this Presentation Provide lessons learned based on how fsQCA was used to explore these questions… • Is there one best way to structure care management or does it vary by context? • What are the necessary and sufficient ingredients for care management program quality?

32. Lesson #1: Set up data collection with fsQCA in mind • Use a logic model as a conceptual guide to start • What are the features or conditions that will potentially affect the outcome of interest? • Within the logic model, identify features and/or conditions, as well as to what extent they exist and to what extent they matter • Helps to calibrate in fsQCA • Specify outcomes in advance • Make sure you are able to get the data • If qualitatively derived into a quantitative value, need to do that in advance and separate from your analysis

33. Care Management and Context Features1.Lit. review identified some key features of CM success; vetted by experts2. Created “scorecard” with Likert-type anchors3. Scored practices on each feature, then inserted notes and later quotations • Patient access to program • CMgr qualifications and personal attributes • Staff CM training • CM resources for staff and patients • CM program attributes (Health promotion, care transitions, disease management) • CM Program duration, dosage, features and length of visit • Practice staff participation in CM • CM team use • Location of CMgr visits with patients • CMgr visit documentation • Incentives and barriers for CM use • PO ownership • Provider buy-in • Practice culture

34. Lesson #2: Know your data Garbage in = Garbage out • QCA is not a “turn the crank” type of analysis – can’t just put the data in and get a result; have to go back and forth with it; understand it and what is happening • Spend time in practices/environments and get to know how systems work, how people interact; Observe and elicit insight from participants • This understanding will be necessary to identify key conditions and to calibrate

35. 5 PO’s, 8 Care Management Models

36. Lesson #3: Specify a reasonable outcome EXAMPLE: • Qualitatively derived overall “Quality Score”→ THIS PRESENTATION • Probably best indicator of quality program, but subject to investigator bias in some cases highly related to the conditions examined • Qualitatively derived “Normalization Score” • Proxy measure of implementation; also subject to investigator bias; Normalization or routinization of poor quality program isn’t a good thing • Quantitative result - “Patient Engagement” • Key problem – difficulties with the limitations of the data which we did not collect; some programs didn’t include all patients – highly dependent on targeting methods and intensity; ideal if we had rates of all patients eligible and solid denominator for targeted • Quantitative result of “Patient Clinical Outcome” • Many factors in line to get to outcome; some we don’t know what they are; but this is likely the ultimate measure; not available yet

37. Raw Data Table Note: These data have already been transformed (calculated in fs terms) to degree of set membership

38. Lesson #4: Careful on selection of cases and sets Warning: LOTS of time and discussion • Our team of three met weekly for about 3 months to determine conditions, examine them and run analysis • Consider what to include as a “row” (case) – PO or practice or provider? • Data coded to that level? Outcome available at that level? • Nesting of practices within PO’s; Is there variation by practice within a PO? If all practices within a PO act similarly then use the PO level • Our decision - Separate by program structure rather than practice (8 cases) • Consider what to include as a “column” (set) • Analysis gets really messy when getting beyond 8; throwing in lots of extra conditions is likely to NOT yield explanatory value • Be thoughtful about what to include; even if you have a lot of data, that doesn’t mean it is all important – step back and look at larger picture; If no meaningful variation do not include as a set – still write up in analysis, but do not include in the QCA (for example we thought incentives might be important, but no one had any incentives)

39. Necessity Analysis • Necessary = condition that the outcome is a subset of • EX: quality (outcome) is a subset of physician buy-in (set or condition) • Scatterplot – all in the bottom right triangle (if outcome on the Y axis) • Consistency = high relationship strength between the condition and outcome • Physician buy-in as a condition seems necessary, but not sufficient

40. Physician Buy-In: Necessary, but not Sufficient ConsistencyCoverage 0.993103 0.798521

41. Sufficiency Analysis • Sufficient = condition that is a subset of the outcome • EX: PO ownership (set or condition) is a subset of quality (outcome) • Scatterplot – at the top left triangle (if outcome on Y axis) • Coverage = high relevance to the outcome • PO ownership seems sufficient, but not necessary – meaning there are other ways to a good outcome

42. PO Ownership: Sufficient, but not Necessary ConsistencyCoverage 0.712644 1.000000

43. Necessary and Sufficient • Necessary • Condition must be present for a good outcome • Having the condition does not guarantee a good outcome, can still have a bad outcome • EX: physician buy-in • Sufficient • Condition(s) alone or in combination will result in a good outcome • Can combine conditions together to create best coverage = combination of conditions that best align with good outcome; unusual to find a single sufficient condition, usually a combination • EX: PO ownership (reason singly sufficient was a proxy for multiple conditions) • Necessary and sufficient • Can be both necessary and sufficient • Can be necessary, but not sufficient • Can be sufficient, but not necessary • Can be neither necessary nor sufficient

44. All Patients Included in CM Program: Neither Necessary nor Sufficient ConsistencyCoverage 0.797701 0.741453

45. If you want Good Quality CM Program… Either: • Have a team-based approach, have an on-site care manager, have good provider buy-in, plus a good care manager OR • Be owned by a PO • Owned practice comes up sufficient because they demonstrate the conditions (team-based, onsite and have good care managers) – proxy

46. Truth Table – Truncated (does not include all the unobserved cases) **There were eight total cases, but two cases did not have >0.5 membership in any configuration so their configurations did not appear in the table; they were consistent with the necessity and sufficiency conditions identified

47. Conclusions • fsQCA can be a valuable tool for health services research if you have… • An appropriate study/question • Variability in conditions and outcomes • Meaningful data that tie to an outcome(s) • QCA seems to work best when a multi-disciplinary team is involved • Gathering and analyzing QCA data for meaningful outcomes can be time-intensive and requires in-depth knowledge of your data

48. Thank you for attending! Register for this and other events in our series on advanced methods in delivery system research, here: http://bit.ly/EconometricaAHRQ Webinar #2: Statistical Process Control Presenter: Jill Marsteller, PhD, MPP Discussant: Stephen Alder, PhD Moderator: Michael I. Harrison, PhD Sponsored by AHRQ’s Delivery System Initiative in partnership with the AHRQ PCMH program May 14, 2013 1 p.m.

49. For more information about the AHRQ PCMH Research Methods briefs, please visit:http://www.pcmh.ahrq.gov/portal/server.pt/community/pcmh__home/1483/pcmh_evidence___evaluation_v2