Developing the CAHPS for Cancer Care Survey

1. Developing the CAHPS for Cancer Care Survey Steven Garfinkel, American Institutes for Research Kathleen Yost, Mayo Clinic AHRQ Annual Conference Bethesda, Maryland September 10, 2012 Project Cosponsored by the National Cancer Institute and the Agency for Healthcare Research and Quality under AHRQ Contract# HHSA290200600019I, Task Order# HHSA29032003T TO#9

2. Why Develop a CAHPS for Cancer Care Survey? • Uniformly measure cancer care by providers from a patient perspective • Compare the care provided by different providers • Provide information for quality improvement

3. CAHPS Development Process

4. Formative Research

5. Item Inventory • 1781 items • CAHPS surveys • Cancer surveys • All items coded into hypothesized domains • e.g., affective communication, shared decision-making • Three independent reviewers of initial domains and items

6. Focus Groups 16 focus groups to assess how consumers identify cancer providers and quality of care • Stage 0/I (solid and common tumors) • Stage II/III (solid and common tumors) • Stage IV (any cancer) • Rare cancers • Hematologic cancers • Spanish speaking • Family and Caregiver

7. Focus Group Topics • Who are the key providers that patients focus on when they think about quality of care? • What are key factors that patients use to determine quality of care? • Are domains developed by the team relevant to patients? • Which domains are most important?

8. Focus Group Findings • Cancer care providers • Oncologists and surgeons most often • Other clinical staff: nurses, nurse practitioners, physician assistants, primary care physicians, and technicians • Non-clinicians: support groups, family members, chaplains, and even a therapy dog • Perceived care coming from a provider and a team or teams

9. Focus Group Findings, continued • Top two domains of quality: • How providers interact with patients • “They care about you. There’s nobody else in the world but you.” • “He’s always been straightforward. He didn’t sugarcoat.” • Technical communication skills • “My doctor explained to me that chemo is very hard on the body. He didn’t want to jump in and do the chemo. When he found another lump in my neck then he said we have to treat it.”

10. Focus Group Findings, continued • Consumers also interested in • Care coordination • Access to information • Providers who are up-to-date • “There is so much research and so many new drugs coming out that you have to be up on it and pay attention to it because the more targeted the chemo the better.” • Shared decision making

11. Stakeholder Interviews • Technical Expert Panel of clinicians and patients • Interviews with representatives of 5 clinical professional organizations • Interviews with representatives of 5 community cancer centers • Discussions with Mayo Clinic Oncology staff

12. Stakeholder Findings • Most cancer care is delivered through cancer centers • Over 1,500 accredited by ACS Commission on Cancer • Survey will be most useful if it provides scores by major treatment modalities: • Medical, radiation, and surgical oncology

13. Cognitive Testing • Detects unexpected problems • Provides an underlying model of the questionnaire item response process

14. Cognitive Testing Findings • Need to define “team” • Medical oncology = drug therapy • Specify “cancer” surgery to distinguish focal surgery from surgery for other conditions received by cancer patients

15. Major Design Decisions • Use the cancer center as the unit to assess • Focus on post-diagnosis and active treatment • Include inpatient and ambulatory experiences • Use 3-month reference period • Sample equally from medical, radiation, and surgical oncology patients • Parallel questionnaires for medical, radiation, and surgical oncology

16. Example Item • In the last 3 months, how often did your radiation therapy team listen carefully to you? • In the last 3 months, how often did your cancer surgery team listen carefully to you? • In the last 3 months, how often did yourdrug therapy team listen carefully to you?

17. Field Test

18. Field Test Activities • Site recruitment • Eligibility and Sampling • Response rate and response bias analysis • Evaluation of mode effect • Comorbidity Index Study • Item functioning • Analysis to determine structure of composites • Case-mix adjustment

19. Field Test Design and Preliminary Results on Survey Operations

20. Purpose of Field Test • To develop an assessment of the quality of cancer care that would obtain the CAHPS trademark • Assess psychometric properties of survey • Item non-response (including failed skip patterns) • Mode effects (mail vs. phone) • Composites • Case-mix adjusters • Assess ability of survey scores to discriminate quality of cancer care across sites • Evaluate feasibility of administering the survey in different treatment settings • Challenges of enumerating a sample • Unit non-response

21. Site Recruitment • Initially intended to work with a clinical trials cooperative group • American College of Surgeons Commission on Cancer (CoC) expressed interest in possibly sponsoring the survey • 6 CoC member organizations were recruited • Large enough case volume to provide 750 eligible patients (250 per treatment modality) within data collection timeframe • 300 completes per site (100 per treatment modality) • Assumed 40% response rate • Geographic representation • Sampling frames sent to Mayo Clinic Survey Research Center • Waiver of written HIPAA authorization at 5 sites • Exception of Site 3

22. Site recruitment

23. Patient Eligibility • Age 18 or older at the time of the recruitment material mailing • Received treatment for cancer in the 10 weeks prior to being selected for the sampling frame. • Treatment setting can be inpatient or outpatient. • Treatment includes surgery, medical oncology and radiation therapy. • Medical oncology includes chemotherapy, hormonal therapy, immunotherapy, and any other treatment for cancer delivered as a drug. • Treatment intent can be curative or palliative • For radiation and medical oncology care, an additional criterion is that the patient had at least two visits to your organization for cancer treatment in the previous 10 weeks. • In one of the following three phases of the cancer control continuum: receiving treatment for a new diagnosis, receiving treatment for a recurrence, or completed cancer treatment within the sampling window and is transitioning to survivorship • Not receiving care for squamous cell or basal cell skin cancer.

24. Data Collection • Mayo Clinic Survey Research Center • Mail arm • Goal – 1500 completes • Initial recruitment packet with letter and survey • Second mailing to non-respondents after 3 weeks • 3 attempted reminder phone call to non-respondents after 3 weeks • No survey administration by phone if contact was made • Phone arm • Goal – 300 completes • Prenotification letter • Phone interviews begin 2 weeks after prenotification • 6 attempts – 2 weekday day, 2 weekday night, 2 weekend across 2 weeks • Embedded study of comorbidity questions • Double data-entry

25. Data Collection Protocol for All Sites

26. Misidentified Cases • Site 2 • 2 cases in cancer surgery arm self-identified as ineligible – did not receive surgery for treatment of cancer • Unanticipated Problems Involving Risk to Subjects or Others (UPIRTSO) • Unanticipated risk of emotional harm • Suspension of recruitment of cancer surgery patients at Sites 1 and 2 pending chart review • Chart review found 1 additional case • Follow-up protocol not completed at Site 2 due to IRB delay • Sites 3 – 6 changed IRB protocols to acknowledge misidentification as an anticipated risk

27. Eligibility Determination Among Respondents

28. Response Rate

29. Distribution of Demographic and Cancer Type Among Responders and Nonresponders

30. Quality of Care Questions That Differ by Mode of Survey Administration

31. Mode of survey administration (cont.)

32. Embedded Study of Comorbidity Questions • 2 versions of survey • Version 1: “Are you currently being treated by a health professional for…” • Diabetes/high blood sugar, high blood pressure/hypertension, heart condition, chronic lung disease/COPD/emphysema/chronic bronchitis, arthritis/joint problems, depression/anxiety • Number of conditions reported • 0 = 43.4% • 1 = 30.2% • 2 = 18.5% • 3+ = 7.9%

33. Embedded Study • Version 2 • Duration of care • “In the last 12 months, did you get health care 3 or more times for a condition or problem? • Is this a condition or problem that has lasted for at least 3 months?” • Duration of prescription medication • “Do you now need or take medicine prescribed by a doctor?” • “Is this medicine to treat a condition that has lasted for at least 3 months?” • Indices were defined as shown below:

34. Embedded Study • Version 2 indices results:

35. Composite Measure Analysis • Create analytic files • Confirmatory Test of Original Composite Structure • Identification and Evaluation of Alternative Composite Structures • Evaluation of Validity of Best Alternative

36. Create Analytic Files • Recode to facilitate interpretation • For some questions higher numbered response indicated better care; for others, a higher numbered response indicated worse care • Reverse-scored questions where higher indicated worse care so that higher always indicates better care • Impute missing values • Multivariate analysis requires complete data for all variables in the analysis • Used CAHPS-adopted method of multiple-imputation

37. Original Composite Structure *All questions in this domain are also listed under other domains.

38. Revised Composite Structure

39. Preliminary Composite Results

40. Affective Communication • In the last 3 months, • how often did your radiation therapy team treat you with courtesy and respect? • how often did your radiation therapy team show respect for what you had to say? • how often did you feel your radiation therapy team really cared about you as a person? • how often did your radiation therapy team listen carefully to you? • how often was your radiation therapy team direct and straightforward when talking with you about your cancer and radiation therapy? • how often did your radiation therapy team spend enough time with you?

41. Shared Decision Making • Since your cancer was diagnosed, did a doctor or other health care professional at [NAME OF CANCER CENTER] • clearly explain the advantages of each choice for cancer treatment, including the treatments you did not get?   • clearly explain the disadvantages of each choice for cancer treatment, including the treatments you did not get? • ask for your opinion about each choice of cancer treatment, including the treatments you did not get? • involve you in decisions about your cancer treatment as much as you wanted?

42. Communication About Therapy • Since it was decided that you would have radiation therapy, • did your radiation therapy team encourage you to contact them with questions between visits? • tell you to call them immediately if you have certain symptoms or side effects? • give you clear instructions about how to contact them outside of regular office hours? • clearly explain how your cancer and radiation therapy could affect your normal daily activities? • In the last 3 months, did your radiation therapy team • seem up-to-date about how to treat your type of cancer? • tell you what the next steps in your radiation therapy would be? • explain test results in a way that was easy to understand? • explain what that medicine was for in a way that was easy to understand?

43. Enabling Patient Self-Management • In the last 3 months, did your radiation therapy team • advise you about or help you deal with these changes in your energy levels? • talk about any emotional problems, such as anxiety or depression, related to your cancer or radiation therapy? • advise you about or help you deal with these emotional problems? • In the last 3 months, did you and your radiation therapy team • talk about additional services to manage your cancer care at home, such as home health care, special medical equipment, or special supplies? • talk about things you can do to maintain your health during cancer treatment such as what to eat and what exercises to do?

44. Evaluation of Composites

45. Questions Significantly, Uniquely Related to Overall Care Net of 4 Composites Q16. …did your therapy team encourage you to contact them between visits? Q18 ….did your therapy team give you clear instruction about how to contact them outside of regular office hours? Q19. …did your radiation therapy team clearly explain how your cancer and therapy could affect your normal daily activities? Q34. How often were the blood tests,… scans, or other procedures scheduled to be done as soon as you thought you needed? Q36. …how often did your team explain test results in a way that was easy to understand?

46. Next Steps • Continue data collection at 4 sites and analysis with additional support from the California HealthCare Foundation • Revise analysis • Final recommended composite structure • Final recommended questionnaire.

47. Challenges • Extremely challenging project • Focuses on a single clinical condition across multiple care sites • Choosing unit to be assessed • Maximizing potential for adoption and sustainability • Defining the care team—all or modality-specific • Sampling feasibility • Determining patient eligibility • Identifying sampling frames

48. Acknowledgements • AHRQ: Judy Sangl, Chris Crofton, Elma Chowdry • NCI: Neeraj Arora, Steve Clauser • CoC: Steve Edge, David Winchester, Andrew Stewart • Donna O’Brien and Arnold Kaluzny

49. Preliminary Case Mix Adjusters • Age • Education • Self-rating of Mental Health • Race: Other‡ • Treatment/Services Received: Surgery (q58_3) • Treatment/Services Received: none/ w&w (q58_6) • Number Visits to Cancer Center for Appts (q21) • How long have you been treated for cancer (q2)