Using Qualitative and Quantitative Methods in Instrument Development

1. Using Qualitative and Quantitative Methods in Instrument Development Gail L. Towsley, PhD Assistant Professor

2. Our Team Members Susan L. Beck, APRN, PhD, FAAN, Professor Associate Dean for Academic Programs University of Utah College of Nursing Patricia Berry, PhD, APRN, BC-PCM, Assistant Professor University of Utah College of Nursing Marjorie A. Pett, DSW MSTAT, Research Professor University of Utah College of Nursing Jeannine M. Brant PhD(c), APRN-CNS, Oncology Clinical Nurse Specialist St. Vincent Healthcare, Billings, MT Ellen M. Lavoie Smith, PhD(c),APRN-BC, AOCN Director of Advanced Practice Nursing Dartmouth Hitchcock Medical Center, Lebanon, NH

3. Study Aims • Project Aim: To develop and test a measure of the quality of nursing care related to pain management at the end of a nursing care shift. • Presentation Aim: To demonstrate how we systematically approached tool development using qualitative and quantitative data.

4. Background • What are mixed methods? “involves the collection or analysis of both quantitative and/or qualitative data in a single study in which the data are collected concurrently or sequentially, are given a priority, and involve the integration of the data at one or more stages in the process of research” (p.212). Creswell, JC., Plano Clark, VL., Gutman, ML., & Hanson, WE. (2003). Advanced mixed methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of Mixed Methods in Social and Behavioral Research. Thousand Oaks: Sage.

5. Measuring Quality of Nursing Care related to Pain Management Aim 1. To evaluate patient understanding of the items in the PaNCQ survey using cognitive interviewing. Aim 2. To establish the internal consistency reliability and construct validity of the PaNCQ survey in a multi-site sample of inpatients with cancer who are experiencing pain. Aim 3. To examine the feasibility and variability associated with repeated administration of the PaNCQ at the end of each shift over 2 to 3 days. Aim 4. To examine the degree to which each factor in the PaNCQ survey predicts pain outcomes (intensity, relief, and interference with usual function). Funded by Robert Wood Johnson Foundation, Grant 58299

6. Modern measurement theory evaluation of evidence • test content • response processes • internal structure • relations to other variables • consequences of testing American Psychological Association, American Educational Research Association, and National Council on Measurement in Education, Standards for educational and psychological testing. 1999, Washington DC: APA.

7. Mixed Methods • Item generation • Content Validity • Cognitive Interviewing • Exploratory Factor Analyses

8. Item Generation • In-depth qualitative interviews • Two groups of patients with unrelieved pain (n=36) • Advanced cancer • Post-operative for cancer surgery • Transcribed and analyzed (N-Vivo) • 102 items generated from interview data

9. Item Generation:Conceptual Definitions • Being Treated Right • Safety Net • Efficacious Pain Management • Partnership with the nurse and healthcare team related to pain management

10. Content Validity Judgment Stage • 5 Research team members rated item relevance • 19 items deleted Examples: My nurse treated me gently. I had complete pain relief. My pain control was as good as it could be.

11. Content Validity PaNCQ Version 3 • 88 items • Reviewed by 9 national pain management and nursing care quality experts • 4 option rating scale Lynn, M.R., Determination and quantification of content validity. Nursing Research, 1986. 35(6): p. 382-5.

12. Sample Survey Item

13. Content Validity Content Validity established (p =.05) if 8/9 experts agreed the item was relevant. • A conservative approach—more items retained. • Expert panel results: items were deleted, reworded, & added.

14. Example of constructs and retained items (n=75) Treated right: My nurse responded quickly to my reports of pain. Safety net: My nurse followed-up to make certain that the pain medications were working Efficacious pain management: The pain medication kept me comfortable Partnership: My healthcare team discussed my pain management plan with me.

15. Cognitive interviewing • Focuses on the cognitive processes used to answer questions • Goal of understanding how participants interpret and respond to survey questions • Well-established techniques • a detailed script • scripted and spontaneous probes • structured recording tool plus audiotaped • rotating starting point • at least 5 responses per item Willis, G.B. Cognitive Interviewing, A How to Guide: Reducing survey error through research on the cognitive and decision processes in surveys. 1999 April 12, 2006 [cited April 12, 2006]; Available from: http://appliedresearch.cancer.gov/areas/cognitive/interview.pdf.

16. Cognitive Interviewing Cognitive interviews with hospitalized cancer patients experiencing pain • how participants understood the 75 items • how they selected a response on 6 point Likert-type scale • could they answer based on nursing care during past shift

17. Cognitive Interviewing Techniques • Detailed script for the cognitive interviewing • Both scripted and spontaneous probes. • Rotate the “starting point” for each interview to minimize patient fatigue and assure at least 5 responses per item.

18. 4. During the past shift, my nurse did not believe my reports about my pain. Strongly Disagree Strongly Agree 1 2 3 4 5 6 What did you think about when you answered this question? What does “believe my reports” mean to you? How did you decide to pick that number? (referring to scale) Do you think other people would answer this question honestly? How could we say this better? Sample from Script

19. A Summary

20. Analysis • Participant responses to the PaNCQ survey items were summarized for each item using a matrix tool and evaluated in team meetings. • We summarized and discussed each item as a team. • Decision to delete, test more (usually with rewording), or keep. • Result was PaNCQ survey (v4) which has 44 items ready for psychometric testing.

21. Lessons • Time consuming • Wave approach didn’t work • Five per item is good for first round • May need further testing • Variability in “informant” quality • Role of nurse in pain management may not be clear • Need for two scales: one re: interdisciplinary care over hospital stay and one re: nursing care during past shift

22. Exploratory Factor Analyses • PaNCQ Surveys administered • 44 items • 6 pt. Likert type scale • within 2 hours of completion of nursing care shift • Interdisciplinary & nursing care components

23. Steps of Analyses • Data cleaning procedures • Missing data • Correlations • Approach • Theoretical Framework – 2 groups • Interdisciplinary & nursing • Evaluated separately & together

24. Steps of Analysis • Principal Axis Factoring (PAF) • Oblimin rotation • Retained items if • Range of responses included entire scale • Inter-item correlations were between .30-.70 • Item loaded >.40 on only one factor • Ran interdisciplinary group, then nurse group, then together again

25. Item Reduction • Item correlations above .8 • Are there similar items that are more clear? • "I was treated like an important member of the team that made decisions about how to manage my pain.” • item similar to other items but wordier • Items that Loaded on 2 factors • "My healthcare team explained that most patients on pain medication will not become addicted to medicine over time.” • Reword?

26. Item Reduction • Evaluated similar items • Considered correlation between items, variability, and case summaries. • If cases portrayed answers that were very different, we didn't take either one out • Potentially ambiguous items • "My nurse asked me about my pain.“

27. Results • Interdisciplinary care scale • 11 items • 2 factor solution • Nursing Care Scale • 22 items • 3 factor solution • 11 items deleted resulting in a 33 item version of the PAIN-CQ

28. Conclusions • Combining qualitative and quantitative methods is an important phase in instrument development. • Integration of qualitative and quantitative data helped us make decisions about items. • Further testing of the PAIN-CQ will include items that are easily understood and judged by hospitalized patients with pain.