KU07 Conference Stockholm. Sweden August 16 th , 2007

1. The dependent variable in knowledge translation research: What should we measure? How should we measure it? Carole Estabrooks PhD, RN Janet Squires PhD Student, RN Lars Wallin PhD, RN Petter Gustavsson PhD KU07 Conference Stockholm. Sweden August 16th, 2007

2. Outline • The need for a measure • Measures to date • The Canada/Swedenmeasurement study • How should we go about measuring RU? • Introduction to group discussions • What does the theory tell us to measure?

3. The need for a measure CE

4. Why measure research use instead of patient outcomes when assessing KT interventions? • Not instead – in addition • However, if we do not measure research use, it is difficult to claim that the intervention tested to increase research use was effective • And, if we only measure patient outcomes….it is difficult to determine if an improvement in such outcomes is attributable to the intervention to increase research use • Finally, working in organizations, it is we argue useful to develop and use measures of KT that reflect use at unit (possibly team) and facility levels, in addition to individual levels

5. What is Research Utilization? Types of Research Utilization • Instrumental • Conceptual • Symbolic/Persuasive

6. Measures to date Nurses & Allied Health Professionals JS

7. Nurses and Allied Health Professionals Review of nursing and allied literature Estabrooks, Wallin, & Milner. (2003). ). Measuring knowledge utilization in health care. International Journal of Policy Evaluation & Management, 1(1), 3-36. Purpose: To assess the state of measurement science specific to the concept of research utilization

8. In that review the group assumed some requirements of robust measurement instruments • Substantive (content) theory • Measurement theory • Minimal self-report & social desirability effects • Unequivocal scale • Acceptability within the research field

9. The review • Search Strategy • Published reports of research ‘use’ • CINAHL and Medline electronic databases • Search Results • 42 publications • 3 commonly used multi-item measures • Multiple single item measures JS

10. Findings Existing Measures Common Multi-item Measures: • Nursing Practice Questionnaire (N=3) • Nursing Practice Questionnaire (Modified) (N=6) • Research Utilization Questionnaire (N=7) • Edmonton Research Orientation Survey (N=3) Other multi-item measures (N=3) Single item measures (N=20)

11. Multi-Item measures • NPQ (Brett 1987) • Based of Rogers innovation-decision process theory, reliability limited to chronbach alpha, content validity assumed • RUQ (Champion & Leach 1989) • reliability limited to chronbach alpha, limited validity assessments, operational defn of use unclear • EROS (Pain, Hagler, & Warren 1996) • reliability limited to chronbach alpha, construct validity assessment, operational defn of use unclear, recall bias

12. Single Item Measures • 20 studies used single-item measures • Similar limitations to multi-item but also: • Different terminology (makes it difficult to compare studies) • Array of different scoring methods and scaling assumptions • Recall bias

13. Measures to date Medicine LW

14. Guideline literature • Based on Grimshaw et al (2004). Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technology Assessment,8, 1-72. • 235 studies were reviewed • Data extracted on two main measures • Process of care results • Outcome of care results • Not described how data was collected, exemplified with charts and self-reports • 225/235 studies used process of care measures • 50/235 studies used outcome of care measures

15. Guideline literature Examples of measures • Process of care • % of patients blood pressure measured (hypertension screen) • % of patients smoking status recorded • % of patients documented nutritionist consult • Rating of time spent counseling (physician questionnaire) • Outcome of care • Systolic blood pressure • % of patients stopped smoking at 12 month • % of patients satisfied with care • Sickness Impact Scores at 6 month • % of patients surviving at 1-year

16. Regardless of the measure…. General Problems Across Measures Measuring the construct of ‘research use’ • Poor construct clarity • Lack of theoretical framing • Presumption of linearity • Common assumptions held about research use Methodological • Lack of measurement theory underpinning • Lack of psychometric assessment • Unit of analysis - ecological fallacies • Retrospective survey design dependent on self-report and recall • Operationalization of ‘use’ unclear • Absence of longitudinal work • Scaling JS

17. Update of the 2003 measurement review – repairing prior sins Expanded Search • Databases:HAPI, CINAHL, MEDLINE, EMBASE, SCOPUS, OCLC Papers First, OCLC WorldCat, ABI Inform, Web of Science, Sociological Abstracts • Hits: 23,065 • Abstracts Screened: 11,810 • Articles pulled: n=105 • Need more information: n=497 • Hand search • Cross references

18. Some notes on Experiences with a single item measure CE

19. Research Utilization Studies (1999-2003) Range: 1-7

20. Alberta Nurse Survey (2002) * = One-Way ANOVA P-value <0.001

21. CIVMIL Study (1999, 2005) * = Gamma (ordinal test) statistic P-value <0.05

22. AKUTE Study * = One-Way ANOVA P-value <0.05

23. Research use in nursing practice in relation to educational, individual and work contextual factors(Sweden) • Data collected within the LANE project (Longitudinal analysis of Nursing Education) • Nationwide survey, prospective longitudinal design, 2002-2008 • Two cohorts of nursing students • Instrumental, conceptual and symbolic research use at 1, 2, 3 and 5 years after graduation • Aims: To identify predictors and to study the variability over time of nurses’ research use LW

25. The current measurement study Canada/Sweden CE

26. Developing a Valid and Reliable Measure of Research Utilization(Ref: Belfast KU04) Purpose: To develop and assess an instrument that measures research utilization among health professionals The Canada ARM:CIHR (2005-2007) Investigators:Estabrooks, Wallin, Scott-Findlay, Profetto-McGrath, Harley The Swedish ARM:Karolinska Institute supported Investigators:Wallin, Strandberg, Kajermo-Nilsson, Gustavsson

27. Measurement Study Planned outcomes: • A set of definitions for research utilization • A finalized version of the research utilization concept model previously developed by the team and expert panel • A preliminary set of indicators of research utilization from which to develop items for the instrument • An instrument

28. The Approach • Construct clarity  Series of focus groups with three sets of experts • Local (research team) expert panel • International panel • Managers and educators • Providers (RNs/RPNs, LPNs/aides) • Indicator and item development  Series of focus groups with two sets of experts • Managers and educators • Providers of nursing care (RNs/RPNs, LPNs/Aides) • Develop and pilot test the draft instrument(s)

29. Measurement Study International Panel: • Jo Rycroft-Malone (UK) • Carl Thompson (UK) • Marita Titler (US) • Anne Sales (US) • Judith Ritchie (Canada) • Jo Logan (Canada) • Nancy Edwards (Canada) • Donna Ciliska, (Canada)

30. Conceptual Use Conceptual Use Conceptual Use Action in Practice Concept Map (Study Team) “Thinking” (i.e., Clinical Decision Making Clinical Reasoning ) “Action” Instrumental Use Task completion Research Use Instrumental Use Task completion Relational Assessment Screening Research Non-Use Non-use A. Informed (non-defensible) Non-use B. Informed (defensible) Non-use Uninformed

31. Research utilization HI Conceptual Instrumental LO HI LO

32. Findings: Canadian ARM JS

33. Focus Groups – Round 1: Themes Identified

34. Terminology • Majority related to instrumental research use • No delineation between low and high conceptual use • “Research Utilization” not commonly used • Terms commonly used included: • Evidence-based medicine (practice) • Research-based knowledge • Based on research • Based on clinical evidence

35. Definitions • Research Use • Putting something into practice • A personal choice or judgement • Conceptual Research Use • Thinking, critical thinking • Instrumental Research Use • Action, implementing, observable

36. Examples • Examples increased in concreteness from manager/educator to RN/RPN to LPN/aide • Most examples r/t instrumental research use • Gradient from conceptual to instrumental • Examples common to groups: • Talking/discussing • Following policies and procedures/protocols • Specific conditions: • e.g. wound care, Braden scale

37. Indicators • Following research-based policies and procedures/protocols • Asking questions • Peer-to-peer interactions • Reading research literature • Bringing in research articles • Self report • Observation • Chart documents

38. Focus Groups – Round 2 Validation for IRU schema developed based on round 1 analysis: • Self-report • Questionnaire • Interview (healthcare provider, manager) • Observation • Chart audit

39. Findings: Swedish ARM LW

40. Terminology • Research Utilization not a common term (but might work) • More accepted than evidence-based practice - “trendy” concept

41. Definitions • Research Use • Basing nursing on scientific findings • Working according to appraised and compiled information • Adopting a critical approach to one’s work • Conceptual Research Use • learning through critical reflection/thinking • Instrumental Research Use • Active process

42. Examples • Most examples r/t instrumental research use • Adherence to research-based routines/guidelines • Using systematic assessment instruments • Quality management work • All groups mentioned application of concrete findings r/t procedures and treatments: • Insulin Injection technique • S/C injection of sterile water for labour pain

43. Indicators • Felt instrumental and conceptual research use were observable • Following research-based routine/guidelines • Updating routine/guidelines • Observation: • of specific practices, using systematic assessment instruments, a structured nursing ‘working method’ • Chart documents • Using internet or intranet to seek information • Providing rationale for one’s actions • Peer-to-peer interactions

44. Similarities between Canadian and Swedish findings Instrumental: • Adherence to research-based policies/protocols • Easier to talk about specific practices • Examples increased in concreteness from manager/educator to RN to LPN/Aide Conceptual: • A progression of conceptual to instrumental • Equated with thinking/critical thinking LW

45. Learnings & Directions • Participants not at all accustomed to discussing research utilization • Consequently, focus groups not an optimal method • Much confusion about conducting research vs. use of research • Can we distinguish conceptual RU from instrumental RU? RU often viewed as a process with similarities to the process of EBP. Is conceptual use ‘necessarily’ a precursor to implementation? What about implementation without understanding?

46. Learnings & Directions • Critical reflection viewed as an important attribute to RU. At the same time participants claimed that much of RU was unreflected. • Enhanced awareness of the knowledge base for clinical practice - a strategy for RU? How to measure RU if practitioners are not aware of if they are using research? • Adaptation to patients’ perceptions and preferences important and contributes to make the concept of RU vague. Consequences for measuring RU?

47. Next Steps • Systematic review (update) of instruments to measure research use (in-progress) • Manuscript on Canada/Sweden findings re instrumental use (in-progress) • Concept analysis (literature based) on conceptual use (in-progress, Canada) • Submit grant applications 2008 • Develop and pilot instrumental measures • Construct clarity re conceptual use (concept analysis using ethnoscience) • Eventually development of a measure of CRU… CE

48. Psychometric theory PG

49. Sources

50. Manifest Observed responses Latent, Not manifest