Evidence Based Practice

1. Evidence Based Practice Steven R. Pruett, Ph.D, CRC SERNRA Conference May 18, 2005

2. Evidence-Based Practice • Since the passage of the HMO Act of 1973, there have been many changes in the American healthcare system. • Consumer-driven • Market-based • Customer choice • Customer satisfaction • Health outcomes

3. Evidence-Based Practice • Rehabilitation Healthcare systems, researchers, practitioners and other stakeholders need to provide evidence to support the effectiveness of services. • Particularly relevant in rehabilitation has has given rise to the emphasis on consumerism, consumer involvement, and program evaluation.

4. Evidence-Based Practice • One of the ways healthcare providers have responded to these managed care system demands is through evidence-based practice.

5. Definitions • Rosenberg and Donald (1996) defined evidence-based medicine as a process of turning clinical problems into questions and thensystematically locating, appraising, and using contemporaneousresearch findings as the basis for clinical decisions. • Sackett, Rosenberg, Gray, Haynes, and Richardson (1996) described evidence-based practice as the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.

6. Definitions • DePalma (2000) further refined the definition of evidence-based practice as a complete process beginning with knowing what clinical questions to ask, how to find the best practice, and how to critically appraise the evidence for validity and applicability to the particular care situation. The best evidence then must be applied by a clinician with expertise in considering the patient's unique values and needs. The final aspect of the process is evaluation of the effectiveness of care and the continual improvement of the process. • Ottenbacher and Mass (1998) indicated that the “best evidence” used to support evidence-based practice is derived from a series of research studies results in an empirical consensus regarding the effectiveness of a treatment approach.

7. Evidence-Based Practice & Rehab Case Management • Limited empirical support for rehab case mgt interventions. • Central focus of this service is to evaluate, managed & coordinate services for PWDs (Chronister, da Silva Cardoso, Lee, Chan & Leahy, 2005) • Medical Case Mgt assesses, plans, implements coordinates, monitors & evaluates options and services to meet individual’s health needs through communication and available resources to promote quality & cost-effective outcomes. (Mullahy, 1995) • Voc case mgt is the movement of a client through the rehab process & includes the mgt and coordination of all services needed to achieve successfully the rehab goal. (Cox, Connolly & Flynn, 1981)

8. Evidence-Based Practice & Rehab Case Management • So…case management emphasizes: • Evaluation • Outcomes • Quality assurance • Therefore…case managers are already working, in part, as evidence-based practitioners without explicitly recognizing it.

9. Rehab and Evidence-Based Practice • Quality Assurance – standards of practice • Case Management Society of America • Individual Case Management Assoc & Aetna Health Plan developed Case Management Practice Guidelines. • CRCC code of professional ethics. • Decision-making process • Evaluation needs to be based on credible scientific evidence versus subjective experience.

10. Rehab and Evidence-Based Practice • Rehabilitation researchers must conduct systematic research to demonstrate the effectiveness of rehabilitation counseling/case management interventions. • RCs/case managers must be able to assist clients in selecting the most appropriate medical, psychological, educational, social and vocational interventions for a client’s particular situation.

11. Rehab and Evidence-Based Practice • To better practice evidence-based rehabilitation RC/case managers must have the following knowledge: • Research design • Statistics • How to access pertinent research

12. Evidence-Based PracticeGeneral Concepts • Gold standard of best practice in medicine • In medicine, with its positivist scientific methods tradition, the “gold standard” for scientific evidence is still randomized clinical trials and the method of choice for determining the cumulative evidence of the effectiveness of a treatment is meta-analysis. • Randomized clinical trials • Meta-analysis

13. Evidence Based Practice • Steps for the evidence-based practice of medicine by practitioners: • Formulate a clear clinical question from a patient's problem. • Search the literature for relevant clinical articles. • Evaluate (critically appraise) the evidence for its validity and usefulness. • Implement useful findings in clinical practice. (Rosenberg & Donald, 1996)

14. Evidence-based Practice:Some Rehab Questions • What processes/techniques make a specific rehabilitation intervention work? • For whom is the intervention most effective? • Are certain interventions/programs better for certain populations? • Who should receive a specific intervention or program? When? And for how long?

15. A Hierarchy of Levels of Best Evidence • Level 1 evidence is defined as strong evidence from at least one systematic review of multiple well-designed randomized controlled trials. • Level 2 evidence is defined as strong evidence from at least one properly designed randomized controlled trial of appropriate size. • Level 3 evidence is defined as evidence from well-designed trials without randomization, single group pre-post, cohort, time series, or matched case-controlled studies.

16. A Hierarchy of Levels of Best Evidence • Level 4 evidence is defined as evidence from well-designed non-experimental studies from more than one center or research group. • Level 5 evidence is defined as opinions of respected authorities, based on clinical evidence, descriptive studies, or reports of expert committees.

17. Related Concepts – Empirically Supported Treatment • APA Division 12 defined empirically supported treatment (EST) as clearly identified psychological treatments shown to be efficacious in controlled research studies with a delineated population. • EST should be evaluated in terms of efficacy (statistical and clinical significance), effectiveness (clinical utility), and efficiency (cost-effectiveness).

18. Related Concepts – Empirically Supported Treatment • Patterned after the FDA guidelines for approval of new drugs, the APA Div 12 Task Force on the Promotion and Dissemination of Psychological Procedures established two criteria for establishing the empirical validity of a psychotherapeutic approach: • The approach is superior to a placebo or other treatment or • The approach is equal to an established treatment, in at least two studies established by different investigators.

19. Related Concepts – Meta-Analysis • Meta-analysis is a method used to review research literature based on statistical integration and analysis of research findings. • In treatment effectiveness meta-analysis, the dependent variable is the effect size (i.e., the outcomes or results of each study selected for review transformed into a common metric across studies) and the independent variables are study characteristics (i.e., participants, interventions, and outcome measures).

20. Related Concepts – Meta-Analysis • Meta-analysis is an effort to review the results of a research domain in quantitative terms. The intent is to identify what significant relationships exist between study features (independent variables) and effect sizes (dependent variable). • Benefits: The benefits of meta-analysis include its ability to: • Synthesize the results from many studies succinctly and intuitively for nonscientific communities, • Illustrate the amount and relative impact of different programs on different criteria for policy decision-making purposes, and • Identify the most effective programs and highlight gaps or limitations in the literature to suggest directions for future research

21. Related Concepts – Meta-Analysis • A common index of the size of the effect produced by each study is the effect size index g, which is the standardized difference between the sample mean of the treatment group and the sample mean of the control group (Wampold, 2001). A positive score indicates that the treatment group outperformed the control group, and a negative score has the reverse meaning. However, the effect size index g is a sample statistics. As such, it is a biased estimator of the true (i.e., population) effect size.

22. Related Concepts – Meta-Analysis • Hedges and Olkin (1985) provided the effect size index d as a good approximation of the unbiased estimator and the index d+ for aggregating the effect sizes across studies as estimate of population effect size. The unbiased effect size indexes d and d+ are commonly reported in meta-analytic studies. • Standardized Mean Difference Effect Size • large effect: d = .80 • medium effect: d = .50 • small effect: d = .20

23. Evidence-Based Practice: Implications for Rehab Counseling • Implications for counseling research – The need for level 1 evidence (empirical supported treatment and meta-analysis) related to the effectiveness of rehabilitation interventions (individual ingredients/components of RC as well as RC as interventions) • Implications for practitioners – For practitioners, evidence-based practice is a research utilization issue (ability to judge the quality of an individual research study and a collection of studies, the ability to select the best interventions on an individualized basis, and the ability to search for research information using the Internet and other library tools)

24. Resources for Evidence-Based Practice • Agency for Healthcare Policy and Research (AHRQ) • Title IX of Public Health Services Act • Healthcare Research and Quality Act of 1999 • Primary agenda is to research health outcomes, develop effective outcome measures & evaluate overall quality of care. • Patient Outcomes Research Teams (PORTs) • Clinical Practice Guidelines • Clinical performance measures • Report Cards

25. PORTs • Designed to determine the most effective treatment and pattern of care for a specified clinical area through lit reviews & meta-analysis. • Partnership with National Institute of Health. • Some PORTs that are relevant to rehab: • Effectiveness of alternative treatments for type II diabetes & mental health problems in the Mexican-American population (funded in part by UT Health Science Center) • Secondary prevention of stroke, hip fx repair, osteoarthritis, TKR, back pain tx and assessment.

26. Clinical Practice Guidelines • Frequently developed from PORT findings • Offer healthcare providers well-founded, cost-effective treatment methods for various clinical conditions. • PORT study on Stroke Rehabilitation led to the clinical practice guideline “Post-stroke Rehabilitation” (AHRQ, 2000). • Development & use of clinical practice guidelines has become so well accepted that AHRQ no longer sponsors development since many public & private entities are doing this independently. • National Guideline Clearinghouse (www.guideline.gov)

27. Clinical Performance Measures • Used to assess services of a healthcare professional by reviewing • appropriateness of service • timeliness of service, and • access to service • Measures are detailed and condition specific • Attempts to measure quality of care. • Currently there are approximately 1,200 performance measures • Are grouped by • Disease/condition • Treatment/intervention • Domain • Organization • Available at the National Quality Measures Clearinghouse (www.qualitymeasures.ahrq.gov)

28. Clinical Performance Measures • Measures developed by: • Joint Commission on Accreditation of Healthcare Organizations • Health Plan Employers Data & Information Set • Healthcare Cost & Utilization Project • Outcome & Assessment Information Set • The University of Wisconsin Nursing Home quality indicators. • VA external review program

29. Report Cards • Information obtained through these Clinical Performance Measures are frequently used for marketing and other communication purpose to consumers and other purchasers through “Report Cards” • Report Cards are a response to the consumer-driven healthcare system and provides consumers with information concerning the care recommended by clinical practice guidelines, outcomes expected under specific situations and the wide range of performance measures used to evaluate the quality different aspects of care offered in a user-friendly format.

30. Other resources • American Congress of Rehabilitation Medicine www.acrm.org/Resources/Evidence-BasedResources.htm

31. Research Issues • Rehabilitation research has identified important rehab counseling/case mgt functions and knowledge domains, BUT… • As of this time, no empirical evidence has been generated that supports the effectiveness of these functions or knowledge domains on rehab outcomes.

32. Research Issues • It is possible to infer some evidence using research from allied disciplines (e.g., counseling and clinical psychology) • For example: the process variable of “working alliance” has gained overwhelmingly strong empirical evidence as a primary influence on counseling outcomes (Wampold, 2001).

33. Research Issues • There is some evidence for effectiveness of counseling factors in rehab counseling • Bolton and Akridge’s (1995) meta-analysis of 15 experimental evaluations of 10 skills training interventions (e.g., social skills, stress mgt., problem-solving skills, & career decision-making skils) across some 61 outcome measures. • Estimated true effect size of +.93 indicated skills training services substantially benefit the typical VR client.

34. Practice Issues • Rehab Counselors/case managers need to aware of contemporary research in rehab as well as in allied fields. • This will promote evidence-based practice and insure people with disabilities and chronic illnesses receive effective services.

35. Practice Issues • RC Master’s education curricula has only one class in research methods • Probably the most unpopular course and unfortunately translates to negative attitudes towards using research. • Other problems to research utilization: • Practicing counselors generally don’t believe research results can translate into their work • Lack of time on the job • High cost of continuing education • Weak research analysis skills

36. Practice Issues • How to correct these problems: • Rethink curricula regarding research • Need for more creative activities that promote working knowledge of manual and computer searches • Solid understanding of different research designs and issues related to power analysis, effect size and meta-analysi. • Regular in-service training within rehab work environments

37. Summary • Rehab is one of many health-related disciplines facing demands of a managed-care system. • Professionals in the social and behavioral sciences are having to rely on credible evidence to justify their interventions and maintain their identity in a climate of healthcare outcomes.

38. Summary • Evidence-based practice and empirically validated interventions dominate the healthcare research and are most likely here to stay. • How well rehabilitation researchers and practitioners work together to respond to the demands of this evidence-based climate is crucial for the profession’s viability.

39. Summary • Rehabilitation researchers need to conduct more experimental studies to validate the effectiveness of specific rehab counseling/case management interventions as well as to determine the overall effectiveness of rehab counseling and case management. • Rehabilitation educators need to focus on improving the quality and effectiveness of courses in research methods to facilitate evidence-based practitioners.

40. Summary • Rehabilitation students need to be able to understand a wide array of research designs and methodologies as well as access and critically evaluate research from rehab and related literature. • Work settings need to provide pre-service and in-service training regarding evidence-based practice and incorporate this type of service delivery into their job descriptions.

41. Summary • Evidence-based practice is the standard of practice in healthcare. The extent to which rehabilitation counselors/case managers, researchers and educators prepare for and accept EBP will dictate the degree to which rehabilitation can be performed correctly and adequately by the professionals providing the service. (Chronister et al., 2005)