The Appraisal, Extraction and Pooling of Cost and Cost Effectiveness Studies

1. The Appraisal, Extraction and Pooling of Cost and Cost Effectiveness Studies JBI/CSRTP/2012-13/0005

2. Introduction • Recap of Introductory Module • Developing a question (PICO) • Inclusion Criteria • Search Strategy • Selecting Studies for Retrieval • This module considers how to appraise, extract and synthesize evidence fromCost and Cost Effectiveness studies.

3. Aim and Objectives • The objectives of this module are to prepare participants to: • critically appraise studies of cost and cost effectiveness, • extract data from cost and cost effectiveness studies, • summarize the results of cost and cost effectiveness studies.

4. Program Overview

5. Session 1: Introduction to review of evidence on cost and cost effectiveness

6. Common study designs • Prospective experimental or quasi experimental effectiveness studies with cost or cost effectiveness components • Modelling studies

7. Four approaches to analysis • Cost-minimization analysis (CMA); • Cost-effectiveness analysis (CEA); • Cost-utility analysis (CUA); • Cost-benefit analysis (CBA).

8. Methods, measures, benefits

9. Cost-minimization analysis (CMA) In cost-minimization analysis (CMA) only the costs of the interventions are compared; the outcomes are assumed to be equivalent.

10. PICO Questions – Cost Minimization What is the evidence on costs (direct and indirect) of laparoscopic compared to open appendectomy for patients aged 15 years or over (assuming the long-term outcome is the same in both groups)?

11. Cost-effectiveness analysis Costs are measured in monetary units; The outcome is common to both alternatives but the effect size and direction may vary; Outcomes are measured in natural/clinical units; (e.g. mortality, myocardial infarctions, lung function, weight, bleeds).

12. Cost–effectiveness Plane • A four-quadrant figure of cost difference plotted against effect difference: • quadrant I, intervention more effective and more costly than comparator; • quadrant II, intervention more effective and less costly than comparator; • quadrant III, intervention less effective and less costly than comparator; and • quadrant IV, intervention less effective and more costly than comparator. (Culyer, 2005:77-78)

13. Cost Effectiveness Plane Q4 Q1 Q2 Q3

14. PICO Questions – Cost Effectiveness What is the cost effectiveness of percutaneous coronary intervention with drug-eluting stents (PES) compared to bare-metal stents (BMS) to reduce angina symptoms for patients undergoing single-vessel percutaneous coronary intervention?

15. Cost-utility analysis Costs are measured in monetary units; Outcomes are common to both alternatives; Effect size and direction may vary; Outcomes are measured as healthy years (typically measured as quality-adjusted life-years (QALYs)).

16. Outcome measures for CUA The primary outcome for CUA is expressed as quality-adjusted life-years (QALYs); Other generic outcome measures for CUA: Disability-adjusted life-year (DALY); Healthy years equivalent (HYE); Saved-young-life-equivalent. (Drummond et al., 2005:14)

17. PICO Questions – Cost Utility What is the cost-utility of the cochlear implant in adults (age >18 years) with profound bilateral, post-lingual deafness compared with no intervention?

18. Cost-benefit analysis Costs are measured in monetary units; Outcomes are identified as single or multiple effects; The effects are not necessarily common to both alternatives; Outcomes are measured in monetary units.

19. PICO Questions – Cost Benefit What is the cost-benefit of donepezil compared to galantamine for cognitive function in patients with mild to moderate Alzheimer’s disease?

20. Searching for Evidence Cost and Cost Effectiveness keywords Clinical keywords General databases Specific databases

21. Specific economic databases • NHS Economic Evaluation Database (NHS EED) • Health Economic Evaluation Database (HEED) • Cost-effectiveness Analysis (CEA) Registry • Health Technology Assessment (HTA) database • Paediatric Economic Database Evaluation (PEDE) • European Network of Health Economic Evaluation Databases (EURONHEED) • COnnaissance et Decision en Economie de la Sante (CODECS)

22. Group Work 1: Identification of Economic Evaluation Study Designs • Refer to Workbook. • Report back

23. Session 2: Critical Appraisal of Cost and Cost Effectiveness Studies

24. Why Critically Appraise? Combining results of poor quality research may lead to misleading understandings of issues explored 1004 references 172 duplicates 832 references Scanned Ti/Ab 715 do not meet Incl. criteria 117 studies retrieved 82 do not meet Incl. criteria 35 studies for Critical Appraisal

25. The Critical Appraisal Process Every review must set out to use an explicit appraisal process. Essentially, A good understanding of research design is required in appraisers; and The use of an agreed checklist is usual.

26. Critical appraisal of cost and cost effectiveness evidence Primary purpose of critical appraisal is to assess a study’s quality and determine the extent to which a study has excluded the possibility of systematic flaws in its design, conduct and analysis.

27. JBI Critical Appraisal Checklist for cost and cost effectiveness studies Is there a well defined question? Is there a comprehensive description of alternatives? Are all important and relevant costs and outcomes for each alternative identified? Has clinical effectiveness been established? Are costs and outcomes measured accurately? Are costs and outcomes valued credibly?

28. JBI Critical Appraisal Checklist for cost and cost effectiveness studies Are costs and outcomes adjusted for differential timing? Is there an incremental analysis of costs and consequences? Were sensitivity analyses conducted to investigate uncertainty in estimates of cost or consequences? Do study results include all issues of concern to users? Are the results generalizable to the setting of interest in the review?

29. JBI Economic evidence appraisal • Is there a well defined question? • Costs and effects; • Comparison of alternatives; • Perspective of the analysis (including the decision-making context).

30. Effects Mortality measurements; Morbidity measurements; Health-related quality of life measurements.

31. Perspective The ‘viewpoint’ adopted for the purposes of an economic appraisal (cost–effectiveness, cost–utility studies and so on) which defines the scope and character of the costs and benefits to be examined.

32. Perspectives Societal perspective; Health sector perspective; Other sector perspective; Health insurance perspective; Hospital perspective; Patient perspective.

33. Is there a comprehensive description of alternatives? Important alternatives Do-nothing alternative JBI Economic evidence appraisal

34. Are all important and relevant costs and outcomes for each alternative identified? Was the range wide enough for the research question; Does it cover all relevant perspectives; Were capital as well as operating costs included. JBI Economic Evidence Appraisal

35. Typical classification of costs • Functional costs; • Financial and economic costs; • Direct, indirect and intangible costs; • Capital and recurrent costs; • Fixed and variable costs; • Opportunity costs

36. Functional costs Can be classified into categories: personnel; buildings and space; equipment; supplies and pharmaceuticals; transportation; training; information, education and communication.

37. Financial and economic costs Financial costs are defined as the actual money spent on the resources; Inclusion of the costs of all resources, regardless of their financial cost is known as the economic cost.

38. Direct, indirect and intangible Costs • Direct costs are associated directly with a healthcare intervention (e.g. drugs, staffing); • Indirect costs refer to the productivity gains or losses (e.g. time off work, illness); • Intangible costs refer to the non-monetary assets that can not be readily seen (e.g. anxiety, fatigue, pain or suffering from an illness or treatment).

39. JBI Economic Evidence Appraisal • Has clinical effectiveness been established? • Was this through experimental research? • If so did the trial protocol reflect what would happen in regular practice? • Was effectiveness established through a synthesis of clinical studies? • Were observational data or assumptions used to establish effectiveness? • If so what were the potential biases in results?

40. JBI Economic Evidence Appraisal • Are costs and outcomes measured accurately? • Were any of the identified items omitted from the measurement? • If so does this mean that they carried no weight in the subsequent analysis? • Were there any special circumstances (e.g. joint use of resources) that made measurement difficult? • If so, were these circumstances handled appropriately

41. JBI Economic Evidence Appraisal • Are costs and outcomes valued credibly? • Were the sources of all values clearly identified? • Possible sources include market values, patient or client preferences and views, policy maker’s views and health professional’s judgements; • Were market values employed for changes involving resources gained or depleted? • Where market values were absent (e.g. volunteer labour) or did not reflect actual values (such as clinic space donated at a reduced rate) were adjustments made to approximate market values?

42. JBI Economic Evidence Appraisal • Are costs and outcomes adjusted for differential timing? • Were costs and outcomes that occur in the future ‘discounted’ to their present values? • Was there any justification given for the discount rate used?

43. JBI Economic Evidence Appraisal • Is there an incremental analysis of costs and consequences? • Were the additional (incremental) costs generated by one alternative over another compared to the additional effects, benefits or utilities generated?

44. JBI Economic Evidence Appraisal • Were sensitivity analyses conducted to investigate uncertainty in estimates of costs or outcomes? • If a sensitivity analysis was employed, was justification provided for the range of values (or for key study parameters)? • Were the study results sensitive to changes in the values (within the assumed range for sensitivity analysis or within the confidence interval around the ratio of costs to outcomes)?

45. JBI Economic Evidence Appraisal • Do study results include all issues of concern to users? • Are the results of cost and effect for the alternative interventions? • Do they clearly specify the relative size of the effects for the interventions? • Do they clearly show how costs differ for the two interventions? • Can we use them with the Cost Effectiveness Plane?

46. Did the study take account of other important factors in the choice or decision under consideration (e.g. distribution of costs or outcomes or relevant ethical issues)? • Did the study discuss issues of implementation such as the feasibility of adopting the preferred program given existing financial or other constraints and whether any freed resources could be re-deployed to other worthwhile programs?

47. JBI Economic Evidence Appraisal • Are the results generalizable to the setting of interest in the review? • Did the study make clear that the findings on costs and effects were generated in a specific setting using particular assumptions? • Was the generalizability of the results to other settings and patients/client groups discussed?

48. Group Work 2: Critical Appraisal of evidence from economic evaluation studies • Workbook • Report back

49. Session 3: Study data and Data Extraction 

50. Data most frequently extracted 1004 references 172 duplicates 832 references Scanned Ti/Ab 715 do not meet Incl. criteria 117 studies retrieved 82 do not meet Incl. criteria 35 studies for Critical Appraisal 26 studies incl. in review