From Scientific Research to Clinical Practice Rome, November 25, 2011

1. Local clinical recommendation: the GREFO programNicola Magrini, Lucia MagnanoNHS CeVEASNHS Centre for the Evaluation of the Effectivenessof Health Care, Modena, Italy From Scientific Research to Clinical Practice Rome, November 25, 2011

2. Contents of the presentation • New cancer drugs recommendations • Why GRADE could help … • A three pillar method: the GRADE system to evaluate quality of evidence and define the strength of a recommendation • An example of grade application: the GREFO program • Conclusions: what we have learnt about how best to formulate a weak recommendation

3. New Cancer Drugs approval:why bother? • New cancer drugs are often approved with limited evidence (only 1 or 2 RCTs in selected populations with limited follow-up, …) • The benefits are often limited (few weeks of increased survival or progression-free survival) and small if compared to clinical expectations • Their cost is very high and not proportional to actual benefits

4. Emilia Romagna New Cancer Drugs recommendations using GRADE • Startedas a research project with the aimofapplying a more transparentgrading system (GRADE) tocancerdrugsrecommendations (2006 – 2007) • GRADE provedfeasible and usefultouseas a common frameworkwith the oncologistsresearch community • though the timerequestedfor the production ofeachrecommendationwasquite long (12 months) • the recommendationsdidnothaveanindicatorofexpecteduse • The research project wasthentransformedintorapidrecommendations (3-4 months) withexplicitindicatorsofexpectedusetoincreasetheirapplicability

5. Emilia Romagna New CancerDrugslocalRecommendations Panel COR Commissione Oncologica Regionale CRF Commissione Regionale del Farmaco Drugs with a not stated benefit/risk ratio (it is necessary an assessment by a panel and formulation of recommendations) Drugs NOT included in the regional formulary by CRF Drugs included in the regional formulary by CRF GREFO Gruppo Regionale Farmaci Oncologici 3 months recommendations

6. JCO editorial comment • In applying GRADE, the most common recommendation categories one might expect in the oncology field are “Probably use it” or “Probably do not use it” . (i.e. weak recommendations) • […] • Just how one interprets these recommendation categories clinically is not known. Brouwers MC, Somerfield MR; Browman GP A for Effort: Learning for the application of the GRADE approach to cancer guideline development. JCO 2008

7. EvidenceRecommendation B Class I C+ 1 IV C Organization AHA ACCP SIGN WhyGRADE?Too many systems?… probably yes Recommendation for use of oral anticoagulation in patients with atrial fibrillation and rheumatic mitral valve disease

8. Why using GRADE GRADE is much more than a rating system • offers a transparent and structured process for developing and presenting summaries of quality of evidence • provides guideline developers with a comprehensive and transparent framework for carrying out the steps involved in developing recommendations • specifies an approach to framing questions, choosing outcomes of interest and rating their importance, evaluating the evidence, and incorporating evidence with considerations of values and preferences of patients and society to arrive at recommendations

9. Facts versus Decisions(Quality of Evidence versus Strength of Recommendations) Very low, Low, Moderate, or High Quality Evidence Facts “Evidence” Values Feasibility Costs Decision Weak or Strong Recommendation (for or against) Paul Glasziou, 2007

10. GRADE Uptake

11. WHO guideline development processesupdate 2010

12. Reportingstandard and process Standards for evidence: GRADE system Title, responsible person, WHO Department - responsible of the clearance process, WHO Departments involved, CC involved, 1. Scoping the document: reasons for choosing the topic, problems with existing guidelines, variations and gaps, 2. Group composition (or consultations) 3. Conflict of interest 4. Formulations of the questions and choice of the relevant outcomes 5. Evidence retrieval, evaluation and synthesis (balance sheet, evidence table) 6. Benefit/risk profile: integrating evidence with values and preferences, equity and costs 7. Formulation of the recommendations 8. Implementation and evaluation of impact Reportingstandard and process 9. Research needs or areas of further research 10.Peer-review process and updating

13. Contents of the presentation • New cancer drugs recommendations • Why GRADE could help … • A three pillar method: the GRADE system to evaluate quality of evidence and define the strength of a recommendation • An example of grade application: the GREFO program • Conclusions: what we have learnt about how best to formulate a weak recommendation

14. GRADE: a 3 pillars approach • Formulate the question, choose and rate your outcomes of interest and perform a systematic review (quality of evidence) • Risk benefit evaluation, consider patients values and preferences and also resource use and feasibility • Strength of the recommendation

15. Formulate the question • Specify your question in an answerable way • A classic example (WHO, 2006): Should oseltamivir be used for treatment of patients hospitalised with avian influenza (H5N1)? • An example relevant to GPs:What is the preferred drug of choice for the treatment of hypertension (in the elderly)? • And the preferred associations of drugs? • New cancer drugs (2009): In patients with NSLC histology non squamous type, is a first line therapy with pemetrexed plus cisplatin recommended?

16. GRADE - Outcomes choice • explicit judgements should be made about which outcomes are critical, which ones are important but not critical, and which ones are unimportant and can be ignored. • studies using surrogate outcomes generally provide weaker evidence than those using outcomes that are important, and these only should be included when evidence for important outcomes is lacking. • Outcomes should be discussed and rated indipendently form the fact of being considered in the available studies Schünemann HG et al. Health Res Policy Syst 2006:4:18

17. Choice and rating ofimportantoutcomes… exampleforptswithbedsores

18. Study design is important - Early systems of grading the quality of evidence focused almost exclusively on study design - Randomised trials provide, in general, stronger evidence than observational studies: • RCTs start at High Quality • Observational studies start at Low Quality - However, other factors may decrease or increase the quality of evidence

19. Quality assessment criteria: the big move

20. Factors that may decrease the quality of evidence • Study limitations(risk of bias) • Inconsistencyamong studies • Indirectnessof evidence • Imprecise results • Reporting bias / publication bias

21. Risk of Bias (study limitations) well established concealment intention to treat principle observed blinding completeness of follow-up Choice of comparator (standard/optimal treatment more recent early stopping for benefit selective outcome reporting bias

22. Factors that may decrease the quality of evidence • Here, I would like to stress the 2 new criteria explicitly introduced by GRADE: • Inconsistencyamong study results … to be applied also when just one clinical trial is available • Indirecteness / transferability / applicability problems: populations, interventions, comparisons, outcomes • These two criteria are particularly relevant to general practice, in particular indirectness … both could be used to identify evidence gaps and to support more and more relevant indipendent research

23. Are these results consistent? Prendiville WJ et al. Cochrane Database Syst Rev2000, Issue 3

24. Quality of evidence Degree of confidence that an estimate of effect or an association is correct

25. Contents of the presentation • New cancer drugs recommendations • Why GRADE could help … • A three pillar method: the GRADE system to evaluate quality of evidence and define the strength of a recommendation • An example of grade application: the GREFO program • Conclusions: what we have learnt about how best to formulate a weak recommendation

26. Specify your question in an answerable way (PICO) • Clinical Question • In patients with NSCLC histology non squamous type, is a first line therapy with pemetrexed plus cisplatin recommended? • Young/notelderlypersons (<70yr) who are fit (PS=0-1) • Elderlypersonswho are (>70yr) who are fit (PS=0-1) • Personswho are unfit (PS>2)

27. 1.Outcomes selection and rating Beneficial outcomes Risk outcomes Risk releted to bevacizumab outcomes

28. Outcomes selection and rating

29. Studies retrieved for pemetrexed

30. Pemetrexed – table of evidence

31. Quality of evidence indirecteness RiskofBias Subgroups analysis prespecified, but histologic groups was not considered in the stratification of randomization

32. Pemetrexed – Qualityofevidence Consistency Imprecision Choice of comparator Publicationbias

33. Pemetrexed: Quality of evidence by subgroups • Young/not elderly persons (<70yr) who are fit (PS=0-1) Downgrading (-1 ) forriskofbias Quality: moderate

34. Pemetrexed: Quality of evidence by subgroups 2. Elderly persons who are (>70yr) who are fit (PS=0-1) • Downgrading (-2) for • - risk of bias • Indirectness • Quality: Low

35. Pemetrexed: Quality of evidence by subgroups 3. Persons who are unfit (PS>2) No studies No quality assessment

36. GRADE: a 3 pillars approach • Formulate the question, choose and rate your outcomes of interest and perform a systematic review (quality of evidence) • Risk benefit evaluation, consider patients values and preferences and also resource use and feasibility • Strength of the recommendation

37. GRADE Determining the benefit risk profile: positive/uncertain/unfavourable

38. Pemetrexed: benefit/risk profile Patients <70 yr Elderly patients >70 yr Unfit patients positive uncertain unfavourable

39. GRADE: a 3 pillars approach • Formulate the question, choose and rate your outcomes of interest and perform a systematic review (quality of evidence) • Risk benefit evaluation, consider patients values and preferences and also resource use and feasibility • Strength of the recommendation

40. Strength of recommendation The degree of confidence that the desirable effects of adherence to a recommendation outweigh the undesirable effects • Desirable effects • health benefits • less burden • savings • Undesirable effects • harms • more burden • costs

41. Categories of recommendations Although the degree of confidence is a continuum, we suggest using two categories: strong and weak. • Strong recommendation: the panel is confident that the desirable effects of adherence to a recommendation outweigh the undesirable effects. • Weak recommendation: the panel concludes that the desirable effects of adherence to a recommendation probably outweigh the undesirable effects, but is not confident. Recommend   Suggest  

42. Quality of evidence: moderate(Risk of bias for subgroups analysis) Pemetrexed Recommendations Benefit/riskprofile: positiveshort explanationofwhy the Panelsuggestedpositivity Pemetrexed+ cisplatin lung cancer, pts <70 yrs 1st line treatmentweak positive recomm. Between 30-50% of patients treated with a 2 drugs regimen should receive pemetrexed+ cisplatin

43. Examplesofrecommendationsusing GRADE A flexible method

44. WHO Guidelinesfor the PsychosociallyAssistedPharmacological Treatment ofOpioidDependence (2009) For opioid agonist maintenance treatment, most patients should be advised to use methadone in adequate doses in preference to buprenorphine. • Strength of recommendation – Strong • Quality of evidence – High • On average, methadone maintenance doses should be in the range of 60–120 mg per day. • Strength of recommendation – Strong • Quality of evidence – Low

45. Considerations on strong and weak recommendations Life iseasierformethodologists and clinicianswith strong recommendations Weakrecommendations are increasinglyimportant and often cover grey or controversialareas. Theirformulationshouldbeasexplicitaspossible in termsofreal-lifeimplications The potentialrangeofapplicationof a weakrecommendationshouldbeoperationalised (whatdoes “suggest” or “couldbeused” mean?) … tobeapplicable and monitored …

46. Emilia Romagna New Cancer Drugs Rec. GRADE steps and polls • We (in Italy) adapted GRADE by: • Rating/voting the evaluation of the benefit-risk profile (favourable, uncertain, unfavourable) • Defining an expected use indicator for each recommendation • Incorporating the results of polls in the final document Quality of evidence: estimates of benefits & harms and risk of bias, directness, … Risk-benefit profile evaluation for different subgroups Quality assessment criteria Quality assessment criteria Strong / weak recommendation Ratings of outcomes Evidence + expected use

47. Emilia Romagna New Cancer Drugs RecommendationsOverview of recommendation strength 1/2

48. Emilia Romagna New Cancer Drugs RecommendationsOverview of recommendation strength 2/2

49. Implications ofstrong and weak … and expected use