Introduction to Critical appraisals of the medical literature

1. Introduction toCritical appraisals of the medical literature Partini Pudjiastuti, Sudigdo Sastroasmoro Department of Child Health Medical School University of Indonesia

2. Evidence based medicine 5 steps Formulate question Evaluate performance Efficiently track down best available evidence Implement changes in clinical practice Critically review the validity and usefulness of the evidence

3. Rationale • >25,000 journals worldwide • >2 million published articles per year • Many published articles have methodological (including statistical) flaws – even in most respected journals • Not all results can be applied due to many reasons, a.o. dissimilarities of study subjects with our patients • Limited time of physicians: focus on articles relevant to your clinical practice

4. Errors and inappropriateness may occur in any part of medical research: • in choosing appropriate design • in choosing population • in selecting study subjects • in the details of the design • in intervention and measurements • in use of statistical methods in analysis • in interpreting statistical analysis • and in writing research report

5. The effect of dietary habit on calcium level in pediatric patients with nephrotic syndromeJ Agric Soc Sci 2006;2 • r = - 0.1213, p = 0.015 • There was a significant inverse relationship between the age and the total calcium level ……

6. IMPORTANT!!! Statistical significance vs. clinical importance • Negligible clinical difference may be statistically very significant if the number of subjects >>>. e.g., difference in reduction of cholesterol level of 3 mg/dl, n1=n2 = 10,000; p = 0.00002 • Large clinical difference may be statistically non- significant if the no of subjects <<<, e.g. 30% difference in cure rate, if n1 = n2 = 10, p = 0.74

7. 200 197 Clinical importance vs. statistical significance Cholesterol level, mg/dl Standard treatment 300 mg/dl R n=10000 Clinical n=10000 New treatment 300 mg/dl t = df = 9998 p = 0.00002 Statistical

8. Clinical Statistical Clinical importance vs. statistical significance Cured Died Standard Rx 0 10 (100%) New Rx 3 7 (70%) Absolute risk reduction = 30% Fischer exact test: p = 0.211

9. Critical appraisal (Making Reading More Worthwhile) What is Critical Appraisal? • Critical appraisal = quality assessment • ….process of weighing up evidence to see how useful it is in decision making • .…a process of assessing the validity, reliability and usefulness of evidence • …..is about considering, evaluating and interpreting information in a systematic and objective way

10. Critically Appraise What You Read. • Separating the wheat from the chaff. • Time is limited – you should aim to quickly stop reading the dross. • Others contain useful information mixed with rubbish. • Simple checklists enable the useful information to be identified.

11. Critical Appraisal – Critical Thinking • Appraising the available evidence to construct clinical reasoning and to make decisions • Finding strengths and limitations of written ‘evidence’ • Deciding what evidence to pay attention to versus what to ignore

12. Why critically appraise? • Supports sound decision making based on best available evidence • Helps us determine (three R’s): • How rigorous a piece of research is - Validity • What the results are telling us - Importance • How relevant it is to our patient -Applicability

13. Morbidity Patient Satisfaction Health Status Mortality QoL Quality Value = Cost Resources used

14. What is “Evidence”? • People disagree on what constitutes “evidence” • Evidence – generally = scientific fact • Evidence - a combination of information obtained from 3 sources: research, clinical experience, and client preferences (Kitson, Harvey, & McCormack, 1998)

15. Why do we need evidence? • Resources should be allocated to things that are EFFECTIVE • The only way of judging effectiveness is EVIDENCE • “In God we trust – all others need evidence”

16. Sources of Evidence • Primary sources • Based on experiments and published research • Secondary sources • Systematic reviews • Clinical guidelines • Journals of secondary publication e.g. Evidence Based Medicine

17. Levels of evidence 1. Syst reviews of RCTs and high quality RCTs 2. Syst reviews of cohort studies, lower quality RCTs, outcomes research 3. Syst reviews of case controls, case control studies 4. Case Series 5. Expert opinion www.cebm.net

18. Types of EvidenceQuestion Types

19. Key quality parameters • Validity • Reliability • Importance

20. Validity • Internal Is the study designed in such a way that I Can trust the findings? • External Is the study designed in such a way that I Can generalize the findings?

21. Reliability If the study was conducted again, would the results be the same? Usually interpreted as the accuracy of measurement.

22. Importance What was the effect size or magnitude of effect? Clinical vs. statistical significance.

23. Tools for Critical Appraisal • Are the results valid? • What are the results? • Will the results help me in patient care? EBM “simplified” approach: V I A

24. Format of research reports • Title • Authors and Institutions • Abstract & keywords • IMRAD • Introduction - why did I start? • Methods - what did I do? • Results - what did I find? • Discussion - what does it mean? • References

25. Check list for medical literature • Title • Authors & Institutions • Abstract: Structured? Informative? Abbreviation? • Introduction: Length? Relevant ref? Objective? • Methods: • Design, time and place • Inclusion criteria • Exclusion criteria • Sample size, sampling method • Randomization technique • Intervention: masking? • Outcome measurement: blinding? • Primary outcome: type of variable • Secondary outcome: type of variable • Analysis: Clinical, statistical

26. Check list for medical literature • Results • Baseline characteristics • Main outcome • Secondary outcome • Discussion • General • Strength and weakness • Conclusions • References • Vancouver style • Constant • Acknowledgments • Ethics approval • Conflict of interest

27. What to assess?(in study of cause-effect relationship) • General description • Type of design • Target population, source population, sample • Sampling method • Dependent and independent variables • Main results?

28. What to assess?(in study of cause-effect relationship) B. Internal validity, non-causal relationship • Influence of bias • Influence of chance • Influence of confounders

29. Bias What is a bias?A process that tends to produce results that depart systematically from the true values existing in the study population Types of bias 1. Sample (subject selection) biases, which may result in the subjects in the sample being unrepresentative of the population which you are interested in 2. Measurement (detection) biases, which include issues related to how the outcome of interest was measured 3. Intervention (performance) biases, which involve how the treatment itself was carried out.

30. What to assess?(in study of cause-effect relationship) C. Internal validity, causal relationship • Temporality (cause precedes effect) • Strength of association (large difference, RR, OR, etc) or small p value or narrow confidence interval • Biological gradient (dose dependence) • Consistency among studies (diff. populations or designs) • Specificity (certain factor results in certain effect) • Coherence (does not conflict with current knowledge) • Biological plausibility: can be explained with current knowledge (at least in part)

31. What to assess?(in study of cause-effect relationship) D. External validity • Applicable to study subjects • Applicable to source population • Applicable to target population

32. 11 items, each with 3 sections • Can you find this information in the paper? • Is the way this was done a problem? • Does this problem threaten the validity of the study?

33. 11 items 1. What is the research question? 2. What is the study type? 3. What are the outcome factors and how are they measured? 4. What are the study factors and how are they measured? 5. What important confounders are considered? 6. What are the sampling frame and sampling method? 7. In an experimental study, how were the subjects assigned to groups? In a longitudinal study, how many reached final follow-up? In a case control study, are the controls appropriate? (Etc) 8. Are statistical tests considered? 9. Are the results clinically/socially significant? 10. Is the study ethical? 11. What conclusions did the authors reach about the question?

34. 1. What is the research question? • (Is the way this was done a problem?) • Is it concerned with the impact of an intervention, causality or determining the magnitude of a health problem? • (Does this problem threaten the validity of the study?) • Is it a well stated research question/hypothesis?

35. 2. What is the study type? • (Is the way this was done a problem?) • Is the study type appropriate to the research question? • (Does this problem threaten the validity of the study?) • If not, how useful are the results produced by this type of study?

36. 3. What are the outcome factors and how are they measured? • (Is the way this was done a problem?) • a) are all relevant outcomes assessed • b) is there measurement error? • (Does this problem threaten the validity of the study?) • a) how important are omitted outcomes • b) is measurement error an important source of bias?

37. 4. What are the study factors and how are the measured? • (Is the way this was done a problem?) • Is there measurement error? • (Does this problem threaten the validity of the study?) • Is measurement error an important source of bias?

38. 5. What important potential confounders are considered? • (Is the way this was done a problem?) • Are potential confounders examined and controlled for? • (Does this problem threaten the validity of the study?) • Is confounding an important source of bias?

39. 6. What are the sampling frame and sampling method? • (Is the way this was done a problem?) • Is there selection bias? • (Does this problem threaten the validity of the study?) • Does this threaten the external validity of the study?

40. 7. Questions of internal validity • (Is the way this was done a problem?) • In an experimental study, how were the subjects assigned to groups? • In a longitudinal study, how many reached follow-up? • In a case control study, are the controls appropriate? • of the study?

41. 8. Are statistical tests considered? • (Is the way this was done a problem?) • Were the tests appropriate for the data? • Are confidence intervals given? • Is the power given if a null result? • In a trial, are results presented as absolute risk reduction as well as relative risk reduction? • How useful are the results?

42. 9. Are the results clinically/socially significant? • (Is the way this was done a problem?) • Was the sample size adequate to detect a clinically/socially significant result? • Are the results presented in a way to help in health policy decisions? • (Does this problem threaten the validity of the study?) • Is the study useful?

43. 10. Are ethical issues considered? • (Is the way this was done a problem?) • Does the paper indicate ethics approval? • Can you identify potential ethical issues? • (Does this problem threaten the validity of the study?) • Are the results or their application compromised?

44. 11. What conclusions did the authors reach about the study question? • (Is the way this was done a problem?) • Do the results apply to the population in which you are interested? • (Does this problem threaten the validity of the study?) • Will you use the results of the study?

45. Appraisal Tools • Tools from the Critical Appraisal Skills Programme (CASP) • Systematic Reviews • Randomised Controlled Trials • Qualitative Research Studies • Cohort Studies • Case-Control Studies • Diagnostic Test Studies • Economic Evaluation Studies Available athttp://www.phru.nhs.uk/casp/critical_appraisal_tools.htm

46. Effectiveness of Therapy Risk Factors / Prognosis Diagnosis Attitudes & Beliefs Randomised Controlled Trial Cohort Study Survey using Gold Standard Qualitative (Interviews, Observations, etc) Study Designs Recap

47. Critical appraisal • Methods • Results • Discussion - Valid - Important - Applicable

48. Sekuens dan hubungan subyek-predikat • 50 massa pemuda dan mahasiswa berunjuk rasa memprotes pelecehan seksual yang makin marak di depan Mabes Polri • Menyadari pentingnya Panduan Pelayanan Medis (PPM), dibentuklah Panitia Penyusunan SPM di RSCM

49. Dalam pertemuan ilmiah yang diselenggarakan setiap tahun yang merupakan ajang untuk menyajikan perkembangan mutakhir dalam bidang ilmu penyakit dalam di tanah air tersebut menyimpulkan bahwa pertemuan tersebut disamping dilakukan oleh Universitas selayaknya juga dilakukan oleh cabang-cabang PAPDI di setiap propinsi, bahkan kalau mungkin disetiap kabupaten

50. Plagiarisme adalah tindakan yang dapat diartikan sebagai pencurian ide atau hasil pemikiran dan tulisan orang lain yang digunakan dalam tulisan seolah-olah ide atau tulisan orang lain tersebut adalah ide atau hasil tulisannya sendiri untuk keuntungannya sendiri sehingga merugikan orang lain baik materiil maupun non-materiil, atau plagiarisme dapat berupa pencurian sebuah kata, frase, kalimat, atau alinea, atau bahkan pencurian suatu bab dari sebuah tulisan atau buku seseorang, tanpa menyebut sumber yang dicuri. (Draft SK Rektor UI)