REPORTING STATISTICAL METHODOLOGY AND ANALYSIS IN MEDICAL RESEARCH

1. REPORTING STATISTICALMETHODOLOGY ANDANALYSIS IN MEDICAL RESEARCH INAS ELATTAR Professor of Biostatistics National Cancer Institute Cairo University

2. INTRODUCTION • 1. Provide the audience with a concise reference on what should be reported. • 2. Advice about important aspects of statistical design, analysis, and presentation.

3. MATERIALS AND METHODS • The method section should include everything relating to the study design, subjects and statistical methods used

4. Clearly describe the design of the study • Type of study: Depends on the purpose of the study and the research question to be answered • Important for understanding the conclusion that can be drawn from the study

5. Give numbers of observations • Sample size needed should be stated: useful to include a justification for the number of subjects studied • Basic observational units should be determined • Proper method of analysis depends on the basic observational units. The analysis of correlated observations raises difficult statistical analysis issues

6. Discuss eligibility of experimental subjects • Reasons for and methods of selecting patients • Some studies have eligibility criteria, eg: exclusion of patients outside a specified age range, those previously treated, and those too ill to answer questions, etc… • Two major goals for reporting detailed ways of selection: 1) can be repeated by other investigators, 2) provides readers with a solid link between patients studied and population • Eligibility constraints can introduce a sizable bias when results are generalized to the population

7. Give details about randomization • Randomization enhances generalization of results and avoids biases • Not sufficient to state that: ”subjects were randomly selected,” ”random” often means “haphazard”

8. Describe the methods for, and success of, any blinding of observations • In studies involving comparisons between two treatments, the most effective way to reduce bias on the part of both patient and physician is blinding • Blinding is the hiding of certain information from patients and physicians • Because blinding can be applied in many different ways, researchers should state exactly “who was blinded to what”

9. Report treatment complications • Nearly every medical treatment carries some risk of complications, side – effects • If no adverse effects can be found, the report should say so and explain what was done to find them

10. Report losses to observation (dropouts from a clinical trial) • Patients lost to follow-up, including losses or exclusions for noncompliance • These patients might be different from the others with respect to factors that might affect the results • Efforts to trace them should attempted • Reasons for losses should be reported in details • When sample size in the text, table or graph differs from the original sample size, the difference should be explained

11. Describe the statistical methods with enough details to enable a knowledgeable reader with access to the original data to verify the reported results

12. Use of statistics in medicine can be summarized as follows: • Statistics used to answer questions concerning differences, • Statistics used to answer questions concerning associations, • Statistics used to answer questions concerning predictions

13. Researchers should always have a clear definition of their goal. The researchers have to choose between different measure and techniques: • Mean or median? • Normal or nonparametricdistribution • Adjustments, matching or stratifications

14. Choice of statistical method depends on the question raised and type of data collected • Search for results that are “significant,” this leads to conclusions that may not hold if study is repeated, this called “Fishing expedition”

15. Authors should always report the statistical methods used • Authors should report positives and negatives of their study design • Most statistical techniques depend on some assumptions, authors should indicate that these assumptions have been checked

16. References for study design and statistical methods should be standard works when possible rather than papers where designs or methods were originally reported • In statistics, an original paper is usually very technical and it can be of great help to methodologists

17. Specify any general – use computer programs used • Computer programs are sometimes found to have errors. • Readers might wish to know about the program used for their own use

18. RESULTS • The results section entails a detailed presentation of the analyses carried out and the outcome of the statistical analysis used to answer the research question

19. Put general descriptions of statistical methods in the methods section. When data are summarized in the Results section,specify the statistical methods used to analyze them • Whenever more than one procedure is used, specify the statistical methods applied in the Results Section

20. Clearly specify the units of data • Units should always be specified in text, tables, and figures. Is time months or years? Is quantities kilograms or grams ? Are rates per 10,000 or 100,000? • Careful reporting of units prevents misunderstanding of the results

21. When possible, quantify findings and present them with appropriate indicators of error or uncertainty • Authors should use the most appropriate measure that describes precisely their data, such as means and standard deviations as well as confidence intervals

22. Reporting test of significance (p-values) only, should be avoided • Use standard deviation rather than standard error of mean. There are two reasons for this recommendations: 1)standard error is a function of sample size, 2)standard error pertains to groups, not individuals • Exact P values should be reported • Specify two-tail or one – tail tests

23. Avoid sole reliance on statistical hypothesis testing such as the use of P values which fails to convey important quantitative information • Report both point estimates (means, proportions,or differences between means) and condifence intervals • Confidence intervals provide information on the magnitude of the effect and how estimates would vary in other samples

24. Restrict tables and figures to those needed to explain the argument of the paper and to assess its support. Use graphs as an alternative to tables with many entries; do not duplicate data in graphs and tables

25. Avoid nontechnical uses of technical terms in statistics, such as “random” (which implies randomized device), “normal,” “significant,” “correlation,” and “sample” • “Normal” refers to some kind of probability distribution, should not be confused with meaning of normal patient, that is free of disease • “Significant” refers to the result of a formal statistical test of significance • “Correlation” is a technical word which refers to a specific method to measure association

26. Define statistical terms, abbreviations, and most symbols

27. DISCUSSION • Clearly point out drawbacks of the study • Do not overemphasize the value of P values. Test statistics depends on the sample size • Clearly identify tests of hypotheses generated from the research question of the study and those defined after initiation of the study • Hypotheses generated after beginning of the study are exploratory. Should be used to answer future research questions

28. CONCLUSIONS • Purpose of statistical methods is to provide a straightforward factual account of the scientific evidence derived from research • To design suitable studies and carry out sensible statistical analyses is of utmost importance for the assessment of results • Communication of findings in a clear and objective manner is as crucial as design and analyses

29. Reading the LiteratureChecklist

30. Reading the Abstract • Is the topic of the study important and worth knowing about? • What is the purpose of the study? Is the focus on a difference or a relationship? The purpose should be clearly stated; one should not have to guess. • What is the main outcome from the study: Does the outcome describe something measured on a numerical scale of something counted on a categorical scale? The outcome should be clearly stated • Is the population of patients relevant to your practice – can you use these results in the care of your patients? The population in the study affects whether or not the results can generalized. • If statistically significant, do the results have clinical significance as well?

31. Reading the Introduction If the article does not contain an abstract, the introduction section should include all of the above information plus the following information • What research has already been done on this topic and what outcomes were reported? The study should add new information

32. Reading the Methods • Is the appropriate study design used (clinical trial, cohort, case-control, cross-sectional, meta-analysis)? • Does the study cover an adequate period of time? Is the follow-up period long enough? • Are the criteria for inclusion and exclusion of subjects clear? How do these criteria limit the applicability of the conclusions? The criteria also affect whether or not the results can be generalized. • Are standard measures used? Is a reference to any unusual measurement/procedure given if needed? Are the measures reliable/replicable?

33. Reading the Methods (cont) • What other outcomes (or dependent variables) and risk factors (or independent variables) are in the study? Are they clearly defined? • Are statistical methods outlined? Are they appropriate? (The first question is easy to check; the second may be more difficult to answer.) • Is there a statement about power – the number of patients that are needed to find the desired outcome? A statement about sample size is essential in a negative study

34. Reading the Methods (cont)Clinical trial • How are the subjects recruited? • Are the subjects randomly assigned for the study groups? If not: • How are patients selected for the study to avoid selection bias? • If historical controls are used, are methods and criteria the same for the experimental group; are cases and controls compared on prognostic factors? • Is there a control group? If so, is it a good one? • Are appropriate therapies included? • Is the study blind? Double-blind? If not, should it be? • How is compliance assured/evaluated? • If some cases are censored, is a survival method such as Kaplan-Meier or the Cox model used?

35. Reading the Methods (cont)Cohort study • How are the subjects recruited? • Are the subjects randomly selected from an eligible pool? • How rigorously are subjects followed? How many dropouts does the study have and who are they? • Are appropriate therapies included? • If some cases are censored, is a survival method such as Kaplan-Meier or the Cox model used?

36. Reading the Methods (cont)Case-control study • Are the subjects randomly selected from an eligible pool? • Is the control group a good one (bias-free)? • Are records reviewed independently by more than one person (thereby increasing the reliability of data)?

37. Reading the Methods (cont)Cross-sectional (survey, epidemiologic) study • Are the questions unbiased? • Are the subjects randomly selected from an eligible pool? • What is the response rate?

38. Reading the Methods (cont)Meta-analysis • How is the literature search conducted? • Are the criteria for inclusion and exclusion of studies clearly stated? • Is an effort made to reduce publication bias (because negative studies are less likely to be published)? • Is there information on how many studies are needed to change the conclusion?

39. Reading the Results • Do the reported findings answer the research questions? • Are actual values reported – means, standard deviations, proportions – so that the magnitude of differences can be judged by the reader? • Are many P values reported, thus increasing the chance that some findings are bogus? • Are groups similar on baseline measures? If not, how did investigators deal with these differences (confounding factors)?

40. Reading the Results (cont) • Are the graphs and tables, and their legends easy to read and understand? • If the topic is a diagnostic procedure, is information on both sensitivity and specificity (false positive rate) given? If predictive values are given, is the dependence on prevalence emphasized?

41. Reading the Conclusion and Discussion • Are the research questions adequately discussed? • Are the conclusions justified? Do the authors extrapolate more than they should, for example, beyond the length of time subjects were studied or to populations not included in the study? • Are the conclusions of the study discussed in the context of other relevant research? • Are limitations of the research addressed?