Critically Examining the Evidence

1. Critically Examining the Evidence Andrea M. Landis, PhD, RN UW LEAH December 7, 2012

2. Learning Objectives • Discuss the importance of critically examining the literature. • Review key issues for evaluating the literature. • Explore in detail each section of a manuscript. • Appraise systematic review article. Sources: “How to read a paper” (2010) Greenhalgh, T. [ebook] Centre for Evidence Based Medicine (Oxford) UW Health Sciences Library - Toolkit

3. Before any research project review the literature relevant to your RQ: • Gain a full and in-depth understanding of a subject. • See if your intended research subject has been done before and avoid duplication. • Avoid any errors made in similar research. • Enable you to place your study within its context (ie so that you can show how your research will add to the existing sum of knowledge). • Provide you with ideas to help you define or amend your own research topic. • Provide you with information with which to compare and contrast your findings.

4. Introduction • One’s ability to review articles is something that requires practice and experience. • Process can be aided by following a checklist of things to look out for and comparing the paper under review to the criteria. • The type of research will effect the information you are evaluating. • Standard journal format subheadings for research reports: title, abstract, introduction, method, results, discussion and conclusion.

5. Summary of key issues that need to be evaluated (APA, 1983): • Is the RQ question significant and is the work original and important? • Have the instruments used been demonstrated as reliable and valid? • Do the outcome measures relate clearly to the variables with which the investigation is concerned? • Does the research design fully test the hypothesis? • Are the subjects representative of the population to which generalizations are made? • Did the researcher observe ethical rules? • Has the research reached such a stage that publication is justified and the results are meaningful?

6. Title • Very important. • Short and informative. • Gives insight into what (was done), whom (it was done to) and how (it was done). • Gimmicky?

7. Author • Some idea of the author(s) academic background and ability to carry out valid research • job title, qualifications, and where they work. • Corresponding author

8. Abstract • The abstract should contain a brief statement about the study's purpose, method, results, conclusion and clinical relevance. • Time-efficient way for readers to determine if the article suits their needs – return to the article later. • Do not to accept the conclusions before critically reading the entire article.

9. Source • Assessment of where the article was published should give some clues as to its potential value. • Is it a peer reviewed journal?

10. Introduction & Statement of the Problem • Research problem/clinical question should be defined clearly. • Expect to find clear descriptions of the research aims, an outline of theoretical issues and the hypothesis should be introduced. • Information should include the current state of knowledge about the research topic and an indication of the gaps in knowledge which the current study will hope to fill. • “Why was the research done?” • Statement of the Problem: should describe the questions and concerns that led the author to undertake the investigation. • “What question did the author try to answer?”

11. Literature Review → Purpose Statement • The literature review should establish a theoretical and historical basis • Survey of current knowledge highlighted by a thorough review of the existing literature. • Identify gap of knowledge between what is known (or previously documented) and what is desired to be known. Find information in the literature that supports the concept and approach of the study. • Attempt close the gap by explaining why the study was conducted. Or point out flaws, inconsistencies or areas where no conclusions can be drawn. • Issues: • Up to date references • Unbiased • The purpose of the study should be described in a direct, clear statement.

12. Methodology • Clearly explain how the study was conducted. • Critical readers should pretend they are going to replicate the study: • Is there sufficient detail in the method to conduct the study and obtain similar results? • Was the design of the study sensible? • Method can be divided into the following subsections: subjects, instrumentation and apparatus, procedure, and data analysis.

13. Subjects • Summarize and describe the subjects who participated in the study in terms of age, sex, diagnosis and other pertinent demographic characteristics (Table). • Inclusion/Exclusion criteria. • How were they recruited? • The extent to which readers are able to use the results (generalization) of the study depends on how the sample of subjects was selected: • Randomly selected • Sample of convenience • RCT - allocation of subjects to experimental and control groups • How many subjects were included in the sample: • Small vs. Large • Impact on power analysis • Diversity (race/ethnicity reported) • IRB approval (includes confidentiality and anonymity assured)

14. Instruments or Apparatus • Any special equipment or instruments (e.g. questionnaires) should be described. • Indicate validity and reliability. • Described in such a way that readers can replicate the study. • Specify model numbers, corporate names and addresses, and other pertinent details about the instruments. • Any apparatus (or questionnaire) designed and developed by the researcher should be fully described with a drawing, photograph and description. • Were the instruments calibrated? How were they calibrated? Are they repeatable day-to-day?

15. Procedures • How and when the steps of the study were applied. • How the data were collected. • Internal validity - changes noted during the study are the result of the devices being studied and not the result of a sloppy procedure. • Did the experimental treatment cause the observed change in the dependent variable? • Could other (extraneous) factors be responsible for that change? • Has the investigator taken steps to improve their internal validity or control sources of secondary variance.

16. Data Analysis • Section should describe all testing applied to the data • NO results. • Did authors chose the appropriate statistical tests for the type of study and design. • When analyzing data, arithmetic operations too frequently are misapplied to data based on levels of measurement. • Common error - analyzing ordinal data as though they were quantitative (interval or ratio). • Did authors screen data for errors in data entry, outliers and distribution. • Conventional parametric statistical analyses are conducted on continuous data. • descriptive, comparative, associative and predictive.

17. Results • Reports what has been discovered. • Reported factually and formally without commentary. • Summary statistics may be presented in tables/figures. • Statistical tests and measures used should be described allowing the reader to evaluate whether the appropriate tests were applied. • Level of significance • Statistical versus clinical significance • Do the authors mention all the relevant results? - even those that actually go against the hypothesis.

18. Discussion • The issues raised by the findings should be discussed and resolved in this section. • Should relate back to the literature/aims of the research as outlined in the introduction. • The author is expected to examine, interpret and qualify the results and draw any inferences from them. • Large enough/long enough/followed-up enough to be credible • Is the research question answered? Has the author given meaning to the results? • Has the author considered broader implications of his/her findings?

19. Conclusion & Recommendation • Brief restatement of the experimental results and describes the implications of the study. • The paper should end with some conclusions about the importance (or otherwise) of the findings. • The author should not make any statements here which are not supported by the facts found. • Recommendations on the basis of the findings are often stated here. • comments on possible improvements • future areas for more study.

20. References • Consistent citing of references. • Endnote, RefWorks • Appropriately extensive and up-to-date.

21. Questions • What question did the systematic review address? • Is it unlikely that important, relevant studies were missed? • Were the criteria used to select articles for inclusion appropriate? • Were the included studies sufficiently valid for the type of question asked? • Were the results similar from study to study? • What were the results? How are the results presented?