Critiquing Quantitative Research

1. Critiquing Quantitative Research Cynthia L. Russell, PhD, APRN, BC There is no conflict of interest in the creation of this education program.

2. Learner Objectives 1. Describe 2 reasons for evaluating quantitative research. 2. Identify 5 sections of quantitative research reports. 3. Discuss the 5 components of the methods section of a quantitative research report.

3. Reasons for Evaluating Quantitative Research Reports • To determine if the findings can be used in practice • (Caelli, Ray & Mill, 2003)

4. Reasons for Evaluating Quantitative Research Reports • To determine the current state of science on a particular topic to direct your next research study

5. Components of a Quantitative Research Report Quantitative research report components: • -- Introduction/Research problem • -- Review of literature • -- Methods • -- Data analysis • -- Results • -- Discussion • (Caelli, Ray & Mill, 2003)

6. Introduction • Describes the background of the problem • Prevalence and/or incidence of the problem • Impact of the problem on patient outcomes • Importance of the problem to health care providers and nurses, in particular • Builds a case for the need for the study • Develops the Research Problem

7. Research Problem • Broadly defines what needs to be studied • Flows directly from the introduction • Concludes the Introduction section • (Caelli, Ray & Mill, 2003)

8. Review of the Literature Summary of pertinent literature Organized and clear Evaluate quality of literature Strengths and weaknesses Organized from broad to specific Identify gaps in literature Need for the study should be clear Discuss theoretical/conceptual framework

9. Methods Describes the steps used by the researcher to conduct the study Design Sampling plan Instrumentation Procedures Protection of human subjects

10. Design Describes the plan for carrying out the study Non-experimental designs Types Descriptive Correlational Experimental designs Types Randomized control trials Quasi-experimental designs

11. Sampling Plan Describes the sample for the study Who will participate How they will be identified Characteristics of the target population (the population to which the findings are generalized) Sample procedure Sample size

12. Instruments Describes the instruments used to gather the data for the study Researcher should identify which instruments measured which study concepts May involve Interviews Questionnaires Scales Observation Biophysiological measures

13. Instrument Reliability and Validity • Reliability is the instrument’s ability to consistently measure the concept • Validity is the instrument’s ability to measure the concept that is supposed to measure

14. Protection of Human Participants • Report should state that Institutional Review Board Approval was obtained • In the Procedure section the report should state that informed consent was obtained

15. Informed Consent Why Am I Being Approached? Why Is This Study Being Done? How Many People Will Take Part In The Study? What Is Involved in the Study? How Long Will I Be in the Study?

16. Informed Consent What Are the Risks of the Study? Are There Benefits to Taking Part in the Study? What Other Options Are There? What about Confidentiality? What Are the Costs?

17. Informed Consent • Will I be Paid for Participating in the Study? • What if I am Injured? • What Are My Rights as a Participant? • Whom Do I Call if I Have Questions or Problems? • Subject’s signature • Signature of Study Representative • If patient is incompetent to give consent, additional steps are taken

18. Procedure • Recipe for the research process • Sufficient detail should be provided • Should be written clearly and flow logically • All steps of the procedure should be clearly stated

19. Data Analysis and Results • Can be intimidating • Confirm that the researcher has presented results that clearly answer the proposed research question(s) • Researchers frequently organize this section by research question to facilitate readability

20. Data Analysis and Results Review the research design Descriptive design should use descriptive statistics such as: Mean, mode, and median (which measure how the data tend to be similar or grouped together) and variance, Standard deviation, Range (which measure how the data tend to be spread out) Correlational design should use correlational statistics such as a correlation coefficient (identifies the strength and the direction of the relationship between two variables. Examples are: Person r Spearman rho.

21. Discussion • Should clearly flow from the data • Should place the study’s findings in context with what is already known • May be linked back to theoretical or conceptual framework • Author interpretations should be clearly identified as such • Reviewer must determine if the author’s interpretations are justified • Should present study limitations • Should present implications for practice, as appropriate • Should present directions for future research

22. Study Limitations Common study limitations include: Small sample size Sample limited to one area or one group of people (this decreases generalizability to other areas and other people) Weak designs used such as correlational or quasi-experimental

23. Checklist for Research Report Evaluation • Research Problem • Is the problem clearly stated? • Is the problem significant? • Review of the Literature • Is the literature summarized? • Is the literature critically evaluated? • Are gaps and inconsistencies in the literature described? • Is the literature current and the review complete? • If presented, is the theoretical or conceptual framework clearly described including concepts and relationships? • Does the problem clearly link to and flow from the theoretical or conceptual framework? • Design • Is the design clearly stated?

24. Checklist for Research Report Evaluation • Sample • Is the sample clearly identified? • Is it clear how the sample will be obtained? • Is the relationship between the sample and the target population clearly delineated? • Is the rationale for the sample size provided? • Instrumentation • Is it clear which instruments will measure which concepts? • Is the rationale for instrument selection acceptable? • Is the reliability for each instrument described and acceptable? • Is the validity for each instrument described and acceptable? • Procedure • Are sufficient details provided in the procedure? • Is the procedure written clearly? • Does the procedure flow logically? • Are all steps of the procedure clearly stated?

25. Checklist for Research Report Evaluation Human Participants Protection • Has the researcher provided sufficient protection of human participants? • Data Analysis and Results • Is the data analysis section well organized? • Is the statistical method used for analysis appropriate for the research question(s) and/or hypothesis and level of measurement? • Are tables and graphs clearly labeled? • Do the tables and graphs complement the text?

26. Checklist for Research Report Evaluation Discussion • Does the discussion clearly flow from the data? • Does the discussion place the study’s findings in context with what is already known? • If a theoretical or conceptual framework is presented, are the nature of the findings discussed in the context of the framework? • If the author presents interpretations of the findings are these clearly distinguished as such? • Are justifications offered for the author’s conclusions? • Are study limitations provided? • Are implications for practice and future research delineated?

27. Summary • You may see yourself as a novice reviewer and not trust your interpretations. • This is a common concern. • Read, review, and discuss research reports with others to increase your skills.

28. References • Alderman, S. (1998). Critiquing research for use in clinical nursing practice: A CD-ROM review. Nurse Educator, 23(2), 8. • Beck, A. T., Steer, R. A., & Garbin, M. G. (1988). Psychometric properties of the Beck Depression Inventory: Twenty-five years of evaluation. Clinical Psychology Review, 8(1), 77-100. • Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 561-571.

29. References • Beck, C. T. (1990). The research critique: general criteria for evaluating a research report. JOGNN, 19(1), 18-22. • Beyea, S. C. (1998). Critiquing research for use in clinical nursing practice (CD-ROM). Computers in Nursing, 16(1), 16-17. • Brink, P. J., & Wood, M. J. (2001). Basic Steps in Planning Nursing Research (5th ed.). Boston: Jones and Bartlett Publishers. • Haughey, B.P. (1994). Evaluating quantitative research designs: part 1. Critical Care Nurse, 14(5): 100-2.

30. References • Haughey, B. P. (1994). Evaluating quantitative research designs: part 2. Critical Care Nurse, 14(6): 69-72. • Holcomb, Z. C. (2002). Interpreting Basic Statistics (3rd ed.). Los Angeles, CA: Pyrczak Publishing. • Pieper, B. (1993). Basics of critiquing a research article. Journal of ET Nursing, 20, 245-250. • Polit, D., & Beck, C. T. (2004). Nursing Research: Principles and Methods (7th ed.). Philadelphia, PA: Lippincott, Williams, & Wilkins.

31. References • Polit, D., & Beck, C. T. (2008). Nursing Research: Principles and Methods (8th ed.). Philadelphia, PA: Lippincott, Williams, & Wilkins. • Russell, C. L., Brown, K. (2002). The effects of information and support on individuals awaiting transplant. Progress in Transplantation, 12(3), 201-207. • Russell, C. L. (2005). Evaluating quantitative research reports. Nephrology Nursing Journal, 32(1), 61-64.

32. References • Ryan-Wengar, N. M. (1992). Guidelines for critique of a research report. Heart & Lung, 21(4), 394-401. • Soeken, K. L. (1985). Critiquing research: steps for complete evaluation on an article. AORN Journal, 41(5), 882-893. • Summers, S. (1991). Defining components of the research process needed to conduct and critique studies. Journal of Post Anesthesia Nursing, 6(1), 50-55.