The State of Eating Disorders Research Publications 1980-2000. An Empirical Analysis

1. 20 Eating Disorders 13.2 16 Anxiety Disorders 10.4 12 Percent 6.6 9.2 8 7.9 3.9 1.2% 6.5 0.0% 5.8 4 3.9 1.3% 0 1980 1990 2000 20 100 Eating Disorders Eating Disorders 16 Anxiety Disorders 80 Anxiety Disorders 12 60 Percent 30 Percent Eating Disorders 8 40 Anxiety Disorders Eating Disorders Anxiety Disorders 4 20 20 30 Eating Disorders 60 Percent Anxiety Disorders 0 0 1980 1990 2000 1980 1990 2000 Methods 10 20 Methodological Limits Discussed 40 Percent Rating Instrument 75-item rating based on recommendations from APA Task Force on Statistical Inference American Psychologist 1999 54(8): 594-604. Percent Eating Disorders 80 20 Anxiety Disorders 10 0 73.7 1980 1990 2000 60 67.5 63.2 • Funding source • Research question • Theory-based hypothesis • Sample size • Completion rates • Follow-up rates • Clear inclusion criteria • Method of determining inclusion • Recruitment strategies • Demographic information • IRB approval • Informed consent • Description of procedures • Random assignment • Description of measures • Psychometrics of measures • Blind rating of outcome • Research Design • Control/Comparison group • Effect size • Clinical significance • Constraint of alpha • Multivariate tests • Post hoc tests • Reporting of statistics • Confidence intervals • A priori power analysis • Methodological limits Percent 0 51.3 1980 1990 2000 40 0 1980 1990 2000 34.9 20 22.1 0 1980 1990 2000 3.9% 0.0% 0.0% 0.0% 0.0% 0.0% Prospective Longitudinal Design Eating Disorders Anxiety Disorders Eating Disorders Eating Disorders Year 60 100 Anxiety Disorders Anxiety Disorders 62.5 62.5 57.1 57.0 25.7 56.6 80 88.2 80 85.5 85.7 84.2 54.5 92.1 77.9 89.6 20.8 40 70.9 45.0 43.4 Percent 60 60 78.9 Percent 76.7 77.6 16.4 Percent 66.2 35.5 40 40 30.8 28.6 25.0 20 10.0 20 8.3 20 3.1 0 0 Year: 1980 1990 2000 0 1980 1990 2000 1980 1990 2000 The State of Eating Disorders Research Publications 1980-2000. An Empirical Analysis Mary J. Markland, MA, AHIP (Southeast Clinical Campus Librarian); Stephen A. Wonderlich, PhD (Associate Chair/Professor Department of Neuroscience) James E. Mitchell, MD (Chair/Professor Department of Neuroscience); Ross D. Crosby, PhD (Director of Biomedical Statistics & Methodology, Department of Neuroscience/Neuropsychiatric Research Institute) Martina de Zwaan, MD (Research Scientist, Department of Neuroscience/Neuropsychiatric Research Institute) Research Questions PsycInfo Search Strategies Population/Sample Clear Inclusion/Exclusion Criteria Interval Estimates Confidence Intervals Presented • What is the quality of eating disorders publications? • Has the quality of eating disorders publications changed over time? • How does the quality of eating disorders literature compare to publications in anxiety? Eating Disorders (anorexia nervosa OR bulimia OR binge eating OR eating attitudes OR body image disturbances OR eating disorder*) AND (treatment* OR therap*) Limits: journal article, English (anorexia nervosa OR bulimia OR binge eating OR eating attitudes OR body image disturbances OR eating disorder*) NOT (treatment* OR therap*) Limits: journal article, English Anxiety (anxiety disorder* OR phobia* OR obsessive-compulsive disorder OR panic disorder OR panic OR post-traumatic stress disorder OR traumatic neurosis OR stress reactions OR ocd OR ptsd OR post-traumatic stress disorder) AND (anxiety management OR treatment* OR therap*) Limits: journal article, English (anxiety disorder* OR phobia* OR obsessive-compulsive disorder OR panic disorder OR panic OR posttraumatic stress disorder OR traumatic neurosis OR stress reactions OR ocd OR ptsd OR post-traumatic stress disorder) NOT (anxiety management OR treatment* OR therap*) Limits: journal article, English The interpretation of the results of any study depends on the characteristics of the population included for analysis. Define the population clearly. If control groups are part of the design, present how they are defined… Describe the sampling procedures and emphasize any inclusion or exclusion criteria. Interval estimates should be given for any effect sizes involving principal outcomes. Provide intervals for correlations and other coefficients of association or variation whenever possible. APA Task Force on Statistical Inference Methods • Raters • 8 raters with master’s or doctoral level training in psychology/psychiatry • Training • 21 practice articles were reviewed jointly • Procedure • Citations were stored in EndNote libraries by subject, treatment vs. non-treatment and year • Citations were randomly selected from EndNote and articles obtained • Article assignments were balanced by field, year, content, and rater pairs • All articles rated independently by 2 raters • Raters were blind to journal, authors, and author affiliation • Disagreements resolved via consensus • Statistical Analysis • Power Analysis: n=456 articles to detect small effect size for ED vs. Anxiety • Interrater reliability assessed using kappa coefficients • Statistical analysis performed using hierarchical log linear analysis APA Task Force on Statistical Inference Interpretation Clinical Significance Data Presented When you interpret effects, think of credibility, generalizability, and robustness. Assignment Random Assignment of Subjects Random assignment allows for the strongest possible causal inferences free of extraneous assumptions…provide enough information to show that the process for making the actual assignments is random. (If) random assignment is not feasible, describe methods used to attenuate sources of bias, including plans for minimizing dropouts, noncompliance, and missing data. APA Task Force on Statistical Inference 25.6 15.8 15.8 9.2 9.1 7.8 Conclusions Note the shortcomings of your study…acknowledging limitations if for the pupose of qualifying results and avoiding pitfalls in future research. APA Task Force on Statistical Inference Procedures Assessor Blind to Condition PubMed Search Strategies APA Task Force on Statistical Inference Clearly describe the conditions under which the measurements are taken (e.g. format, time, place, personnel who collected data). Describe the specific methods used to deal with experimental bias… Eating Disorders (eating disorder* OR anorexia nervosa OR bulimia) AND (therapy OR therapies OR treatment) Limits: human, journal article, English (eating disorder* OR anorexia nervosa OR bulimia) NOT (therapy OR therapies OR treatment) Limits: human, journal article, English Anxiety Disorders (anxiety disorder* OR obsessive compulsive disorder OR panic disorder OR phobia* OR agoraphobia OR stress, disorders, post-traumatic OR post- traumatic stress disorder* OR ptsd) AND (therapy OR therapies OR treatment) Limits: human, journal article, English (anxiety disorder* OR obsessive compulsive disorder OR panic disorder OR phobia* OR agoraphobia OR stress, disorders, post-traumatic OR post- traumatic stress disorder* OR ptsd) NOT (therapy OR therapies OR treatment) Limits: human, journal article, English Conclusions • Eating Disorder publications tend to be less methodologically rigorous than Anxiety Disorder publications in many important areas. Structured Interviews Prospective Longitudinal Designs Random Assignment Blind Outcome Assessment • Both Eating Disorder and Anxiety Disorder publications demonstrate improvement in methodological rigor over the last 20 years. Sample Size Alpha constraint Effect Size Psychometrics • The majority of Eating Disorder and Anxiety publications do not include many of the APA Task Force recommendations. Confidence Intervals Clinical Significance A priori power analysis Alpha constraint APA Task Force on Statistical Inference Power and Sample Size A Priori Power Analysis Mentioned Study Design Provide information on sample size and the process that led to sample size decisions. Document the effect sizes, sampling and measuring assumptions, as well as analytic procedures used in power analysis. APA Task Force on Statistical Inference Results Effect Sizes Effect Size Data Presented Limitations Research Question/Hypothesis Stated Design Always present effect sizes for primary outcomes. • Literature searches may not have found all the articles • Excludes animal/basic science studies • Excludes non-English articles • Low interrater agreement on some ratings • Raters could deduce field & year from articles Make clear at the outset what type of study you are doing. Do not cloak a study in one guise to try to give it the assumed reputation of another…be sure to define and prioritize (the) goals. APA Task Force on Statistical Inference APA Task Force on Statistical Inference