Evaluating Research Reports

1. Evaluating Research Reports Chen-Sheng Chen Institute of Rehabilitation Science and Technology National Yang-Ming University September 15, 2003

2. Analyzing content of an article • Title and abstract • Introduction • Methods • Subjects • Design • Instrumentation • Procedures • Data analysis • Results • Discussion and Conclusion

3. Consideration of merit of an article • Reputation of Journal • Two experts to review • Meet the criteria of journal • Elements considered by reviewers • Importance of research • Originality • Appropriateness of research design • Adequacy of the method • Relevance of discussion • Clarity of writing

4. Science Citation Index • 科學情報研究所 • Institute for Scientific Information，簡稱ISI • 約3500種 理、化、農、林、醫、生命科學、天文、地理、環境、材料、工程技術等自然科學各學科的核心期刊影響因數 • 影響因數 • Impact Factor (IF) • 該刊前二年發表的文獻在當年的平均被引用次數

5. Spine (4/39)

6. Title and Abstract • Title should be informative • Next to read abstract • Information of abstract • Purpose • Method • Results • Major consideration • Without studying the details of paper, no way to sure that the study was valid

7. Introduction • Provide the impetus for the study • What is the problem? Why is it important? • How has the author used the literature to form a logical rationale? • What is the theoretical context for the study? • What is specific purpose? • What are hypotheses or guiding questions?

8. Methods - subjects • Readers need to know who the subjects were, can be applied to clinical situations • Who were the subjects? • What were inclusion and exclusion criteria? • How were subjects selected? • How many subjects were studied? Is the sample size adequate?

9. Inadequate situation • Apply the normal to predict the patients • Comparison under totally different groups • Numbers of subjects is an important consideration when interpreting the results

10. Methods - Design • What is the research design? Is it appropriate for answering the question? • How many groups were tested? • How often were treatments and measurements applied?

11. Three reasons to notice • First • Relate the finding to the ‘real world’ • Alert to the possibility that extraneous variable interfered with the results • Second • Statistical analysis: numbers of groups, numbers of dependent variables, frequency of measurement, etc. • Third • Clinical trials model or single-case design or, readers should know special rules if can follow up the rules or not

12. Methods - Instrumentation • In instrumentation described insufficient detail? • How have the authors documented the reliability and validity of the instruments?

13. Commercial device • Model numbers, names, addresses of manufacturers • Calibrated regularly • Standardized Questionnaire • References must be cited • Reliability • Reproducibility and consistency of measurements • Specific populations

14. Example • Test grip strength • Reliable to test normal subjects ! • Does it reliable to test patients with wrist pain? • Does reliability differ if measurement are taken at home or in the clinic? • Was the status determined by self-report, interview or observation?

15. Method - Procedures • Are data collection procedures described clearly and insufficient detail to allow replication? • What are operational definition for all variables? • Position, velocity, acceleration

16. Data collection • Who performed the measurement? • When or how often measurement were taken? • In vitro test: ligament strength • Spine specimen • Conducted by surgeon • Age of specimen • Temperature • Procedure about removing soft tissue

17. Data analysis • What statistical procedure were used? • Are they appropriate to analyze the data? • Has the author justified the use of any unique or unusual statistical tests? • Has the author addressed each research question in the analysis? • What alpha level was used for the level of significance? • Inferential statistics, α=0.05

18. Results • Report finding without interpretation • Do the results address the research question? • Are figures and tables presented accurately? • Are the results statistically significant? • Something to notice • Wrong data collection • Missing information • Subjects -> statistical results

19. Discussion and Conclusion • How does the author interpret results? • Did the author clarify if hypotheses were rejected or accepted? • How are the finding related to prior reports? • What limitations are described? • Are there limitations that are not addressed? • Does the author discuss how the results apply to practice? • Does the author present suggestions for further study? • Do the stated conclusions flow logically from the obtained results?

20. Study for what? • What do you do in your study? • Why trust your study? • Any evidence observed in your results? • Inference logically • Strictly obey the test procedure • To THINK any possibility (台積電: 張忠謀)

21. 核子反應爐工程師 : 9年 麥肯錫東京事務所 核子工程 ??=> 企業管理 損益平衡分析? 公牛乳頭? 圖書室 客戶資料個案微卷 每晚28分鐘的電車 10:48 -11:16 PM 思考問題解決方案(1year) 蕃茄醬廣告 如何擴大某一品牌的蕃茄醬市場? 看板是否能反應銷路? 多元思考 蕃茄醬或蕃茄汁的不同 客戶問題需求 相關思考案例結合 管理大師 / 大前研一 顧問費:43萬/1日

22. 1920年: 十家啤酒廠商 排名第八 啤酒廣告 最純的啤酒 沒人體會什麼是「純」 專業知識與製酒過程 清澈密西根湖 自鑽500呎深自流井 5年進行1623次實驗 開發最好的酵母 裝酒瓶: 600度蒸汽殺菌 最「純」的啤酒 告訴消費者過程 體會「純」的定義 排名第一的啤酒 (6 months) 施麗茲啤酒(schlitz beer)行銷顧問

23. Strength and Weakness RST • 聯強國際總裁:杜書伍 • 懂80%:常識 • 專業20:知識 • 贏20%做老大

24. Do exercise other than study hard • Data adopted from • Ch 31. Evaluating Research Report. • 鄭誠功所長 : 科技論文寫作 • 商業週刊 : no. 805 (2003.4.28-2003.5.4) • :贏20%做老大 (page 125-131)