190 likes | 202 Views
The quality of reporting of Health Informatics evaluation studies. Jan Talmon, Elske Ammenwerth, Thom Geven University Maastricht, UMIT. Content. Background STARE-HI Study design Results Discussion Future prospects. Background. HISEVAL workshop 2003
E N D
The quality of reporting of Health Informatics evaluation studies Jan Talmon, Elske Ammenwerth, Thom Geven University Maastricht, UMIT
Content • Background • STARE-HI • Study design • Results • Discussion • Future prospects
Background • HISEVAL workshop 2003 • Visions and strategies to improve evaluation of health information systems: Reflections and lessons based on the HIS-EVAL workshop in Innsbruck, IJMI, 2004, 479 • Poor quality of manuscripts submitted to MI journals • Editor and reviewer perspective • Development of STARE-HI • STAtement on the Reporting of Evaluation studies in Health Informatics
Background • Study questions: • Can STARE-HI be used for the assessment of the quality of evaluation studies in HI • What is the current quality of reporting • What areas are open for improvement
STARE-HI • Iterative development • Core writing team (JT, EA), active discussants, open comments through publication on Internet: http://iig.umit.at/efmi/ • There are 12 item categories described, some expanded • Title, abstract, keywords, introduction*, study context*, methods*, results*, discussion*, conclusion, conflict of interest, references, apendices
Study Design • Hand search of all issues of 2005 of three major MI journals – consensus (JT&EA) • IJMI, JAMIA, MIM • Develop a scoring list from STARE-HI (TG) • Test usability of scoring form • 3 papers assessed by 5 reviewers • Apply revised form on all selected papers
Results – paper selection • 282 papers reviewed on basis of title and abstract • 55 selected by JT • 37 selected by EA • Initial agreement on 32 • Final selection 48 papers: • 21 IJMI, 23 JAMIA, 4 MIM
Results – Scoring form • Extract issues from description in STARE-HI • The abstract should preferably be structured and must clearly describe the objective, setting, participants, measures, study design, major results, limitations and conclusions • Each issue scored • Maximum score/item • For abstract maximum is 9
Results – usability of form • 5 reviewers • EA, TG, JB, PN, NdK • All familiar with STARE-HI • 3 papers • Fitted to various degrees with STARE-HI (TG) • Intraclass Correlation Coefficient for each paper • 0.82-0.85
Results – usability of form • Feedback by scorers revealed some problems • Three items of STARE-HI were not clearly described • Twice it was unclear how extensive a description should be • Six issues could appear at any place in the article – affects reliability of scoring
Results – quality of reporting • Not all 48 papers could be properly assessed: • Some papers were more descriptions rather than evaluation studies • Secondary analyses • Evaluation of an algorithm or a general application (like email) • Final analysis on 39 papers • 19 IJMI, 20 JAMIA
Results – some observations • Keywords • Study design and “evaluation” often missing • Methods • Arguments for study design/case selection are often lacking. Description of intervention, study flow , outcome measures all reported <70%
Results – some observations • Results • Major findings reported, seldom unexpected findings or unexpected events influencing study • Conclusions • Sometimes lacking – even no summary statement, impact of findings, recommendations for future research • Conflict of interest/acknowledgment • Often missing. Relation authors-system
Discussion/conclusion • Applicability STARE-HI • YES • Scope of STARE-HI could be improved • Need for more supporting material on STARE-HI • Quality of reporting • Completeness: room for improvement • Completeness measure of quality? • Pilot study
Future developments • STARE-HI will be published in the MI literature • Seeking for broader support of STARE-HI • EFMI, IMIA, Journal Editors • Development of paper with elaboration on the reasons for items in STARE-HI • Broader study on the quality of reporting of evaluation studies in Health Informatics
Thank you! Questions?