Ann Versporten, Ingrid Morales, Carl Suetens

1. Scientific Institute of Public Health Data validation study of the National surveillance of nosocomial infections in intensive care units Ann Versporten, Ingrid Morales, Carl Suetens IPH, wednesday seminar: May 7, 2003

2. Overview • Background: overview national surveillance ICU • Reasons for validation • Validation study • Aims • Methods • Results • Pneumonia • Bacteraemia • Discussion • Conclusions • Recommendations

3. Background: National surveillance ICU • 1996: Start National Surveillance of Hospital Infections (NSIH) : intensive care component (Pneumonia & Bacteraemia) • HELICS-based protocol (Hospitals in Europe link for Infection Control through Surveillance) • patient-based surveillance: 1 file by patient, + infection file if ICU-acquired PN or BAC • Nosocomial: infection acquired during hospital stay (admitted >48h in ICU)

4. Background: National surveillance ICU • Objective: to follow-up nosocomial-infection rates • Risk-adjusted infection rates are used as external benchmarks for comparison purposes

5. Methods: Data collection for ICU surveillance • Data at admission • Day-by-day e.g. central venous catheter, mechanical ventilation, antibiotic use • Infection data e.g. diagnostic criteria of PN, origin of BSI • Data at discharge

6. Reasons for validation • Assessment of the validity of the findings • Need to evaluate the accuracy of infection data reported to the NSIH program

7. Validation study

8. Main aim • Validate reported ICU-surveillance data (ICU protocol: PN & Bac) against a reference gold standard • Evaluate the accuracy of all data reported to the surveillance • Evaluate the credibility of the surveillance

9. Specific aims • Exhaustivity (completeness) denominator • Sensitivity: probability of reporting a true PN & Bac to the ICU-surveillance • Specificity: probability of reporting a PN & Bac as negative to the ICU-surveillance if the disease is truly absent • Positive predictive value • Negative predictive value

10. Methods - 1 • Sampling of hospitals: • Systematic sampling of 45 hospitals on the base of a list of hospital-trimesters (ICU participation period 01/01/1997 – 31/12/1999) • Replacement: later period accepted • Informed consent, voluntary participation • Retrospective chart review methodology

11. Methods - 2: Research program Sampling of patient files: • All reported PN+ & Bac+ (from surv.) • All records with a positive hemoculture reported on a laboratorium list (for all admitted patients on ICU) (estimation false-neg Bac) • A 20% random sample of the negative files (estimation of false-neg PN) Estimation of exhaustivity of denominator on the base of administrative lists of ICU-admissions

12. Methods - 3 • Calculation Se, Sp and Predictive values • “gold standard” = research team • Trained data collectors (IPH) • Application protocol definitions • validation: uniform & standardised • evaluation = blind • discrepant infections: reviewed by other colleague • Confidential & anonymous treatment of patient data

13. Methods - 4 • National results • No individual hospital results, only discussion at end validation proccess • Quality of data • Questions

14. Results - 1 • 563 investigated patient files in analysis: pts staying >24h in ICU (23 hospitals) • Infections reported by hospitals to surveillance: • 147 Pneumonia • 49 Bacteraemia • Type of ICU: 91% polyvalent • Size of ICU: mean 10 beds • Length of stay: median = 4,7 days

15. Results - 2 • Exhaustivity of denominators: • For all patients staying >24h in ICU • 72,8% • For all patients staying >48h in ICU • 81,2%

16. Results - 3: Pneumonia Results of validation study for PN (inf. file &/or dbd)

17. Results - 4: Bacteraemia Results of validation study for Bac (inf. file &/or dbd)

18. Results: SE & SP

19. Results: predictive values

20. Discussion - 1 • Exhaustivity denominator: improvement possible – risk of bias, e.g. if only high risk patients included • Pneumonia: low Se., good Sp. • Bacteraemia: low Se., good Sp.

21. Discussion – 2 • Possible reasons for lack of sensitivity • 30% of the results originate from 1997 (start surv. NI in ICU). • 50% of the collected data correspond with the 3 first surveillance trimesters that hospitals participated to our ICU surveillance. • = Explanation of lack of accuracy in the interpretation of the protocol ?

22. Who are those missed patients ?? Why are there so many false negative Pneumonias ?

23. Characteristics false negative PN

24. Factors influencing the Se. & Sp. of the infection data • Who collects data ? • Who decides whether a PN should be reported or not ? • Criteria of bloodculture? • Adherence to protocol definitions • Degree of workload (ratio pat.-staff) • Size of hospital • …

25. Conclusions • Exhaustivity varies for each hospital, but remains satisfactory in general • Bac more accurately reported than PN (Se) • Seldomly infections reported which were not a nosocomial infection (Sp) • Absence of a gold standard ! (problem for diagnostic of PN)

26. Conclusions (next) • Establishing Se & Sp only possible at the end of validation study • Preliminary conclusions: Sensitivity rather low (identification of a NI through surveillance) Specificity is high (% files truly classified as non-NI) • Low Se has also been reported by the CDC: “The data collectors detected over 2,5 times as many PN, ..” (Emori, Edwards, et al. 1998)

27. Recommendations • Training of professionals in charge of surveillance (Ehrenkranz, Shultz, et al. 1995) • case definitions (e.g. PN-diagnostic: use of micro-biologic reports & AB-administration) • surveillance-methods • Simplification of protocol • Development of electronic surveillance

28. Recommendations (next) • Validation on continuous basis • Training on the field • Optimalisation contacts IPH / hospitals