Agency for Healthcare Research and Quality Survey Administration Logistics

1. Agency for Healthcare Research and Quality Survey Administration Logistics 2nd Technical Assistance Conference Call on the Hospital Survey on Patient Safety Culture (HSOPSC) March 24, 2005

2. Objectives • Discuss survey administration issues, data analysis, and feedback report preparation • Additional details and tips • Assume listeners are familiar with the survey and toolkit materials • On AHRQ web site at www.ahrq.gov/qual/hospculture/

3. Speakers Cynthia Barnard, MBA MSJS CPHQ, is the Director of Quality Strategies at Northwestern Memorial Hospital in Chicago. (cbarnard@nmh.org) Marilyn K. Szekendi, RN, MSN, APRN, BC, is aResearch Nurse Coordinator for the Patient Safety Team, Division of Quality and Operations, at Northwestern Memorial Hospital in Chicago. (mszekend@nmh.org) Veronica Nieva, PhD is a Vice President and Director of the Organizational and Management Research Group at Westat. Dr. Nieva has a background in social/organizational psychology and is one of the principal investigators responsible for the development of the Hospital Survey on Patient Safety Culture under an AHRQ contract. (veronicanieva@westat.com) Joann Sorra, PhD is a Senior Study Director at Westat. Dr. Sorra has a background in industrial/ organizational psychology and is one of the principal investigators responsible for the development of the Hospital Survey on Patient Safety Culture under an AHRQ contract. (joannsorra@westat.com)

4. Northwestern Memorial Hospital • A 725-bed academic medical center in Chicago • Decision to use HSOPSC based on: • Value of using validated instrument • Hope to compare results with other institutions Northwestern Memorial Hospital slide

5. Survey Administration • Web survey of the HSOPSC • Dec 2004-Jan 2005 (~6 weeks) • Developed web version in-house • Email notification to all 6000 staff, plus physicians • Anonymous • Modifications to survey • Changed midpoint to “Neutral” • Changed “staff” to “people” • Added OR/Surgical Services to list of hospital units • Next time: Comments indicated need to add technologists to list of staff positions Northwestern Memorial Hospital slide

6. Survey Response • Good response compared to prior 1-page paper-based survey • 28% overall response rate • 32% among employees • 15% among attending & house staff • 1,654 usable surveys completed Northwestern Memorial Hospital slide

7. Data Cleaning • Removed some responses • Based on comments, removed some non-clinical staff (I/S, marketing, etc.) (~120 surveys) • Blank responses or all same answers (~40 surveys) • Missing some patient safety grade data due to technical problem Northwestern Memorial Hospital slide

8. Results • 80% of respondents had direct patient contact • Responses by provider type/position • Nurses less positive on staffing and teamwork across units • Physicians less positive on handoffs and feedback • Management most positive • Hospital-based services (pharmacy, radiology, etc.) most negative • Recommended 75%/50% cutoffs for identifying strengths & areas for improvement too high • Examined means converted to a 100-pt scale • (1=0, 2=25, 3=50, 4=75, 5=100) Northwestern Memorial Hospital slide

9. Sharing Results/Follow-Up • Not yet formally distributed hospital-wide • Senior management welcomes data; domains are effective for communication • Pleased with the extent of participation • Results useful when assessed by • Type of provider • Clinical area • Bedside or Hospital-based staff • Plan to administer annually Northwestern Memorial Hospital slide

10. Lessons Learned • I/S skills are needed • Web access is needed • Paper option may be helpful? • Periodic reminders do help • High response rate (including MDs) suggests both format and content were intuitive • Domains are intuitive and effective for communication and linking to actions • Data will enhance our understanding of employees’ perceptions of culture • Future: how to address non-clinicians? Northwestern Memorial Hospital slide

11. Project Planning • Task Timeline for Project Planning (Ch. 2, page 8 of Survey User’s Guide)

12. Deciding Whom To Survey • Who should be included? • Goal is for survey respondents to be representative • All staff—a census • Subset of staff—a sample, selected by: • Departments/units • Staff categories • Some other criteria • Don’t use haphazard approach • Don’t simply leave stacks of surveys in hospital areas or hand out surveys to staff passing by • Sample size depends on purposes of survey

13. Survey Modifications • For comparison and benchmarking—don’t modify the survey • Changes will affect the degree to which your hospital will be able to compare with other hospitals • Deleting questions • Concerns about length—takes 10 minutes to complete • Do not “shorten” instrument by deleting single questions; instead, delete all items in dimensions that are of lower priority • Adding questions or response categories • As a rule: Add new questions to the end

14. Modifications by Setting • Administration in non-hospital settings • Ambulatory/outpatient facilities • Nursing homes • Administration in very small/critical access hospitals • Goal is to orient ALL questions toward the facility as a whole, not to units or departments • Use the “facility” version but replace terms “work area” and “unit” with the word “facility”

15. Response Issues • Instrument does not have “Don’t know” option • Consider adding instruction: • “If you do not wish to answer a question, or if a question does not apply to you, you may leave your answer blank.” • This instruction already added to the image-scannable versions of survey

16. General Data Collection Issues • Data Collection Methods • Web surveys • Paper surveys (“regular” & scannable) • Mixed mode • Anonymous vs. Confidential Data • Maximizing Response Rates • When to Conduct Survey

17. Web Surveys • PROS • Simple to administer, no data entry, simpler data cleaning than paper surveys • CONS • Staff need email access, internet access, familiarity with computers/internet use

18. Paper Surveys • PROS • Can reach everyone regardless of email or internet access, may result in higher response rates for some staff, may be easier when have small sample • CONS • More labor intensive in administration and data cleaning; may be more costly for very large samples; may take more time

19. Mixed Mode • Some groups complete web survey, others complete paper surveys • PROS • May be able to reach more staff; may obtain higher response rates than single mode • CONS • More complicated administration and data manipulation

20. Web Survey Administration • Develop in-house vs. using web survey vendor • Administration tips • Thoroughly test web survey • Don’t force a response to every item • Consider distributing hard copy prenotification letter followed by email notice with hyperlink to web survey • Set up a computer room with assistant • Expect biggest wave of responses within first week • Survey data collection period of 4 to 6 weeks

21. “Regular” Paper Surveys • Require manual data entry • Feasible for smaller samples • In-house data entry or through a vendor • Can use Premier Excel Tool for data entry and analysis/reporting

22. Scannable Surveys • Minimum of 1,800 to 2,000 in sample to make scanning more cost effective than manual data entry • Industry moving toward image-based scanning (imaging) vs. optical mark read (OMR) scanning • Advantages of imaging • Can be formatted in any word processing software • Can be printed from electronic file (but NO photocopying) • AHRQ web site has image scannable surveys (hospital and facility versions)

23. Assumptions for Image Scanning Costs • 5-page HSOPSC image-scannable survey (8 ½ x 11) • 51 closed ended questions • Data entry of written comments: Assume 30% of respondents will provide written comments averaging 300 words long • Double-sided printing, black-and-white, one staple upper left corner • Hospital handles survey administration and delivers completed surveys to vendor • Vendor delivers data in ASCII

24. Costs for Image Scanning • Printing surveys @ $0.30 per survey • Vendor imaging setup of survey @ $200 • Scanning @ $0.40 per survey • Processing open-ended responses (other specifies, written comments) • $0.0006 per character • Interpreting/coding of write-ins • Additional hospital costs for postage (if applicable, for sending and returning surveys)

25. Some Image Scanning Vendors • Optimum Solutions Corp. (OSC) www.oscworld.com Contact: Lesia Milan lesia.milan@oscworld.com 516-247-5322 • NCS-Pearson www.PearsonNcs.com Contact: Lawrence (Laurie) Lawson laurie.lawson@pearson.com 703-284-5802

26. Anonymous vs. Confidential • Anonymous • No way to identify or track individual responses • All follow-up reminders, surveys go to everyone • Possibility of more than one response • For paper surveys—can use different “Form Number” for different hospitals or color-code to identify groups

27. Anonymous vs. Confidential • Confidential • Track individual response through an identifier or code • Ensure one response from each individual • Eliminate respondents from further follow up/reminders • Precisely track response rates • Requires careful handling during administration • Lists of survey respondents and linking identifiers • Storage/handling of completed paper surveys • Use of outside vendor may provide additional assurance • Methods • On web, can use individual usernames/passwords • On paper, can use preprinted identifiers, labels, or barcodes

28. Maximizing Response Rates • Secure support for survey throughout hospital • Advance publicity & communication! • Use of individual and group incentives • Catered meal for units with XX% response rate • Raffles or lottery for weekend getaways, dinner or store gift certificates • Printable cafeteria meal ticket upon web survey completion • Monitor response statistics every week

29. When to Conduct Survey • Do not recommend administering this survey at same time as another staff survey • Not necessarily an annual survey

30. Preparing and Analyzing Data • Clean the data • Delete blank records or those with few items completed • Delete records that are exact duplicates • For hand data entry—fix out-of-range values • Analyze data—keep it simple • Frequencies of response on items • See Premier Excel Tool as example • http://www.premierinc.com/all/safety/culture/index.jsp • Handles maximum of 2,500 responses • Breakdowns of data • Decide which subroup analyses will be meaningful to administration and staff • Rule of 10--Do not report responses for groups with fewer than 10 respondents

31. Sample Charts from Premier Tool

32. Sample Charts from Premier Tool

33. Benchmarking • Preliminary benchmarks from over 1400 staff from 20 pilot hospitals now available on AHRQ web site • No central repository for data yet, but AHRQ has plans to support a national benchmarking database for survey • Average % of positive response for items and for composite scores on the safety culture dimensions • 5% difference as a rule of thumb when comparing your hospital’s results

34. April HSOPSC Conference Call • “Taking Action” • You have your survey results, Now what? • How to communicate results throughout the hospital • Determining areas of priority • Action planning and accountability for change • Tracking progress • Email questions to Westat at • safetyculturesurvey@westat.com • Save the date: Friday, April 22 at 2pm EST • To register, send email with “April 22 TA Call” in subject line and provide name