Design and Analysis of Open Response Surveys: Lessons Learned

1. Design and Analysis of Open Response Surveys: Lessons Learned Dr. Joan Burtner Associate Professor of Industrial Engineering Mercer University

2. Design and Analysis of Open Response Surveys Designing Survey Administering Survey Conducting Survey Analysis Reporting Survey Results Mercer University School of Engineering

3. Open Response Surveys Ask participants to respond to a question such as “What can we do to improve patient care? Excellent method for collecting “Voice of the Customer” data within a Quality Mgt. System Allows analysts to interpret qualitative characteristics (tone, frustration, pride) Opportunity to discover unanticipated responses Mercer University School of Engineering

4. Designing and Administering the Survey Purpose of survey Delivery method (paper, electronic, phone, in-person, etc.) Length of survey, allowable response time Incentives for participation Anonymity, confidentiality Who will collate and/or analyze data? How will data be analyzed? Mercer University School of Engineering

5. Design of Survey Questions • Structure, specificity of written questions • What can we do to improve the process? • What is the one thing that will make your job easier? • What is the “one idea” that will improve the registration process? • Text-based questionnaires vs. oral interviews • Limited, prescribed, uniform questions • Non-uniform follow on questions in addition to prescribed questions Mercer University School of Engineering

6. Example 1: On-line Survey Question Please describe challenges that make it difficult to deliver patient care. Over 700 responses submitted anonymously over a two week period Data downloaded into an Excel file Many rows of data contained no text, just time and date of submission Two 2-person teams reviewed the data and developed preliminary codes Mercer University School of Engineering

7. Example 1: Initial Coding Coding done electronically by labeling each response in Excel Initial decision to allow only one major code for each entry to simplify analysis. Twenty-four categories emerged between the two teams. Overlap and clarification of labels resulted in the elimination of four categories. Mercer University School of Engineering

8. Example 1: Revised Coding Difficult to gain consensus among all four coders on the “one” main concept for each response Realization that multiple codes increased validity of work Excel spreadsheet expanded to four columns to accommodate multiple codes for each response Mercer University School of Engineering

9. Code - Process Change? Patients should not be transferred to med surg during change of shift. There needs to be a window of time that patients cannot be transferred. It is very unfair to the patient to be transferred at that time as going off nurses are trying to get caught up and give report; and coming on nurses are trying to get report. For example; no transfers between 6:00 and 07:15pm Mercer University School of Engineering

10. Code - Work Load? Policy? If a patient is computer literate, why do we not have some of the admission forms on line so that patients can fill out the questions at home? They would certainly have access to their medications & we might have a more exact list of what they take. They would be in a more comfortable & less stressful environment when filling out the forms & the older folks would most likely have a family member to assist in remembering when they had previous procedures, etc. Mercer University School of Engineering

11. Audience Participation Exercise Form teams of two or three Consider the following responses (handout) How would you code the data? Do multiple codes increase validity? Mercer University School of Engineering

12. Example 2: Nursing Interruptions Research agreement with a local hospital Students observed nurses at work Data collected orally and transcribed by observers Anecdotal reports from nurses collected periodically as time permitted Mercer University School of Engineering

13. Example 2: Nursing Interruptions Data Excerpt 1 April 15th Comments by Nurse - Interruptions Helping with code lift or code response Family Telephone calls Doctors When technician needs help Comments by Observer 1 Tech was using chart station when Nurse was trying to chart Having room in another section causes a lot of extra walking This could be a system failure interruption Mercer University School of Engineering

14. Example 2: Nursing Interruptions Data Excerpt 2 April 16th Comments by Nurses - Interruptions Family Lab Calls asking about results or talking about results Doctors calling Patients – getting called by other patients and helping other patients Unexpected events – Patients getting out of bed, codes Helping nurses, covering nurses when they are out for lunch Comments by Observer 2 Quiet day with not many interruptions Nurse didn’t have much to do Mercer University School of Engineering

15. Coding Nursing Interruptions • Nature of the interruption • Phone call , call light, patient’s family, healthcare professional, etc. • Urgency of the interruption • Avoidable, justifiable, etc. • Consequence of the interruption • Delay, error, etc. Mercer University School of Engineering

16. Example 3: On-line Survey Question Please list and describe ideas that will contribute to cost efficiency within our system. Over one hundred responses submitted Data downloaded into an Excel file Single person reviewed the data and categorized each response Fifty-eight usable responses Data organized into a pivot table Mercer University School of Engineering

17. Reporting Analysis Results • Example 1 • Ten-page written report • Power-Point presentation to five-person committee • Follow-up submission: frequency distribution models overall and departmentalized • Example 2 • In-person feedback to nurse manager • Brief report with Pareto diagrams • Example 3 • Electronic feedback to client • Pivot table demonstration Mercer University School of Engineering

18. Code Frequency - All Locations Mercer University School of Engineering

19. Frequency - Aberdeen Mercer University School of Engineering

20. Frequency - Dallas Mercer University School of Engineering

21. Frequency - Greenville Mercer University School of Engineering

22. Example 3: Pivot Table 1 Mercer University School of Engineering

23. Example 3: Pivot Table 2 Mercer University School of Engineering

24. Example 3: Pivot Table 3 Mercer University School of Engineering

25. Lessons Learned 1 Nurses are willing to spend time completing surveys. Many are eager to have their ideas heard. It is important to give feedback, when possible, with respect to how the survey results may change the process. Mercer University School of Engineering

26. Lessons Learned 2 The process of analyzing and quantifying open-ended responses is very time-consuming. It is very difficult to gain consensus on some responses. System-wide data should also be reported at the unit level if possible. In the case of “one best idea” it is difficult to prioritize. Mercer University School of Engineering

27. Lessons Learned 3 Due to the somewhat subjective nature of the coding and frequency tabulation, it may be difficult to use open-ended survey responses to measure the effectiveness of process changes. However, changes in the culture and attitudes of the respondents may be apparent. Mercer University School of Engineering

28. Questions / Contact Information Dr. Joan Burtner Burtner_J@Mercer.edu Associate Professor of Industrial Engineering and Industrial Management (478) 301- 4127 Mercer University School of Engineering 1400 Coleman Avenue Macon, GA 31207 Mercer University School of Engineering