Using Surveys to Support OD projects

1. Using Surveys to Support OD projects Mick Verran for Boston Facilitators Roundtable 02-07-2014 Handout www.MickVerran.com Password BFR • This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. You are free to copy and distribute the work under the following conditions: you must attribute the work to “Mick Verran Mick@MickVerran.com” ; you may not use the work for commercial purposes; if you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.

2. Issues in gathering data Sensitivities about the role of an outside ‘expert’ Impact of the intervention itself Barriers to participation – distribution, trust Challenges in balancing the loudest voices Availability & accessibility of usable data Whole or representative population required Degree of accuracy required Funding & time available

3. Data-gathering methods

4. Survey tools Google Constant Contact SearchGizmo Capricity Qualtronics – QuestionPro – Vertical Response ‘… where I do not speak the language’ http://qualtrics.com/https://www.questionpro.com/ ‘Analysis, vendors who specialize in analysis and reporting, current tools and trends, live poling, web-based live poling’ www.DiscoverSurveys.com – Bruce Katcher

5. ‘How others are using surveys’ Download Handout 1

6. ‘Other ways to leverage surveys’ • M • O • V • E (Role = moving the process along) Motivation Ownership Voice Expectation Download Handout 2

7. ‘What makes for a good survey?’ Everyone concerned agrees that the survey asks an importantquestion that each respondentyou need to hear from will understand, know their answerto, and be willingand able to answer, to produce information you can reliably interpret and use.

8. Online survey workload

9. Needs of interested parties • Inform the content of what you’re doing • Move the process along Client • Validation • Efficiency • > Participation • Temperature check Survey Participants • A voice • Understand what the others involved think. Consultant • Reach around the client to uncover hidden issues • Platform for success • Attune to the language and issues

10. Applications – Goals Download Handout 3

11. Response rates Online, ‘out of the blue’ market research = 30% average Our pre-survey = 27/30 = 90% What impacts the response rate? • Connected with issues of importance to the respondents. • Promise of report back on results. • Credible commitment of the sponsor to address issues. • Confidence in confidentiality. • Accommodation of e.g. shift schedules. • Easy-to-answer. • Questions & scales • Goal; 20 questions or 5 minutes (5 closed or 2 open per minute)

12. Choosing respondents Who to ask? Stakeholders – Knowledgeable – Opinion Leaders ‘Gathering data from a large sample…’ http://www.calculator.net/sample-size-calculator.html ‘Surveying groups for the same project with variants of the survey’ Separate surveys – no value in complexities of creating too much conditional/skip logic/branching/piping

13. Employee Engagement Experiences Which ‘drivers of engagement’ do you experience? Commitment How committed are you to our new strategy? Confidence How confident are you that we will successfully execute our new strategy? Concerns What concerns do you have about our new strategy? What concerns do you have about our ability to successfully execute our new strategy?

14. Organization/Team Effectiveness Experience How often does the team ……? Evaluation How well does the team do ……? Satisfaction How satisfied are you with the way the team ………? Opportunities for improvement Improvements in which of the following would have a significant impact on the team’s effectiveness?

15. Training Needs Analysis Relevance How often do you use this skill at work? Importance How important is this skill to success in your work? Competence How skilled are you in ……? Development Need I need to improve my ability to …. Commitment to Improve How likely are you to attend ….?

16. Applications – Measures Download Handout 3

17. Survey sections Why doing survey, benefits, how results will be used Time commitment, when required Contacts – content AND completion Call for Honesty – use full scale, specific examples Confidentiality framing Demographic data Questions Open ended comments Thanks

18. ‘Collecting demographic info’ Only those that are vital to the achievement of your objectives Not so narrow that one can guess Optional space for self-identification if open to follow-up SurveyMonkey tracks IP address; individual tracking requires unique survey Positioning Demographic questions At the end of the survey, when respondents have developed confidence in the survey and it’s objective OR Early, straight after the reassurance of anonymity, so don’t feel cheated if asked at the end

19. So many options! How many times in the last (period of time) have you bought a Coca-Cola? 0 1 – 5 6 – 12 More than 13 How much do you enjoy Coca-Cola? I enjoy Coca-Cola I neither enjoy nor do not enjoy Coca-Cola I do not enjoy Coca-Cola Given a choice, which would you choose? Coca-Cola Pepsi I prefer Coca-Cola to Pepsi; Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree If you were to buy 10 soft drinks, how many of each would you buy? Coca-Cola Pepsi I have chosen Coca-Cola over Pepsi; Always Often About half and half Rarely Never I enjoy Coca-Cola; More than I enjoy Pepsi As much as I enjoy Pepsi Less than I enjoy Pepsi Put these soft drinks in order of your preference; Coca-Cola Pepsi Mountain Dew Are you willing to; Drink Coca-Cola Buy Coca-Cola Offer Coca-Cola to your guests Buy Coca-Cola for your children Consult to Coca-Cola Inc. Invest in Coca-Cola Inc. Work for Coca-Cola Inc.?

20. Question types Download Handout 4

21. Respondent Tendencies Agree with statements as presented. Respond from general opinion of positives & negatives. Have own definition of e.g. ‘frequently’ v. ‘occasionally’. Avoid using extreme response categories/poles. “I Tend to Disagree” is more negative than “I Tend to Agree” is positive. Portray selves or group in a more favorable light when asked about; Feelings of powerlessness, "kindness" or "benevolence“, honesty, willingness to contribute. Ability to do job, stress levels, company loyalty, value of own contribution, compliance with procedures.

22. How the question and the scale work together

23. ‘Preventing bias’ – ‘Examples of what not to do’ Download Handout 5 Reliability – Problem questions & scales

24. Anchored rating scales

25. Rating Scale Template – ‘gold standard’ Download Handout 6

26. Favorite questions & scales Download Handout 7

27. Questions about scales 1 How many points should there be on the scale? How many degrees of difference can respondents discriminate? How many responses will give rise to different actions? Should I include a mid-point? Use is heavily influenced by “social desirability bias”. “… a four point scale without a mid-point appears to push more respondents towards the positive end of the scale.” (a four point scale) “… can … (force) subjects to indicate their opinion when they actually have no opinion and are not simply reluctant to reveal it.” “This study found the opposite … the way in which people will respond to a balanced Likert type scale without a mid-point is content-specific.” (To minimize the use of the mid-point) “… either not include it at all or use scales with many points.”

28. Questions about scales 2 Should I start with the “agree” or “disagree” options? Some evidence of a small bias towards left-side options. No evidence of significant bias whether start with positive option or negative in a survey (different when face-to-face due to “social desirability bias”). Should I give people the option to NOT respond to a question? I Don’t Know – I Have No Evidence – I Have No Experience – I Have No Opinion. SurveyMonkey: “By allowing respondents to opt out due to non-familiarity with the question or if it does not apply to their situation, this will help increase the response rate and quality of the collected data.” Exclude the question if too many respondents will be unable to respond. How will high “don’t know” responses impact your interpretation of the results? Omit if it’s impossible NOT to have an opinion on or experience of the issue.

29. Sequencing the questions “Respondents asked to answer questions dealing with fraud and waste in government programs, were subsequently MORE likely to oppose welfare programs than those who were asked the questions in the reverse order.” Meeting Review Content we covered Actions you will take What’s left to be done (Logistics)

30. Ok, …. one ‘Tip on using SurveyMonkey’ To allow multiple entries from same computer or IP address Create the survey Click ‘Collect Responses’ Select ‘Web Link’ & click ‘Next Step’ Click ‘Change Settings’ Under ‘Allow Multiple Responses’ click ‘Yes’ Click ‘Save Settings’

31. Deployment schedule (time zones, shift patterns) Download Handout 8

32. Analyzing responses ‘Evaluating data based on … demographics’ SurveyMonkey filters What to do about missing responses What response rate & coverage is critical to the goals?

33. ‘Communicating results to the event sponsor’ ‘Improving the presentation of data’ http://www.edwardtufte.com Boston, March 11–14 $380

34. Reporting principles Separate results & interpretation. Low response; how representative are the responses you got? ‘Use-by’ date on the report? Summarize themes in open-ended comments. Respondents’ own words – including typos. .pptx for the report, .docx for the record. Present all the data at end in a Table.

35. Means, standard deviations, confidence interval A = 2.60 : B = 2.95 : 5 point scale : 12 respondents Very Strongly Agree is not 20% > Mostly Agree “I can do that for you, but I’m afraid I’ll have to add $nx2 to my bill”.

36. Presenting the results

37. Format options All responses • Bar chart % responses • Pie chart Changes over time • Line graph • Side-by-side bar chart Comments • Verbatim • Themes • Word frequency www.Wordle.net You can struggle with Excel & CSv or PDFs or ….. Manual Templates!

38. Priorities for a new CEO

39. Hospital Housekeeping

40. Team surveys Team Effectiveness Initiative Implementation

41. Team survey with leader’s response

42. 360 feedback

43. Repeat 360 1 Making condescending or dismissive statements N = Never R = Rarely O = Often F = Frequently 2 Talking over and/or interrupting people N = Never R = Rarely O = Often F = Frequently

44. ‘How to summarize open-ended data’ 1 Read through once List the themes Cut ‘n’ paste sentences to themes Cluster overlapping themes Editing decisions; Every theme? Every sentence, or representative quotes? Edit easily-identified sources? Omit obvious grudges? Edit same point by same person over and over? Correct spelling, grammar?

45. ‘How to summarize open-ended data’ 2 Strengths • Leadership Team members step up & take a leadership role when required. • Leadership Team meetings & conference calls are effective. • Leadership Team members collaborate with & support each other. • Leadership Team members give & receive feedback without becoming defensive or aggressive. • XX is willing to listen & be influenced, & offers significant support. • Leadership Team members are optimistic about the team's future & ability to accomplish it's mission. Noteworthy Limitations • Team goals & responsibilities are not as clear as individual assignments. • Team capabilities and resources are a concern.

46. www.Wordle.net ‘How to summarize open-ended data’ 3

47. Event reviews Goals Assess achievement of objectives Isolate impact of content – process – presenter Assess commitment to Actions Identify need for further support & ‘next agenda’ Items Pre-work relevance & use Content Relevance Presenters: Relevance of examples – Responses to questions –Interactions Clarity of action plan Outstanding issues Personal commitment to change Ongoing support requirements

48. Thank you. Handoutwww.MickVerran.comPasswordBFR mick@mickverran.com 978-505-5275 • This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. You are free to copy and distribute the work under the following conditions: you must attribute the work to “Mick Verran Mick@MickVerran.com” ; you may not use the work for commercial purposes; if you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.