1 / 97

INNOVATION IN RESEARCH

INNOVATION IN RESEARCH. By JON PULESTON VP Innovation GMI Interactive. WHAT HAVE WE LEARNT ABOUT CONDUCTING ONLINE RESEARCH?. WHAT SHOULD YOU BE WORRYING ABOUT WHEN CONDUCTING ONLINE RESEARCH?. A MINDFIELD OF ISSUES. Speeding. The impact of question styles. Age balance of sample.

illiana-hamilton
Download Presentation

INNOVATION IN RESEARCH

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. INNOVATION IN RESEARCH By JON PULESTON VP Innovation GMI Interactive

  2. WHAT HAVE WE LEARNT ABOUT CONDUCTING ONLINE RESEARCH?

  3. WHAT SHOULD YOU BE WORRYING ABOUT WHEN CONDUCTING ONLINE RESEARCH?

  4. A MINDFIELD OF ISSUES Speeding The impact of question styles Age balance of sample Translation issues Survey design effects Cultural differences Gender variance Intra Country differences Intra panel differences Panel quality issues Untruthfulness

  5. AN INTERNATIONAL STUDY HOW? 2 waves across 15 countries 12,000+ responses Mixed panel sources Age & Gender quotas 22 mini-research experiments Split samples with varied question design styles Speed measurements of every question Respondent level analysis of data variance Unique lie detection process EXAMINED THE IMPACT OF: GENDER AGE PANEL SOURCE SPEEDERS LIARS COUNTRY QUESTION STYLE

  6. FOR EXAMPLE IMAGINE IF YOU HAD AN ALL MALE SAMPLE…

  7. FOR EXAMPLE IMAGINE IF YOU HAD AN ALL MALE SAMPLE…

  8. DATA VARIANCE BETWEEN MEN & WOMEN… Benchmark Data variance of 5.9%* v *Average level of data variance for likert range questions

  9. AVERAGE LEVEL OF VARIANCE BY FACTOR *Average level of data variance for likert range questions

  10. UNDERLYING CULTURAL DIFFERENCES

  11. Japanese Camel Indian Camel

  12. EXAMPLE ANSWER VARIANCE: AGREE DISAGREE RATIO’S

  13. DIFFERENCES ARE COMPOUNDED BY LANGUAGE ISSUES…

  14. LANGUAGE ISSUES…

  15. LANGUAGE ISSUES…

  16. CROSS CULTURAL FACTORS ARE THE MAIN UNDERLYING CAUSE OF DATA VARIANCE

  17. LYING LYING UNTRUTHFULNESS OVER REPORTING

  18. OVER REPORTING HAS HIGH IMPACT ON LOW INCIDENCE STUDIES…

  19. SPEEDING IS THE NO.1 CAUSE OF DATA VARIANCE AT THE END OF A SURVEY

  20. RESPONDENTS DON’T THINK MUCH Dimensions of Data Quality, Eggers & Puleston: ESOMAR Congress 2012

  21. WHEN WE ARE BORED WE TEND TO ANSWER LIKE THIS IN THE UK = slightly agree = slightly like = slightly appealing Source: GMI Example of aggregate date from set of 650 range questions

  22. THIS LEADS TO BADLY DIFFERENTIATED DATA How much do you like watching these sports on TV? Source: GMI Anecdotal example

  23. COMPARE THIS TO BEHAVIOURAL REALITY FOOTBALL OTHER SPORTS

  24. HOW TO TACKLE THIS?PEOPLE INITIALLY STARTED PLAYING AROUND WITH USING WIDER RANGES

  25. BUT FOUND THIS JUST MADE THE HUMPS MORE HUMPIER!

  26. 7 point scale Source: GMI Source: GMI Anecdotal example

  27. 9 point scale Dislike alot Like a lot Source: GMI Source: GMI Anecdotal example

  28. AND DID NOT REDUCE THE LEVEL OF PATERN ANSWERING Source: Dimensions of data quality Puleston & Eggers ESOMAR 2012

  29. WHAT YOU NEED TO FIND A WAY TO GET THEM TO STOP & ACTUALLY THINK!

  30. BY EMOTIONALISING THE PROCESS WE FOUND YOU COULD ENCOURAGE PEOPLE TO THINK MORE AND THIS IN TERN IMPROVES THE DATA

  31. FLATTEN THE HUMPS AND REDUCE PATTERN ANSWERING Patter answers Source: Dimensions of Data Quality Puleston & Eggers ESOMAR 2012 research extract

  32. WE ALSO DISCOVERED THAT THE ERGONOMICS OF QUESTION DESIGN COULD IMPROVE FEEDBACK TOO… +30% consideration time

  33. FLATTER HUMPS & MORE CROSS CONSISTENT DATA Source: Can gaming techniques cross cultures? Puleston & Rintoul ESOMAR 2012

  34. EXAMPLE OF THE ERGONOMIC DESIGN OF QUESTIONS

  35. QUESTIFICATION We then started to realise the impact of how you worded questions & pitched questionnaires could have a big impact

  36. How much do you like watching these sports on TV? Imagine you were in charge of the coverage of Sport on TV…

  37. THESE THINGS LED TO WHOLE RANGE OF BREAKTHROUGHS IN QUESTION DESIGN UNDER THE BANNER OF “GAMIFICATION”

  38. LEADING TO MORE ENTERTAINMENT BASED WAYS OF DESIGNING SURVEYS

  39. CASE STUDY: RE-INVENTING A RETAIL EVALUATION SURVEY

  40. BASED AROUND THE CONCEPT OF CHALLENGING RESPONDENTS TO DESIGN THEIR PERFECT SUPERMARKET

  41. VISUALISED THE QUESTIONS

  42. WE PLACED THEIR CURRENT SUPERMARKETS ON TRIAL

  43. RE-ENGINEER THE SEGMENTATION QUESTIONS INTO A SHOPPER PERSONALITY TEST Feedback was given to the respondent to tell them what broad shopper segment they were part of

  44. Respondent Enjoyment score Original version = 7.3 Gamified version = 8.5 Bottom 5% Top 5%

  45. Dropout comparisons Reduced dropout by 2/3rds

  46. Data granularity: respondent level standard deviation Showed consistent improvements in data quality

More Related