Exit Polling Conference, Moscow 2011

1. Exit Polling Conference, Moscow 2011 Exit Polling in UK, Ukraine and Georgia Nick Moon Managing Director GFK NOPSocial Research, UK nick.moon@gfk.com

2. A perspective (mainly) from the UK • Exit Polls in the UK since 1989 • Two exit polls in Ukraine • One exit poll in Tbilisi Georgia

3. Exit Poll Problems – issues (UK) We are really predicting the result of over 600 separate elections, and can’t cover each one The only local level information available on actual voting is at ward level, and only for local elections Local level turnout is far lower and behaviour often very different This makes stratification of the sample by voting virtually impossible NB Local level data is available in Greece, Ukraine and US But not Tbilisi because of a change in precincts

4. Exit Poll Problems - practical Not always easy to approach people immediately after they have voted Although response rates are high (>80% in UK) there is still a high risk of politically differentiated non-response The weather!

5. Exit Poll Problems - solutions Use as many polling stations as possible Develop good relations with the authorities Interviewers estimate the voting behaviour of all people approached Can be used to weight for differential non-response Use the same polling stations as the previous election Can look at change rather than simple totals Eliminates effect of bias IF bias is constant (Big IF)

6. The 2010 GB Exit Poll • Conducted jointly by GfK NOP and Ipsos MORI • Conducted on behalf of BBC, ITN and Sky • 130 polling stations • Most used at previous exit polls • Approx 18,000 interviews in all • Just measuring claimed current vote

7. The Practicalities (1) • Polling runs from 7am to 10pm • Split into two equal shifts • Each has an interviewer and a “clicker” • “Clicker” uses preordained sampling interval to select respondent • Interviewer estimates voting behaviour, then asks voter to fill in an exit poll ballot • No replacement for refusals

8. The Practicalities (2) • Ballot designed to replicate actual one as closely as possible • Has candidate names and parties • In alpha order as on real ballot • Individual results phoned back regularly to team of telephone interviewers • Speed essential, especially in the evening • 130 interviewers started phoning in at 20.45, and all the data were in by 21.05

9. The Practicalities (3) • Aggregate data at level of time by polling station produced at regular intervals, plus individual level data • Results weighted to take into account any discrepancy between intended and actual number of interviews in each hour • Collected by the BBC analysts from SFTP site • Run through various algorithms • Initial projection of result at 10pm • Final projection at 11pm

10. The BBC/ITN exit poll 2005 • Forecast majority 66 seats • Actual majority 66 seats

11. The BBC/ITN/Sky exit poll 2010

12. Moving to the East….

13. Weather • Ukraine has no provision to conduct elections only in moderate weather seasons  • 2009-10 winter was one of harshest in last decades with -10-20° temperature and heavy snowfalls from end of December till mid-February

14. Scope of work • 300 polling stations, team of 3 interviewers • 20,000 interviews, from 8am – 8pm • Each respondent will fill in the questionnaire and put it into the special box with the Client’s logo. • Telephone interviewers called each polling station at regular intervals to collect the results • Interim data provided by 19.30, and final data by 22.00 • Data provided to the Client for all Ukraine, and for regions

15. Challenges • GfK Ukraine had no experience in conducting exit polls • We were not sure about abilities and integrity of our field force • Weather was against us with heavy snowfalls • We had no chance to check our weighting system in advance • The client had strict requirements in terms of timing

16. Planning the sample of polling stations • The formal way • Sampled with probability proportional to size by region and settlement size • The informal way • If no an acceptable road connection, or no mobile coverage, we replaced it with one of the same size within 10-20 km • If over 200 km from the oblast centre we replaced it with one of the same size within 30 km that had train connection with oblast centre • each polling station inspected to ensure that it is convenient for exit polling - easy to count voters leaving the station, convenient place for interviewers by the exit, polling station management is not against polling

17. Putting interviewers under control • We used SIM-cards to • link phone number to exact polling station • minimize telecom costs • control all moves of interviewing team • The most important moments: • start of the election day (“Have they arrived?”) • end of election day (“Are they still there?”)

18. Problems during election day • Some (few) teams arrived late • 2 or 3 stations were changed because of aggressive polling station staff; in some cases our Field Dept. senior staff successfully negotiated with them by phone • Many interviewers turned out to be unable to calculate number of exiting voters with needed precision • We underestimated time needed for “interviewing interviewers”, it resulted in delays in data collection • After 17.00 (especially in second round) some teams in rural areas and in small towns decided to leave but were stopped and turned back by Control Department

19. Second Round First Round Results • First round: 25,105 interviews! Second: 20,708. • First round: • Second round:

20. Second Round First Round Success • Conducting exit poll for a first time, we had the second most correct poll for both rounds among 6 exit polls conducted during these elections • Error of only 0.5% on the gap between the two candidates in second round, and the average error in the poll on all three possible options was a tiny 0.2%. • In the first round we were within 1% for all the main candidates

21. Moving further East….

22. Tbilisi mayoral election design Georgian President Mikheil Saakashvili cannot stand for another term when his current term expires The mayor of Tbilisi is seen as a strong contender to succeed him Provided he did well in his own re-election Electorate of around 1 million 738 precincts – all very similar in size Sample of 50 precincts Target of 4,000 interviews

23. Tbilisi mayoral election practicalities Team of two covered precinct for whole voting period Mobile team toured city collecting completed ballots and returning them to central office Questionnaires keyed by data entry team Flow weighting based on count per 2 or 3 hour period Weighting for differential refusal by age and sex

24. Tbilisi mayoral election concerns Recall vote bore little relation to actual previous election Made us worried about differential refusal Though most of parties in previous election no longer existed! Used what information we had to look for clues Used actual election results at precinct level No correlation between refusal and level of support for main opposition at last elections No correlation between refusal and level of gap between recall vote and actual vote

25. Tbilisi mayoral election results We decided against any weighting by previous vote But we did overstate the winner by 3% Probably concern about stating opposition to the government                                   Exit poll          Election result   Difference                  %                      % % Giorgi Ugulava  58                      55                 +3 Irakli Alasania                 19                      19                     0 Giorgi Chanturia                 8                       11                    -3 Zviad Dzidziguri                  7                       8                     -1 Giorgi Topadze                  4                       5                      -1

26. Back to the UK

27. The Main Lesson Learned • The best way to gain a good reputation is to produce a result that no-one believes – and then be right! • Our very first by-election exit poll, was announced with virtual incredulity by BBC host David Dimbleby • It proved to be almost exactly right • Our most recent exit poll was announced with considerable scepticism by BBC host David Dimbleby • It proved to be almost exactly right

28. Exit Polling Conference, Moscow 2011 Thank You Nick Moon Managing Director GFK NOPSocial Research, UK nick.moon@gfk.com