Sampling and non sampling errors in the Italian Television Audience Measurement system

1. European Conference on Quality in Official Statistics- Q2008 Participants to research group: De Vitiis, D’Alò, Di Consiglio, P.D. Falorsi (chief), S. Falorsi, Orsini, Pallara, Russo, Seeber, Tuoto Sampling and non sampling errors in the Italian Television Audience Measurement system Speaker : Alessandro Pallara Istituto Nazionale di Statistica Rome, 9-11 July 2008

2. Outline of the talk • Television Audience Measurement (TAM) and the “meter panel” • Survey parameters and sampling design • Estimation of sampling error • Sources of bias in TAM estimates • Measurement errors: E&I • Panel attrition and conditioning • Comments and concluding remarks European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

3. Television Audience Measurement • Television Audience Measurement (TAM) data have a high social and economic impact. • Essential information to: • Broadcasters,for programming policy and programme scheduling • Broadcastersand advertising agencies, for agreeing upon the price of commercial air-time and advertising campaigns European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

4. Context and purposes of the research The context for this research is the agreement signed in 2006 between Italian NSI and the Italian Communications Regulatory Authority (Agcom), so that Istat has been appointed for carrying out a study on the statistical methodology behind the national TAM system. • Purposes (and Research reports) • review current estimation procedures for estimating daily ratings and associated sampling errors (released June ’07); • 2)evaluate accuracy of the survey estimates with respect to the various sources of non sampling errors(Dec. ’07); • 3) put forward tools and recommendations for checking statistical quality (both sampling and non sampling errors) of the output of TAM survey(under release, July ‘08) European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

5. Standard TAM methodology • Current worldwide standard in TAM methodology has two basic features: • a viewing household panel sample(the People meter panel)selected according to certain household demographic characteristics (age of the householder, number of components, city size, geographical region) • a measurement device (the people meter) that register (a) TV set status (i.e. which channel is being tuned to with certainty) and (b) viewer presence, which isquite demanding on panelists (i.e. pressing their remote control button each time they enter or leave a television viewing session) European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

6. Survey Parameters Let r denotes a generic TV channel and T a given time interval (daypart, day, week) Main Parameters The Audience is the average number of individuals (homes or target groups) viewing a TV channel over a given time interval (e.g. programme, daypart). European Conference on Quality in Official Statistics - Q2008 The Reach (or cover/cume) is the cumulative percentage or total (usually expressed in thousands) of a population that has been counted as viewers at least once during a specified interval. Rome, 10 July 2008

7. Survey Parameters (cont.ed) TheShare (of Audience) is defined as the percent of Households Using Television (HUT) or Persons Viewing Television (PVT) which are tuned to a specific program or station at a specific time. European Conference on Quality in Official Statistics - Q2008 The Ratingisthe size of television audience relative to the total universe, expressed as a percentage Rome, 10 July 2008

8. Survey population, statistical units, data analyzed • Survey population : members of household aged 4 or more • Survey estimates refer to in-home TV viewing (persons and households, including viewing of guests of the sample households), of total population and selected target subpopulations • Elementary data used for estimating parameters • Individual viewing statement: meter records (raw data) converted after data processing into summary statements of individual viewing over time (each minute). Each statement contains information concerning (a) Start and end time of the viewing session; (b) identification of signal source and TV set being viewed; (c) identity of viewer • Data analyzed • Raw and validated panel meter micro-data (daily data for 4 weeks between Sept. ’05 through June ‘06) • Population total of auxiliary variables and sampling weights used in the estimation procedure European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

9. Sampling design of most TAM survey • Two phases in TAM sampling strategy • In the first phase, a face-to-face interview (the Establishment Survey, ES) is carried out each year, based (in 2006) on a sample of approximately 30,000 households and using a two-stage stratified sample: •  provides certain universe estimates (in terms both of individuals and household) which will be used in the TAM estimation procedure, such as education attainment, socio-economic status or number of children per household, •  provides a database of potential households for recruitment in the second phase sampling • In the second phase a panel of about 5100 households, is “broadly” randomly selected (within control strata) from ES respondents (the people meter panel sample). European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

10. The Meter panel sample Meter panel selected characteristics (used for panel turnover control), Active vs. Lost Panelists, Compared to Total population benchmarks European Conference on Quality in Official Statistics - Q2008 * Estimated through ES Rome, 10 July 2008

11. Problems with TAM sampling design in Italy • quota sampling • unknown selection probability of units from the recruitment households database (originating from different ES’s) • rules for field substitution of non responding households: different contact rates between basic households and substitutes, interviewer may influence substitutions • very high total (non response to ES + refusal to panel recruitment) non response rate: >90% • Non respondents may be different as for the amount of television viewed: light viewers out of home a lot and less available for interview, light viewers may feel their cooperation less important European Conference on Quality in Official Statistics - Q2008 . Rome, 10 July 2008

12. Approach to measuring accuracy of TAM estimates • MSE of an estimator of an unknown population parameter • Approach to quality assessment : • Direct (smooth) estimators of the sampling variance • (Indirect) indicators of the Bias European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

13. Estimation of sampling variance European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

14. Variance estimation – an example Estimates of Audience(each minute)and Coefficient of Variation (CV) for a large channel of the public network – 4 Sept., 2005 European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

15. Variance estimation – an example (cont.d) Scatter plot of CV by Audience Size (minutesanddayparts) European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

16. Sources of bias in TAM estimates • Potential sources of bias in meter panel sample • coverage errors (e.g. non-TV homes not included in estimates, ≈ 1,500,000 est. persons in Italy) • (wave) non responses • model assumptions errors • measurement errors • attrition andpanel conditioning European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

17. Measurement Errors in meter panel data • measurement errors: mis-match between a signal source of a TV set being viewedand a person registered as a viewer through the people meter Main sources of measurement errors (data gathering and editing phases) • Meter statements indicating that the TV set is switched on, but without any persons registered as present (uncovered viewing). • Long viewing session without any change in registered set use or viewer presence (signing on/off of viewing individuals, channel switching, long/constant viewing) • TV OFF viewing • Same individual registered as a viewer for two or more TV sets at the same time(concurrent viewing) • Undue or wrong re-assignment of uncovered viewing to a household component (processing errors) European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

18. Processing TAM data – E&I • Editing checks: • rejection of certain panel households from the daily reporting samples because of suspected faulty compliance by panelists[excess (24 hours) viewing, long/constant viewing above set threshold values] • records of individual viewing are canceled out (concurrent viewing, overnight constant viewing, unassigned uncovered viewing) • records of individual viewing are edited in (uncovered viewing assigned to viewer) European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

19. Processing data - Editing and Imputation Percent variation of audience estimates (unweighted) from raw to validated data resulting from treatment of uncovered viewing European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

20. Processing data - Editing and Imputation Percent variation of audience estimates using different cut-off values and criteria for deletion of records with long constant viewing European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

21. Panel attrition European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

22. Attrition rates by subgroup of population European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

23. Months-in-sample percent distribution of households panel sample Panel attrition and conditioning European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

24. Age effects Daily estimates of audience (thous. of individuals) of satellite TV channels by some dayparts (4 weeks between Sept. ’05 through June ’06) – households below and abovemedian of time-in-sample European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

25. Comments and concluding remarks - 1 • Sampling errors • The CV decreases as the estimation increases. • The larger estimates (major networks) quite reliable • The lower estimates (local networks) quite unreliable • The CV slowly decreases as the size of time interval of estimates increases • Non sampling errors • coverage errors related to list problems (non-TV homes, non-voting resident households, ……..) • non standardized criteria for substitution of non responding households to ES may lead to select in the panel heavy viewer households • some evidences of the presence of an upper bias in survey estimates: editing checks seems to be unbalanced towards editing viewing statements in rather than out, threshold values for considering long viewing as unrealistic result in canceling out viewing statements only in the case of overnight viewing • the lack of an upper limit to time-in-sample for households in the panel suggests the presence of panel attrition and conditioning because of modifications in panelist viewing behavior and compliance with the measurement device during their presence in sample European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

26. Comments and concluding remarks – 2 • Recommendations for Improving quality • coincidental surveys on a regular basis to check real viewing status of panelists vs. registered meter data • occasional surveys of non respondents to analyze independence of response mechanism from viewing behavior • introducing a method for panel rotation, with an upper limit to time-in-sample of panel households European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008

27. Thank you for your attention! European Conference on Quality in Official Statistics - Q2008 Rome, 10 July 2008