Innovation Surveys: Advice from the Oslo Manual

1. Innovation Surveys:Advice from the Oslo Manual South Asian Regional Workshop on Science, Technology and Innovation Statistics Kathmandu, Nepal6-9 December 2010

2. Ch 8 OM - Innovation Survey Procedures • Guidelines - collection and analysis of innovation data; • Comparable results; • Particular circumstances may require other methodology comparability!

3. Populations • The target population: • Business enterprise sector (goods-producing and services industries); • At a minimum, all statistical units with at least ten employees.

4. Populations • The frame population: • Units from which a survey sample or census is drawn = frame population; • Basis: last year of the observation period; • Ideal frame = up-to-date official business register established for statistical purposes  NSOs; • If the register forms the basis for several surveys (innovation, R&D, business), the information collected can be restricted to issues specific to innovation.

5. Survey methods • Mandatory surveys increase response rates; • Census or sample survey? • Sample surveys should be representative of the basic characteristics of the target population (industry, size, region) a stratified sample is necessary; • Census, though costly, might be unavoidable in some cases (legal requirement, small frame population, inclusion of all units in the frame with a certain number of employees).

6. Survey methods • Domains (sub-populations) are subsets of the sampling strata; • Potential sub-populations: industry groupings, size classes, regions, units that engage in R&D and innovation-active; • Guidelines: • Same statistical units and classifications; • Consistent methods; • Documentation of deviations in data treatment or differences in the quality of the results (from the domains).

7. Survey methods • Sampling techniques: • Stratified sample surveys (reliable results): based on the size and principal activity of the units; • Sampling fractions should not be the same for all strata: the sampling fraction of a stratum should be higher for more heterogeneous strata and for smallerstrata. • Cross-sections: standard approach - new random sample drawn from a given population; • Alternative/supplementary approach: panel data.

8. Survey methods • Suitable respondents: • Various methods: postal surveys, electronic surveys,personalinterviews; • Questions are very specialised and can be answered by only a few people in the unit; • It is highly recommended to make a special effort to identify respondents by name before data collection starts.

9. Survey methods • The questionnaire: • Pre-test before fieldwork; • Simple and short; • Order of the questions; • Questions on qualitativeindicators - binary or ordinal scale; • International innovation surveys: translation and design of the questionnaire; • Short-form questionnaires for units with little/no innovation activity previously reported.

10. Survey methods • Combination of Innovation and R&D surveys: • Reduction in the overall response burden; • Scope for analysing the relations between R&D and innovation activities at the unit level; • Efficient method of increasing the frequency of innovation surveys; • It is possible to obtain reliable results for R&D expenditures; • Longer questionnaire; • Units not familiar with the concepts of R&D and innovation may confuse them; • The frames for the two surveys are generally different.

11. Survey methods • Guidelines for conducting combined surveys: • The questionnaire should have two distinct sections; • Individual sections for R&D and innovation should be smaller than in separate surveys; • Comparisons of results from combined surveys with those from separate innovation surveys should be done with care; • Surveying methodsshould be reported; • Samples to carry out such surveys should be extracted from a common business register.

12. Estimation of results • Weighting methods: • The simplest one is weighting by the inverse of the sampling fractions of the sampling units, corrected by the unit non-response; • Stratified sampling technique with different sampling fractionsweights should be calculated individually; • Commonly based on the number of enterprises in a stratum; • In international and other comparisons, be sure that the same weighting method is used.

13. Estimation of results • Non-response: • Unit non-response: a reporting unit does not reply at all; • Item non-response: response rate to a specific question / % of blank or missing answers among the reporting units; • Disregarding missing values and applying simple weighting procedures based only on the responses received implicitly assumes that non-respondents are distributed in the same way as respondentsbiased results; • Possibility: imputation methods to estimate missing values on the basis of additional information.

14. Presentation of results • Inferential analysis: • Drawing of conclusions about the target population; • The results should give a representative estimation of the situation; • Weighted results; • Unit non-response rate is very important. • Descriptive analysis: • Description of the statistical units in terms of their innovative or non-innovative activities without drawing any conclusions about theunderlying survey or target population; • No generalisation of the results; • Unit non-response rate is of minorimportance.

15. Presentation of results • Errors: • Random errors due to the random process used to select the units; • Systematic errors containing all non-random errors (bias); • Results’ variance: it is recommended to calculate both (average) values for innovation indicators and also their coefficients of variation and/or confidence intervals; • Results presentation:metadata (including information on data collection procedure),samplingmethods,proceduresfordealingwithnon-responseandqualityindicators.

16. Frequency of data collection • Innovation surveys: every two years; • When not economically feasible  three or four years; • Surveys must always specify an observation period for questions on innovation; • The length of the observation period for innovation surveys should not exceed three years nor be less than one year.

17. Annex A - 5. Methodological issues for developing country contexts • Information system specificities: • Relative weakness of statistical systems • Absence of linkages between surveys and data sets; lack of official business registers information from other surveys cannot be used; • Involvement of NSOs; • When lacking, basic variables about firms’ performance can be included in the innovation survey - to enable further analysis.

18. Annex A - 5. Methodological issues for developing country contexts • General methodological considerations: • Questionnaire design: • Separated sections - different respondents; • Guidance / definitions; • Language and the translation of technical terms; • Survey application: • In-person; • Trained personnel.

19. Annex A - 5. Methodological issues for developing country contexts • General methodological considerations: • Frequency: • Every three to four years (e.g., timed to CISrounds); • Try to update a minimum set of variables every year; • The purpose of surveys needs to be clearly stated and the questions clearly formulated; • An adequate legislative base for the collection of innovation statistics can help ensure the success of such an exercise; • The results should be published and distributed widely.

20. Thank you! http://www.uis.unesco.org l.marins@uis.unesco.org (CIS:http://www.oecd.org/dataoecd/37/39/37489901.pdf)