Cognitive testing CIS 2013 Belgian findings

1. Cognitive testing CIS 2013Belgian findings Anneleen Bruylant, Petra Andries and Machteld Hoskens ECOOM Presented at het Eurostat CIS Task Force meeting, 3-4 May 2012

2. The original question

3. The question aims at getting an estimate for the expenses firms had in year X to perform their innovation activities with regard to the four given areas.Firms often gave their total expenses though, rather than their expenses for innovation. • The question was rephrased, and Belgium decided to test the longer version. Issue to resolve

4. In 2012, howmuchdidyour company spend on the followingactivities or acquisitions? Thenpleaseestimate the share of eachactivity or acquisitionthat was meantfor product or processinnovation, as definedearlier. ActivityExpenditures 2012Estimated share of activityfor product and/or process innovation • Internal R&D _____________€ ____% (includinginvestmentsin property and equipment, personnelandrelatedcostsspecificallymeantfor R&D) • Purchase of external R&D _____________€ ____% • Purchase of property, buildings, _____________€ ____% machineryand equipment (excludingpurchases made for R&D) • Purchase of software _____________€ ____% • Purchase of externalknowledge _____________€ ____% (includinglicenses) • Training paidby the employer_____________€ ____% • Advertisement, market campaigns, _____________€ ____% market research • Design activities_____________€ ____% (in-house andexternalcontracts) Revised question: Version 1

5. Version 1 was tested on nine subjects, six French-speaking, three Dutch-speaking. • Main issues: I. Some subjects read the question vertically, instead of horizontally. In the first column they filled in the expenditures specific for innovation, rather than general expenditures. This way, they sought to make the sum of the percentages add up to 100%. So ironically, subjects answered the new question as if it were phrased in the old way, not seeming to notice what the revised question was asking. This problem occurred in 4 out of 9 cases. Results of version 1

6. II. “Design” is too vague a term, as people didn’t know what is meant by it (four out of nine subjects). Different industries had a different take on the term, varying from understanding “design” as referring to everything from user interface to fliers. The vagueness of the terms used (“innovation”, “R&D”) was mentioned several times. III. One of the subjects (an R&D manager) thought the question was difficult to answer, as he didn’t know the total expenditure (only the innovation expenses), and would thus have to gather the information. Also, calculating the percentages was seen as something very difficult to do.

7. In 2012, how much did your company spend on the following general activities or acquisitions? Then please estimate the share of each activity or acquisition that was specifically meant for product or process innovation, as defined earlier. Activity Expenditures 2012 • Purchase of property, buildings, machinery and equipment: _____________€, of which ____% for product or process innovation (excluding purchases made for R&D) • Purchase of software: _____________€, of which ____% for product or process innovation • Purchase of external knowledge (including licenses): _____________€, of which ____% for product or process innovation • Training paid by the employer: _____________€, of which ____% for product or process innovation • Advertisement, market campaigns, market research: _____________€, of which ____% for product or process innovation • Design activities: _____________€, of which ____% for product or process innovation (in-house and external contracts) • Internal R&D: _____________€, of which ____% for product or process innovation (including investments in property and equipment, personnel and related costs specifically meant for R&D) • Purchase of external R&D: _____________€, of which ____% for product or process innovation Version 2

8. We moved R&D to the end of the question, hopingthiswould make the context of the question lessfocused on R&D andinnovation, so subjects wouldn’tanswer the question the way itusedtobephrased. • We highlighted the words “general” and “specifically” in the question, hopingthiswouldemphasize the intent of the question. • We testedthisversion on onlyone subject. Results of version 2

9. There was still confusion about the percentages: the subject read the question vertically, not horizontally. The subject admitted it was actually very clear once it was pointed out to her there was a second column to the question. • There was also confusion about what to include in the innovation expenses, and what not to include. This was the case for “purchase of buildings, machinery and equipment”, “publicity”, and “design”.

10. In 2012, how much did your company spend on the following general activities or acquisitions? • Then please estimate the share of each activity or acquisition that was specifically meant for product or process innovation, as defined earlier. Version 3

11. We split the question in two, thinking that this would make the purpose of the question clearer. • The question remained the same as in version 2, only the lay-out of the question was changed. • This version was tested on eight subjects. Results of version 3

12. Three subjects remarkedthat the way the question was phrased made itnecessarytoinvolveseveralpeopletoanswer the question, instead of justone person. • One person stillread the question vertically, the othersevenreaditcorrectly. • The exact meanings of the following terms were unclear: “external R&D”, “design”, “external knowledge”, “innovation”, “new product”, and “training”.

13. “Innovation” and “R&D” were seen as interchangeable terms, which they are not. This was exemplified by the remark one subject made, namely, that the clause “exclude purchases for internal R&D” was confusing, as we later asked for the percentage dedicated to innovation (and “innovation” was seen as synonymous with “R&D”). • Some subjects were not entirely clear about the distinction between “design” and “internal R&D”, or between “software” and “IP”. Also, the distinction between “external R&D” and “external knowledge” was not clear to several subjects.

14. Two subjects pointed out that firms may use a narrower definition of “R&D”, thus under-reporting the actual R&D expenditures. This is because the person who fills in the questionnaire is most likely the same person who applies for R&D subsidies, which are granted based on a narrower definition of “R&D” than the one used in our questionnaire.

15. Although the need for rephrasing the original question was abundantly clear, our experience showed that subjects – when faced with the revised question – either answered as if they were answering the original question, or – when noticing the difference – remarked that the new version was more complicated, as they would now have to involve several people to answer the question. conclusions

16. Clearer definitions of the terms used are necessary, and they could be repeated for clarity on all relevant pages. • We propose rephrasing the original question again, as the revised question seemed to confuse subjects more, or make it more difficult to answer. Possible rephrasings for consideration could include: