The Benefit-of-the-doubt Approach to KBE-Composite Indicators

1. The Benefit-of-the-doubt Approach to KBE-Composite Indicators Laurens Cherchye, Wim Moesen, Nicky Rogge, Tom Van Puyenbroeck Leuven, 5-6 September 2006

2. Structure Background : Lisbon, KBE, OMC, KEI (Essence of) Formal approach Illustrative example Possible extensions / future work

3. Lisbon Agenda / Lisbon Strategy The “new strategic goal” = “to become the most competitive and dynamic economy in the world capable of sustainable economic growth with more and better jobs and greater social cohesion”. Achieving this goal requires an overall strategy aimed at • preparing the transition to a knowledge-based economy and society by better policies for the information society and R&D, as well as by stepping up the process of structural reform for competitiveness and innovation and by completing the internal market • modernising the European social model, investing in people • applying an appropriate macro-economic policy mix

4. Political engines for the Lisbon Strategy Maria Rodrigues, 2003, European Policies for a Knowledge Economy. “The implementation of any strategy requires a political engine with the power to coordinate policies and adapt them to each national context. The Lisbon decisions made this governance centre stronger, in three ways: • Stronger role for European Council as co-ordinator (devoting the Spring Council to monitoring the Strategy) • Broad Economic Policy Guidelines with improved synergy between macroeconomic, structural and employment policies • the adoption of the Open Method of Co-ordination, stepping up the translation of strategic priorities into national policies

5. Open Method of Co-ordination “This Method (…) involves: (…) • Establishing, where appropriate, quantitative and qualitative indicators and benchmarks against the best of the world and tailored to the needs of different member states as a means of comparing best practices • Periodic monitoring, evaluation and peer review” • (origin of “list of Structural Indicators”) “A fully decentralized approach will be applied in line with the principle of subsidiarity” (Lisbon European Council, Conclusions, 2000)

6. “to develop and improve indicators for the knowledge economy, including composite indicators” • What phenomenon do we measure? WP1 • indicator selection WP2/4/6 • Data quality assessment/statistical analysis WP3/4/6 • Building composite indicators WP5 • Robustness and Sensitivity WP7

7. What phenomenon do we measure? Knowledge-based economy/society • Although we all have a fairly good idea what it comprises, there is no clear-cut and universally agreed upon “model” that aptly describes the KBE

8. What phenomenon do we measure? • Knowledge policies • E.g. the OECD’s (2001) five policy recommendations: • Seizing the benefits of ICT • Harnessing potential of innovation and technology diffusion • Enhancing human capital • Fostering firm creation and entrepreneurship • Getting the fundamentals right

9. 2. indicator selection KEI current database: over 100 (potential) indicators

10. 2. indicator selection “classical performance indicators” “new economy indicators” kbe “internationalisation indicators”

11. Production/diffusion of ICT HR, skills, creativity Production/diffusion of knowledge Innovation, entrepreneurship New economy-indicators “classical performance indicators” “new economy indicators” kbe “internationalisation indicators”

12. New economy-indicators • Economic impact of ICT • Share of ICT employment • Share of ICT sales • Investment in ICT • Patents • Internet Use by firms • Businesses receiving orders over internet • Internet use by individuals • Government ICT • % individuals/firms who use internet to interact with public authorities Production/diffusion of ICT HR, skills, creativity “new economy indicators” Production/diffusion of knowledge Innovation, entrepreneurship

13. KBE-indicators Income productivity Employment and welfare “classical performance indicators” environmental Production/diffusion of ICT HR, skills, creativity “new economy indicators” kbe Production/diffusion of knowledge Innovation, entrepreneurship “internationalisation indicators” Trade “International aspects of knowledge production” International asp. of human resources

14. Outline for a KBE-composite • Must not hide ‘ambiguity’ of underlying phenomenon in a so-called “simple” construction method • Must take expert opinion into account, while recognizing that agreement on a unique and fixed weighting scheme is the exception rather than the rule. • Within the confines set by experts, allow subsidiarity to play its role • Create composites for “subcategories” (ICT, “the fundamentals”,…) • Robustness assessment

15. Composite Indicators Indicators are observed (measured in different units) We want an overall ‘score’ in order to compare countries… …but weights for aggregation are not known with certainty

16. Classical DEA-problem Inputs and Outputs are observed (measured in different units) We want an overall expression of productivity in order to compare firms… • Productivity = (weighted sum of outputs) / (weighted sum of inputs) …but weights for aggregation are not known with certainty • (production function, right prices,… unknown)

17. « The typical composite indicator » • For country c, given j = 1,…,m policy dimensions • Xcj: (normalised) variable of country c in dimension j • wj: weight . (Normally 0 wj 1 and sum wj =1)

18. Towards DEA composite indicator • Weighted sum of indicators = ‘country total’ • Since the eventual purpose of a composite indicator is to compare (with other countries), we will express this ‘country total’ RELATIVE TO A SIMILARLY WEIGHTED SUM of BENCHMARK SUB-INDICATORS

19. DEA composite indicator, Step 2  • Which benchmark should we choose? • Look WITHIN SAMPLE of COUNTRIES for the one that yields the HIGHEST POSSIBLE TOTAL (given the weights) • ‘Best Practice’ notion

20. Step 3: Benefit-of-the-doubt weighting • What weights should we choose? • CHOOSE the weights such that the evaluated country has a MAXIMAL COMPOSITE INDICATOR VALUE • ‘Benefit-of-the-doubt’-notion

21. Advantages Qua model: better description of reality than (e.g.) equal weighting Embedded concern for national diversity: no other weighting scheme yields higher composite indicator value (political acceptance in EU « open method of co-ordination »-context) Principle is easy to communicate • Since we are not sure about the right weights, we look for « benefit of the doubt » weights (such that your overall relative performance index is as high as possible) • If ANOTHER country gets a higher overall score, using YOUR weighting scheme, we regard that country as outperforming you.

22. Benefit-of-the-doubt + expert opinion • The BoD approach seems especially valuable when individual expert opinion is available, but when experts disagree about the right set of weights • Foster and Sen « On Economic Inequality »: • « While the possibility of arriving at a unique set of weights is rather unlikely, that uniqueness is not really necessary to make agreed judgments in many situations, and may indeed not even be required for a complete ordering » Must take expert opinion into account, while recognizing that agreement on a unique and fixed weighting scheme is the exception rather than the rule.

23. = ≥ ≥ ≥ GDP Patents Publ school e-com Appending expert opinion • Several types of “Pie Share restrictions” can be appended to Example 1: consensus on order of pie shares

24. Appending expert opinion • Several types of “Pie Share restrictions” can be appended to Example 2: pie shares and limited flexibility

25. Outline for a KBE-composite • Must not hide ‘ambiguity’ of underlying phenomenon in a so-called “simple” construction method • Must take expert opinion into account, while recognizing that agreement on a unique and fixed weighting scheme is the exception rather than the rule. • Within the confines set by experts, allow subsidiarity to play its role through benefit-of-the-doubt weighting • Create composites for “subcategories” (ICT, “the fundamentals”,…) • Robustness assessment

26. Outline for a KBE-composite Income productivity Employment and welfare “classical performance indicators” environmental Production/diffusion of ICT HR, skills, creativity “new economy indicators” kbe Production/diffusion of knowledge Innovation, entrepreneurship “internationalisation indicators” Trade “International aspects of knowledge production” International asp. of human resources

27. KBE-indicators “new economy indicators” • R&D • GERD expenditure/GDP • Business R&D/GDP • BERD/value added in industry • % of BERD in service industries • … • Patents • Bibliometrics • … Production/diffusion of knowledge Each raw indicator comprises at least 5% and at most 50% of R&D share in knowledge composite

28. KBE-indicators “new economy indicators” • R&D • patents • Bibliometrics • … Production/diffusion of knowledge As in previous step

29. KBE-NEW ECONOMY composite Production/diffusion of ICT Each basket : “+/- equally important” HR, Skills, Creativity Production/diffusion of knowledge Innovation - Entrepeneurship Organisational indicators

30. ‘New Economy’ composite

31. ‘New Economy’ composite Finland: 100 Belgium: 58.3

32. Outline for a KBE-composite “classical performance indicators” Each basket : “+/- equally important” “new economy indicators” kbe “internationalisation indicators”

33. ‘KBE’ composite Internationalisation Economic/social New economy

34. Critiques / extensions/ remarks • Further data screening needed • Expert opinion (?) • Robustness assessment • Benchmarking after the KOK-report? • (Lisbon 2: “no naming and shaming”) • or “last exit to Lisbon” ?