200 likes | 324 Views
Evaluation culture in SEE (Public procurement in SEE innovation evaluations: A comparative and needs assessment study; Lena Tsipouri – Nikos Sidiropoulos University of Athens, Centre of Financial Studies. December 12, 2013, Zagreb. Outline.
E N D
Evaluation culture in SEE (Public procurement in SEE innovation evaluations: A comparative and needs assessment study; Lena Tsipouri – Nikos Sidiropoulos University of Athens, Centre of Financial Studies December 12, 2013, Zagreb
Outline • Does a comparative study and benchmarking make sense (objective/subjective indicators)? • Methodological remarks (literature; research) • Experiences from the EVAL INNO countries • A Synthesis • Lessons from intelligent benchmarking
Does a comparative study and benchmarking make sense • The background of the countries studied differs considerably (Austria-Greece-Hungary-Bulgaria / Montenegro-Serbia) • The legal framework is hardly applicable to the size of the procurement studied • Objective indicators (measurable such as number of evaluations; budgets; frequency, but also qualitative regarding quality and response) are limited • Subjective indicators can be obtained but are subject to criticism (the solution is stakeholder approval)
Methodological remarks (literature) • Common rules for public procurement, as a tool of the Single Market (contracts over 499000; Open procedures, Restricted procedures, Negotiated procedures and Competitive dialogue). Usually RTDI Evaluations fall in the budget range of 50000-500000 Euros • There is no real academic literature; There is a broad number of tender documents and terms of reference available, mostly (but not exclusively) originating in procurement by the European Commission; • Information of the later stages of the procurement, namely monitoring of the evaluations and their acceptance, is practically absent;
Methodological remarks (research) • Borrow notions for benchmarking from the pverall literature on public procuremnt • Quality provision of the service • Knowledge of the international standards and good practices • Knowledge of the local needs and capabilities • Knowledge of international capabilities and interest to respond to national tenders (price and reputational issues) • Good and timely decisions
Public procurement of RTDI evaluations: our approach • Decompose the process: • Identifying the requirements and user readiness • Market intelligence • Tendering process (Terms of reference: background, data availability, questions and methods) • Assessing tenders and awarding contracts • Managing contract delivery • Response to recommendations
The basic dimensions • The institutional set up (formal and informal rules) • Key organisations involved • Tendering process
The institutional set up (formal rules) • The budget thresholds for general provisions for public tendering • The existence (or not) of special provisions for RTDI evaluations (e.g. specific thresholds; individual selection procedures etc.) • Explicit legislation (or not) regarding the legal obligation of awarding authorities to evaluate their programmes or organisations. • The existence (or not) of evaluation standards
The institutional set up (informal rules) • Relevant parameters for launching tenders (strategy issues): • Frequency • Type of evaluations
Key organisations involved • Awarding Authorities (how many, how good, how can they improve) • Evaluators (local, national, international; issues of independence, expertise and reliability for evaluators called for direct or restricted tenders; how good, how can the market evolve) • Other stakeholders (exercise pressure for RTDI evaluations)
Tendering process • Terms of Reference (how good they are/could be) • Smooth process (no legal or other complications) • Time to contract (benchmarks) • Monitoring (hands on or off?) • Content (how ambitious are the Terms of Reference?) • Adoption of recommendations (of the specific evaluation and more in general)
Experiences from the EVALL-INNO countries • Measuring/subjective rating per parameter decomposed • Comments per parameter • Comments per country
Experiences from the EVALL-INNO countries • The European Commission defined PRAG – Practical Guide to Contract Procedures for EU External Actions
What next: • Do countries want to incorporate the benchmarking lessons (what should be their priorities; what is their distance to top; who are the top performers; from whom to learn)? • Should/could the EU play a role in taking the (obvious but documented) results a step further? If yes how? • Are other stakeholders interested in the results (to be read horizontally or vertically)
tsipouri@econ.uoa.gr nikos.sidiropoulos@gmail.com