1 / 23

The Evaluation Phase

Understand the evaluation phase in research funding. Learn about independent experts, proposal assessment, criteria ranking, and impact analysis. Discover common evaluator irritants and key evaluation ratings.

brumfield
Download Presentation

The Evaluation Phase

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Evaluation Phase Juras Ulbikas

  2. Evaluation – Overview

  3. Evaluation – Overview • Selection of independent experts (=evaluators) • Submitted proposals • Eligibility check • Assessment (by evaluators) • Ranking • Report to coordinator

  4. Evaluation – Overview • Selection of independent experts (=evaluators) • From database • Balance of gender, specialism, countries, industry vs academic, etc… • Submitted proposals • Eligibility check • Pre-screening of data • Call objective • Assignment to evaluators

  5. Evaluation – Who are the evaluators? • Highly-educated people • Specialist vs. generalist • Technically competent • Impact issues • Knowledge of English • Professional integrity • Evaluations on a personal basis • Professional, independent, impartial, and objective

  6. Evaluation – Who are the evaluators? • Contract between evaluator and Commission • Conflict of interest • Confidentiality • An expert is human! • Novelty • Feasibility of the concept • Logical flow of arguments • European dimension • Consistency

  7. Evaluation – Assessment • First impression • Show that sufficient time and care has been spent on the proposal • Layout and structure • Tables, graphs. text margins, font size • Win or lose in the first two pages • The “instant kill”

  8. Evaluation – Common evaluator annoyances • Unnecessary repetition • Too much cross-referencing • Unclear intents of the proposers • Inconsistencies and no logical links • “Statements” and “conclusions” to be believed on the basis of the assumed authority of the applicants

  9. Evaluation – Ratings

  10. Evaluation – Ratings • Gender issues and ethical issues • They do count but not formally rated • Weights may be applied • At the moment seldom applied

  11. Evaluation – Ranking

  12. Evaluation – Criteria

  13. Evaluation – S&T Quality • Soundness of concept and quality of objectives • matches the Work Programme? • clear objectives? • Progress beyond the state of the art • State of the art clearly described? • Quality and effectiveness of the S&T methodology and associated work plan • Sound and logical S&T approach, with clear technical route?

  14. Evaluation – Implementation • Management structures and procedures • Project management demonstrably of high quality? • Quality of the individual participants • Well-suited and committed participants ? • Quality of the consortium as a whole • Complementarity between consortium participants? • Allocation and justification of the resources to be committed (budget, staff, equipment)

  15. Evaluation – Impact • Contribution to the expected impacts listed in the Work Programme • Impact on reinforcing competitiveness or on solving societal problems • Clear added value in carrying out the work at European level • Account of research activities at national and European level initiatives. • Dissemination and/or exploitation of project results, and management of IPR

  16. Evaluation Summary Report – ESR (S&T Quality) • From a “Research for the benefit of SMEs” project: 4.5 • […] The state of the art is clearly described, systematic and provides a good assessment of flaws in current soft heeled shoes demonstrating a high level of scientific and technological innovation supported by a patent. As a result, the contribution to advancement of knowledge / technological progress is clear and based on a patented concept. […] • […] There is a good risk analysis with appropriate control through milestones having go/no-go decision at M12. […]

  17. Evaluation Summary Report – ESR (S&T Quality) • From a “Research for the benefit of SMEs” project: 3.5 • […] The innovation level is not very high but it is a step forward in building technology which can be very useful if broadly used.[…] • […] the focus of the project is still too much on basic research and not enough on its concrete application (e.g. no prototype is foreseen).[…]

  18. Evaluation Summary Report – ESR (S&T Quality) • From an NMP CP-IP project: 3.0 • […] Unfortunately, no details are given to be able to appreciate the progress beyond state-of-the-art regarding the weaknesses of existing technologies. The important energy consumption aspect for the envisaged equipment has not been clearly addressed. […] • […] The project has unclear targets regarding material systems to be selected and tested (see for example WP1 & 2) in specific applications for which real advantages are expected. […] • […] The proposers should have conducted prior lab scale investigations on certain material combinations […]

  19. Evaluation Summary Report – ESR (Implementation) • From a “Research for the benefit of SMEs” project: 4.0 • The quality of the management structure and procedures is excellent. • A great number of measures is considered for succesful implementation of the project and they are all very clearly presented and defined and focused on the needs of the project. • The conflict resolution measures are realistic and workable. • The role of the coordinator demonstrates clear leadership is properly defined and appropriate experience is demonstrated. There is a good vertical integration within the SME partnership and, although some are very small, their roles are appropriate and competences relevant. • Overall this project got 13.0, and made it.

  20. Evaluation Summary Report – ESR (Implementation) • From a “Research for the benefit of SMEs” project: 4.0 • The project shows a good structure of the management with experienced partners that have already participated in other related projects. • The consortium is trans-national but the partners from one of the participant countries seem to have an over-weighted role. • Overall this project got 11.5, and didn’t make it.

  21. Evaluation Summary Report – ESR (Implementation) • From an NMP CP-IP project: 3.0 • The profiles of individual participants and personnel resources have been well presented. • There is no demonstration of the management capabilities of the Coordinator. • Another notable weakness is the lack of any procedures for resolving conflict in the various committees. • Regarding exploitation, the proposal only presents general principles and does not enter into the specifics of the work programme. • Overall this project got 10.0, and didn’t make it.

  22. Evaluation Summary Report – ESR (Impact) • From an NMP CP-IP project: 4.0 • […] A serious weakness of the proposal is the lack of any in-depth financial analysis for the growth of the industrial sectors related to processing and material development. In this respect, energy consumption for the running of the equipment which is a key issue in materials processing is not addressed. […]

  23. Evaluation Summary Report – ESR (Impact) • From a “Research for the benefit of SMEs” project: 4.0 • […] The project shows good exploitation and dissemination plans that covers academic society, industrial stakeholders and experts. • However, no demonstration activities are foreseen in order to show the real potentialities of the developed system. • The management of the IPR should be described in more detail.[…]

More Related