The Evaluation Process, Tips from an Evaluator’s Point of View

1. The Evaluation Process, Tips from an Evaluator’s Point of View Leonardo Piccinetti EFB

2. OUTLINE • Evaluation principles • The experts, who are they ? • Role of Commission staff • FP7 evaluation process • FP7 Evaluation Criteria

3. Basic principles • Excellence.Projects selectedfor funding must demonstrate a high quality in the context of the topics and criteria set out in the calls. • Transparency. Funding decisions are based on clearly described rules and procedures, and applicants should receive adequate feedback on the outcome of the evaluation of their proposals. • Impartiality.All proposals submitted to a call aretreated equally. They are evaluated impartially on their merits, irrespective of their origin or the identity of the applicants.

4. Basic principles • Confidentiality.All proposals and related data, knowledge and documents communicated to the Commission are treated in confidence. • Efficiency and speed. Evaluation, award and grant preparation should be as rapid as possible, commensurate with maintaining the quality of the evaluation, and respecting the legal framework. • Ethical and security considerations:Any proposal which contravenes fundamental ethical principles, or which fails to comply with the relevant security procedures may be excluded at any time from the process of evaluation, selection and award

5. The experts, who are they ? • The Commission draws on a wide pool of evaluators (database) in all scientific fields • c. 50,000 in FP6 • Experts/evaluators from ICPC are very welcome! • Calls for “candidates” • Call for applications from individuals; and from institutions • Applications via CORDIS (database of experts) • A mass-emailing of FP6 experts was sent • A simple tick-box will ensure registration for FP7 • Commission invites individuals on a call-by-call basis • Not self-selection! • Expertise, and experience are paramount • Geography, gender and “rotation” also considered

6. Independent experts • Expert evaluators are at the heart of the FP7 system • Expert providesindependent, impartial and objective advice to the Commission represents neither the employer, nor the country! • Significant funding decisions will be made on the basis of expert advice • The integrity of the process is crucial • Experts have to read the Code of Conduct annexed to the appointment letter…and follow it!

7. Independent experts • Experts agree to terms and conditions of an “appointment letter” • Typically, an individual will review 6-8 proposals “remotely”…. • …then spend a couple of days in Brussels • Some will participate in “hearings” with the consortia • Travel and subsistence reimbursed • Plus €450 honorarium per day • Experts sign confidentiality and conflict of interest declaration • Names published after the evaluations

8. Actors Confidentiality • The content of proposals, or the evaluation results, can’t be discussed with anyone • The sole exception: in the presence of the EC moderator with experts who are evaluating the same proposal in a consensus meeting group or final panel • Is not possible to distribute any documents related to the evaluation of a proposal, or take any documents from the evaluation building • Note:The Commission publishes names annually, but as a group – no link between expert and proposal

9. Conflicts of interest (2) • Types of COI set out in appointment letter • Check the exact wording! • Disqualifying COI • Involved in preparation of proposal • Stands to benefit directly • Close family relationship • Director/trustee/partner • Employee of a partner in a proposal • Member of Advisory Group • Any other situation that compromises impartiality • Potential COI • Employed within the last 3 years by a partner in a proposal • Involved in research collaboration with proposers in the previous 3 years • Any other situation that casts doubt…or that could reasonably appear to do so…

10. Role of Commission staff • Check the eligibility of the proposals • Oversee work of experts • Moderate discussions • Organise the panel and its work • Ensure coherence and consistency

11. Evaluation • Peer-Review System • Two-stage evaluation procedure • Remote evaluation • Evaluation on a non-anonymous basis • Unless otherwise specified in call for proposal • Register as an Evaluator • https://cordis.europa.eu/emmfp7/

12. Role of Commission staff • Commission staff may advise on: • Background on previously supported or on-going projects • Relevant supplementary information (directives, regulations, policies, etc) • Evaluation rules • Key points within the Work Programme,e.g. issues related to “Relevance” • Commission staff may not introduce: • New elements (cannot fill in “gaps” in proposals) • Interpretations

17. Writing an FP7 proposal is NOT just a creative process for A NICE IDEA • It requires to SHOW scientific, technological and depth knowledge of the subject • You must present references, legislatures, previous work and experience

18. Be precise and to the point • Use drawings to show your methodology at once • Use bold types in phrases that you would like to emphasize • Present cohesion and interaction among work packages

19. Full Proposal Proposal forms Overview of the Evaluation Process “remote” may be “remote” Submission Individual reading Consensus Panel Finalisation Final ranking list Evaluators Evaluators Evaluators Rejection list Criteria Criteria Criteria Proposals in suggested priority order Eligibility COMMISSION COMMISSION Role of experts

20. FP7 evaluation process IAR 1 CONSENSUS MEETING RECOMMENDANTION FOR FUNDING OR NOT IAR 2 CONSENSUS REPORT BY THE RAPORTER IAR 3 PANNEL MEETING – RESULTS ARE SEND TO APPLICANT EVERYIAR IS 3 TO 4 HOURS 0 - the proposal fails to address the issue under examination or can not be judged against thecriterion due to missing or incomplete information 1 - poor 2 - fair 3 - good 4 - very good 5 - excellent

21. RELEVANCE QUALITY OF THE CONSORTIUM 1. CONCEPT – CAPABILITIES OF THE PARTNERS QUALITY OF COORDINATION QUALITY OF MANAGEMENT MOBILISATION OF RESOURCES 2. ACTIVITIES – AND CAPACITY TO DO THE TASK POTENTIAL IMPACT 3. THE TARGET GROUPS THAT BENEFIT

22. Process Evaluation - Individual reading(Will be done remotely) The experts: • Evaluate the proposal individually • without discussing with the other evaluators • Check whether the proposal is‘in scope’ • Complete an Individual Evaluation Report (IER) form giving comments and scoreson all criteria • Sign and date the form • IERs should be checked and, if necessary, returned with a request to further justify the score given • Scoresmustbe in line with comments

23. Process Evaluation - Consensus (1) • Built on the basis of the individual evaluations • The aim is agreement on scores and comments • Usually involves a discussion • 1st part may be carried out remotely • “Outlying” opinions need to be explored • Not just a simple averaging exercise • It is quite normal for individual views to change • Moderated by a Commission staff-member • helps the group reach a conclusion • provides information if necessary • does not contribute opinions

24. Process Evaluation - Consensus (2) • A rapporteur is appointed, who is responsible for drafting the consensus report (CR) • includes consensus marks and comments • The quality of the CR is paramount • It is not often changed at the panel review stage • The aim is: • a clear assessment of the proposal, with justification • clear feedback on weaknesses & strengths • To be avoided: • scores that don’t correspond with the comments • recommendations in view of resubmission

25. The Panel Review • EC ask some evaluators in each sub-activity to examine and compare the CRs of every proposal that passes all thresholds • Key function is to ensure consistency • The Panel will recommend for a sub-activitya priority order including final marks and comments for each proposal • Evaluation Summary Reports (ESR) • Any new scores (if necessary) … should be carefully justified • Ranking of proposals with identical consensus scores • Prioritise certain criteria? • Consider overall balance? • Budget? • Clear guidance for contract negotiation

26. Process Commission Follow-up • Evaluation summary reports sent to applicants • “initial information letter” • Redress procedure • Draw up final ranking lists • Information to the Programme Committee • Contract negotiation • Formal consultation of Programme Committee (when required) • Commission decisions • Survey of evaluators • Independent Observers’ reports New for FP7

27. The evaluation criteria • Criteria adapted to each funding scheme and each thematic area • specified in the work programme • Three main criteria: • S&T Quality (relevant to the topic of the call) • Concept, objective, work-plan • Implementation • Individual participants and consortium as a whole • Allocation of resources • Impact • Contribution to expected impacts listed in workprogramme • Plans for dissemination/exploitation

28. Process FP7 Evaluation Criteria Applicable to ALL funding schemes

29. Process Funding schemes • Collaborative projectsSupport to research projects carried out by consortia with participants from different countries, aiming at developing new knowledge, new technology, products, demonstration activities or common resources for research.The size, scope and internal organisation of projects can vary from field to field and from topic to topic.Projects can range from small or medium-scale focused research actions to large-scale integratingprojects for achieving a defined objectiveProjects may also be targeted to special groups such as SMEs, Specific International Co-operation Actions, etc. Template of the IER for your information(double click on the object) Guidance for evaluators – Call FP7-ENV-2008-1

30. Process FP7 Evaluation Criteria Collaborative projects

31. Process Funding schemes • Coordination & support actionsSupport to activities aimed at coordinating or supporting research activities and policies (networking, exchanges, trans-national access to research infrastructures, studies, conferences, etc).same scope and objectives as in FP6 Template of the IERs for your information(double click on the object) Guidance for evaluators – Call FP7-ENV-2008-1

32. Coordination& support actions

33. RELEVANCE • Examines if the objectives of the work programme are met • The proposer must read in between lines • It must be clear what the commission wants from the call • Good practice: attend info day, inside information • Covering the objectives u get a 4, additional objectives might give u a 5 • TIPS: • Do not copy paste the objectives from the work programme • U must copy and specialize them to the concept of the proposal • FIND THE KEY WORDS (ie networking, knowledge transfer) relating the key word to the objective • TRY TO QUANTIFY OBJECTIVES ( i.E. Not just networking – but networking of three clusters) • Always make a direct reference under each objective with the related work packages of how this objective will be achieved

34. Proposal relevance tips • Identification of impact from Call • Definition of strategic goals for our proposal which clearly fulfill the expected. • Definition of main actions/activities clearly related to the strategic goals • Work Packages for actions/activities.

35. QUALITY OF CONSORTIUM Examines the experience of the consortium in relation to the objectives and the work need to be done, complementarities between partners, suitability of work undertaken, and geographical coverage TIPS Always include a description in the beginning of consortium description indicating the role of each partner and their added value to the project. It is preferred to have inside the consortium mix Member States, New member states and Associated states indicating transfer of good practices to new member and associated states. Under each partner description indicating the excellences (previous projects, research activities and experiences) and of the partner related to the proposal followed by the CVs of key persons involved and their role in the project These excellences should be summarized into an excellence key areas of the consortium in relation to the work need to be done Indicate the logic behind consortium geographical and context selection (why these partners are selected)

36. The coordinator • There is no limitation related to the size, nature, legal status and the years of existence for the proposed coordinator. • In the management forms (A1, A2 and A3) the turnover is presented. The is a indirect but very important effect to the decision of the Commission.

37. QUALITY OF COORDINATION • Examines coherency of the proposal. If there is a methodology that defines the interrelationships between the work packages and the work tasks • The plan must convince the evaluators that is a bottom down approach from the implementation plan to the work packages • TIPS • Start with the relation of the objectives to the work need. • Define your methodology of work mentioning entities (not yet work packages) • Define the interrelationships of the WPs and the work tasks inside the work packages • Your methodology should have a clear description of impact creation in the target groups and European Added Value through the actions of the dissemination plan • Do not include management packages, just coordination packages this would be explained in another section. • YOU MUST ALWAYS START WITH THE METHODOLOGY OF WORK AND THEN WRITE THE WORKPACKAGES • Work packages should mirror the work in the implementation plan with CLEAR milestones and derivables • In general you could exceed the number of pages asked from the commission

38. General tips • Every text with more than three paragraphs must begin with a 2-3 lines abstract with a clear reference to what the remaining text presents. • In most cases the opinion of the evaluator is coming from this abstract. • Phrases 8 words, paragraphs max 3-4 lines.

39. QUALITY OF MANAGEMENT Examines the experience and management mechanisms of the management unit The evaluator examines the capacity of the coordinator to handle the tasks They like to have a strong centralized management and not very complex structures They like to see decision making mechanisms from steering committees TIPS Always start indicating the central management control indicating the experience of the coordinator to undertake the tasks in similar projects Illustrate decision making mechanism (simple but partner inclusive) Show contingency plan with risks (i.e. what happens if partner withdraws) Show IPR management Show quality assurance mechanisms Show knowledge management with in partners. For each of the above relate them with management instruments (web sites, consortium agreements, meetings, etc)

40. MOBILISATION OF RESOURCES Examines the coherence of the financial plan to handle the tasks, relating to the allocation of work in PM within work packages and among partners. The financial plan must be delivered with the proposal although there are no predefined forms from that (except A3, deliverable lists) TIPS ALWAYS state the CVs of key persons involved with the project and their role Illustrate other resources for each organization that are allocated to the project in the partnership description (ie equipment, previous work done etc) Subcontracting is always examined in detail (why and who will do it?) Avoid large differences in work allocation between partners If a partner gets more than 30% of the budget (management and coordination) is negative Management no more than 7%

41. POTENTIAL IMPACT Examines the impact (results) to target group defined, and the European Added value and mainly your dissemination plan Relates to the previous work done in European and National Level TIPS Always start with the clear definition of the proposal target groups indicating the potential impact in quantitative terms if possible Use structured bulleted writing and not abstract descriptions The dissemination plan should provide evidence of European added value in quantitative terms (not general instruments i.e. web page but exclusive actions such as connection with specific initiatives, work groups etc) Always illustrate impact in relation to previous work done – specific national and European programmes indicating the approach mechanisms (even it is obvious) The main evaluation criteria is the dissemination plan. The above issues should be described as work to be done in the dissemination work package

42. ABSTRACT Should start with objectives followed by the consortium logic Then should describe the implementation plan with the expected results and finish with the impact through the dissemination plan

43. Self-assessment process • Most of failed proposals are taking a low mark in 1-2 evaluation criteria (relevance is the most usual). • The pre-evaluation of the proposal by internal or external executives (definitely not involved in the proposal preparation process) based on the published evaluation criteria.

44. Conclusion • Become evaluator is the best way to learning to write proposals • Understanding how EC works • Networking • Well paid

45. Thank you! Leonardo Piccinetti