1 / 25

RTD FP7 PROPOSAL EVALUATION Jo Prieur

RTD FP7 PROPOSAL EVALUATION Jo Prieur. Covent Garden, the FP7 Proposal Evaluation Building. General Principles. Evaluation by Commission based on peer review by independent experts, with: Transparency Equality of Treatment, Impartiality & Fairness Confidentiality

gerry
Download Presentation

RTD FP7 PROPOSAL EVALUATION Jo Prieur

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. RTD FP7 PROPOSAL EVALUATION Jo Prieur Covent Garden, the FP7 Proposal Evaluation Building IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  2. General Principles Evaluation by Commission based on peer review by independent experts, with: • Transparency • Equality of Treatment, Impartiality & Fairness • Confidentiality • Efficiency & Speed (???) • Quality/Excellence • Ethical & Security considerations IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  3. Facts & figures • 16 000 proposals / year in FP6, on average • 5000 evaluators/year in FP6, on average • An expert data base of over 50 000 names • Over 26 000 proposals in FP7 (incl.9000 for ERC) for 2007 for a budget of € 6 billion, in 57 Calls. • 6700 evaluators appointed for 2007 (close to 9000 with ERC referees) • A dedicated building (Covent Garden) for evaluations, all year round IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  4. Proposal complete (A & B) Submission via EPSS Overall evaluation scheme May be “remote” Proposal Submission Individual reading Consensus Panel (+hearings) Follow up by Commission Final ranking list Evaluators Evaluators Evaluators Criteria Criteria Criteria Rejection list Questions answers Proposal ranking Eligibility check Expert involvement COMMISSION COMMISSION IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  5. Evaluation steps • Evaluation criteria, scores & thresholds • Selection of evaluators • Commission officials • Proposal eligibility • Evaluator briefing • Individual evaluations • Consensus meetings and reports • Hearings (if applicable) • Panel • Finalisation IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  6. Evaluation criteria, scores & thresholds • 3 major criteria (with funding scheme-dependent sub-criteria) • S/T Quality (concept, objectives, workplan) • Implementation (individual participants, consortium, resources) • Impact (expected impacts listed in WP, dissemination, exploitation) • Scores from 0 to 5 for each criteria (not for sub-criteria) • Individual thresholds (3) for each criteria • Weightings if specified (in general no weighting) • Overall threshold (10) higher than the weighted sum of individual thresholds («being good enough is not good enough») • Relevance (FP6 criteria) integrated in S/T quality and impact…but clear irrelevance (“out of scope”) means “non eligible” IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  7. Scoring scale 0The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information 1 Very poor - The criterion is addressed in a cursory and unsatisfactory manner. 2 Poor - There are serious inherent weaknesses in relation to the criterion in question.knesses that could not be corrected at the later changing the project) 3 Fair - While the proposal broadly addresses the criterion, there are significant weaknesses that would need correcting. 4 Good - The proposal addresses the criterion well, although certain improvements are possible. • Excellent - The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor Half marks may be given…..Full score range should be used! IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  8. Selection of experts/evaluators • Independent expert database made from calls for individual candidates and to organisations to appoint experts https://cordis.europa.eu/emmfp7/ • Experts selected « ad personam » (not representing their employer, nor country, nor any other entity) • Expertise & range of competences are paramount for the choice of evaluators. These are assessed from data (CV) in the database (and possibly from earlier participation in proposal evaluations) • Balance between countries, types and sizes of organisations (large companies/SMEs, industry / research / academia) and gender aspects are taken into account as long as above main criteria are satisfied. • Sufficient renewal (about 30% new experts for each call) • Constraints: availability, avoidance of conflicts of interest, number and coverage of proposals IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  9. Role of experts / evaluators • Experts agree to terms and conditions of an “appointment letter” with clear rules on confidentiality and conflicts of interest. They adhere to a code of conduct. • An individual will review typically 7 to 10 proposals in 4/5 days (for small CPs, including consensus meetings… “productivity” very much instrument-dependent • Some will participate in panel work • Some will participate in hearings • Names of evaluators are published once a year, per theme, after the evaluations http://cordis.europa.eu/fp7/experts_en.html • One evaluator is acting as an independent observer: he “evaluates the evaluation”, assure the process is fair, and make a report and recommendations to the Commission IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  10. Role of Commission Officials • Check eligibility of proposals and put each eligible proposal in a predefined group/category (sub/theme, activity, etc.) • Brief the evaluators and answer relevant questions from them • Oversee the work of the evaluators and ensure a confidential, fair, coherent and consistent evaluation process according to established criteria, rules and procedures. • Control the information available to the evaluators (proposals, earlier evaluation results for resubmitted proposals, etc.) • Collect/Compile individual evaluation results • Moderate the consensus meetings and control quality of the consensus reports (completeness, correctness, fairness) • Do not attempt to influence the opinion of the evaluators nor express views about any proposal or applicant. • Organise panel meetings and hearings IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  11. Eligibility Check • Timely proposal receipt (before closing date and time) • Minimum number of independent partners as requested by the rules of participation, or the call, or the work programme • Proposal completeness (all parts and forms) • Formal submission as requested in the call (e.g. electronic submission only via EPSS) • Any other specific criteria (i.e. no classified information in Security proposals) • Out of scope • Out of budget limits Difficult cases submitted to an internal eligibility committee New for FP7 New for FP7 IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  12. Evaluation groups • Eligible proposals grouped into categories according to Work Programme (sub-theme, activity, area, etc.) • Evaluation group of experts assigned to each category • Number of evaluators in each group based on expected workload (number of proposals for the group and expected «productivity» of the experts: typically from 2/ 3 readings a day for a small CP to 1 reading in 2 days for a large CP or IP, including consensus) Difficult exercise due to uncertainty on the expected number of proposals IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  13. Evaluator briefing by the Commission • Explain what is expected from the evaluators (evaluation guidelines, objectives, procedures, criteria, scoring, thresholds, weightings, timing) • Explain the role of the observer and of the rapporteurs • Answer questions from evaluators • May provide general information about the call such as overall number of proposals, over-subscription ratio… or recommendations (typically: be fair but severe, use full score range, be coherent between scores & comments, etc.) • Explain specific situations (e.g. handling of sensitive information for Security proposals) IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  14. Individual evaluations • Number of independent readings: from 3 minimum for a small CP to 5 for a large CP • Evaluators should declare any potential conflict of interest before starting reading • Individual scores & comments, based only on the written proposal (no reading between the lines!) are entered into an IER (Individual Evaluation Report) • No discussion between evaluators at this stage • No contact with any partner of the proposals, no telephone, no computer, all documents locked in the evaluation room in the evening, etc. • Under the scrutiny of the Commission assisted by the independent observer IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  15. Consensus • For each proposal, all evaluators come together for a consensus meeting chaired by the moderator • Meeting aims at an agreed view on the proposal, and comments and scores accepted by all (typical duration: from < 0.5 hour, up to several hours). It is a true discussion/debate, not a mere averaging exercise! • In case of lasting disagreements, additional evaluators (up to 3) can be called in. • The consensus report is written by the rapporteur (one evaluator or the moderator), signed by all evaluators and moderator. • The moderator ensures that the discussion is coherent, complete and fair, that each evaluator can express his views, that conclusions are properly reflected in the consensus report and that conclusions do support the scores. IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  16. Hearings (Large CP, IP or NoE only) • Hearings take place a few days after the write up of the consensus report, only for proposals above all thresholds • Objective is to clarify the proposal (and not to modify or rewrite, or to make a show) • Hearing based solely on a set of questions prepared in writing by the evaluators during the consensus meeting. • Hearing conducted by the same evaluators as the initial evaluation (plus extra experts on specific issues if necessary) • A hearing report is prepared. • If necessary, the consensus report can be modified (scores & comments) in the light of the hearing results IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  17. Panel • Key function is to ensure overall consistency • Composed of a limited number of evaluators of each group and moderated by a Commission official • Review and discuss all consensus reports and arrive at a collective consensus on each proposal (harmonisation and standard of scoring between evaluation groups) • Write an ESR (Evaluation Summary Report) for each proposal (scores & comments), not necessarily identical to the initial consensus report…but deviations must be justified. • Establish and sign the evaluation report containing all ESR and recommendations on proposal ranking (split equal rating proposals), possible clustering of proposals, grant agreement negotiation, etc. IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  18. Ethical Review • Triggered by the experts or the Commission • Carried out on the proposal submitted • Only exceptionally will additional information be requested • The ER considers, for example: • ethical aspects and the social impact of the research • respect for the FP7 ethical standards • application of relevant European Directives, conventions etc • whether the consortium is seeking the approval of relevant local ethics committees • balance between the research objectives and the means IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  19. Finalisation New for FP7 • Approval of ESR and issuing ESR to coordinators (initial information letter, redress procedure) • Establishment by the Commission of the final list taking into account evaluation results, in consultation with other relevant services of the Commission. • Programme committee information and discussion • Establishment of the lists of projects for negotiations and of a reserve list taking into account budgetary constraints. • Notification of rejected proposals to coordinators • Preparation & approval of an implementation plan • Notification of successful proposals to coordinators and start of negotiations. IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  20. Redress Procedure • Included in the rules of participation by the Council • Complaints to be submitted within 1 month from the initial information letter (before selection/rejection decisions) • Complaints to be based only on eligibility and the evaluation process • Calling into question the judgement of appropriately qualified experts is not permitted • A redress committee will examine the complaints • Redress procedure rarely leads to re-evaluation (4 case out of 700 in 2007) • Benefits: coherent and consistent approach to complaints, transparency & equal treatment • Difficulty (for the Commission): labor-intensive IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  21. Keep in mind (1) • Fail to prepare= Prepare to fail. Time is of the essence. Start early, do not wait for the call to come out: read important documents, even in draft form, and build up your partnership. • The 2 “bibles” to be read carefully are the work programme and the guide for applicants… Read them carefully… and read the right ones! • Read also how your proposal will be evaluated ftp://ftp.cordis.europa.eu/pub/fp7/docs/fp7-evrules_en.pdf • Understand the structure of the Work Programme (Activities, Areas, Topics) and the “jargon” (Levels 1 & 2, Funding Schemes, etc.) • Do not ignore the general parts of the WP: they give you contextual information and references which will help you positioning your project within a broader policy, society and/or strategy context (useful for the “impact” aspects) • Build up your partnership: no alibi-partner. All must have a clear function and bring a real added value to your project IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  22. Keep in mind (2) • Do not deviate from the required structure of the proposal (the template). It is designed to ensure practicality & commonality, to match the evaluation criteria, and make life of the Commission officers and evaluators easier • Do not write too much: over-long proposals are rarely viewed in a positive light…the limits on the number of pages is important. Avoid repeating entire paragraphs. • Do not write too little. Cover all what is required in part A (the forms) and B (the text) of your proposal. • Do not assume, but demonstrate. • Leave nothing to the imagination of the evaluators: they are not allowed to have any! They are instructed to make their judgments solely on the basis of what they read, nothing else!... IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  23. Keep in mind (3) • Don’t forget your major objective at the submission stage: convince well informed and educated evaluators • Who have been selected to make judgements against 3 equal criteria • But…who may not be specialists of your particular topic • And…who are just human beings: they do not know everything, they get tired by the end of the evaluation week, etc. • Use simple words. Avoid specialist jargon. Be concise and accurate. Avoid acronyms or explain them • Don’t waste time with coloured graphics or images. Evaluators have only black & white copies to look at. Make sure that illustrations and graphics, if any, are meaningful to the evaluators. IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  24. Keep in mind(4) • All evaluation criteria are important and not only scientific & technological excellence. Impact has become more important in FP7. • Perform an internal evaluation before submitting your proposal (quality assurance), and therefore allow enough time for an iteration. • Submit, re-submit, re-re-submit…only the latest submitted version stays in the system and will be evaluated (if eligible) • Register in the database to become an evaluator: It’s real hard work…but also an occasion to meet new faces, network, and understand what makes a good proposal. It will help you for your future proposal preparation. • For hearings, answer the questions, all questions, nothing but the questions. IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

  25. A typical evaluation room IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

More Related