1 / 18

Dir S ERC/European Commission RTD, Directorate S

The European Research Council. The evaluation of the ERC-2007-StG Call. Dir S ERC/European Commission RTD, Directorate S. IDEAS Programme Committee, January 31, 2008. Overview. Status of evaluation report Key data – both stages The eligibility process Budgets by domains and panels

arich
Download Presentation

Dir S ERC/European Commission RTD, Directorate S

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The European Research Council The evaluation of the ERC-2007-StG Call Dir SERC/European CommissionRTD, Directorate S IDEAS Programme Committee, January 31, 2008

  2. Overview • Status of evaluation report • Key data – both stages • The eligibility process • Budgets by domains and panels • The panel meetings and the ranked lists per panel • The consolidation by the Panel Chairs • Recommendations on grant levels • Feedback to applicants • Status of redress

  3. Status of evaluation report • Will beconform to that of other programmes • Offeringsamefacilities to PMC members • Wedid not succeedhaving a fullyqualitychecked document for this meeting. However, key data tables are distributed • FormalE.R. statisticalreportingwillbebased on the sample of 201 proposals in the main list – by convention • Informalreporting, including public dissemination, isbased on a sample of the top 300 proposals • This is areasonableestimate of the numberthatmaybefunded

  4. Key data- number of proposals by evaluation step 368 ineligible Submitted stage 1 9167 5 withdrawn Evaluated stage 1 8794 8235 rejected Selected stage 1 559 5 not submitted to second stage Submitted stage 2 554 1 withdrawn 2 passed away, 4 ineligible 201 in main list 116 in reserve list 113 reserve: reject for no budget 117 rejected: below thresholds 547 Evaluated stage 2

  5. Key data: Evaluation processStage 2: 559 proposals expected 554 7 Reception of proposals Eligibility and withdrawals INDIVIDUAL ASSESSMENTS Panel members Referees 1 day INTERVIEWS 2 to 3 days PANEL CHAIR MEETING PANEL MEETINGS 20 panel-ranked lists 547 Single Consolidated list

  6. The eligibility process Eligibility decisions have been taken conform the standard FP7 methodology - and in accordance with its ‘case-law’ Some of the decisions have been complex, and have required discussion in parallel with the review We attempt to minimise the number of such cases However, given the numbers and the time-constraint, partial parallel processing of review and eligibility is necessary – and is in accordance with the Rules

  7. Budgets by domains and panels, based on adopted / published call budget, 289.5M€ Additional budget contributions add to reserve 289.5M€ 80% 20% 8 PE panels 7 LS panels 5 SH Reserve Nominal panel budgets, proportional to sum of stage-1 requested grant 2 3 4 … 1 E.g. PE Additional budget E. g. PE4 Main list Reserve ……….. Reject Fits in nominal panel budget, Score > 9, no ordering Candidate for reserve budget, Ranked in priority order, with scoring convention Not fundable, Score < 8

  8. The stage-2 interviews / panel meetings • Panels operated independently, typically 3 days • Interviews with applicants perceived as extremely useful complement to individual assessments • Panels identified four groups of proposals: • Main list proposals, inside nominal panel budget • Reserve list, serious candidate, priority ordering • Reserve list, good proposal, clearly outside budget • Rejected proposal, failing threshold • Panels tagged inter-disciplinary, cross-panel / domain proposals • For groups 1 and 2, panels recommended the grant level

  9. The panel chair consolidation meeting • Purpose of the meeting: to establish a consolidated ranking of the serious candidate reserve (group 2) proposals • With special emphasis on inter-disciplinary proposals • Given 20 panels across all scientific fields, not a trivial affair …. • Because no absolute excellence standard exists: small score differences are meaningless across panels • To faciliate, a starting point ranking was needed • Not constraining the freedom of the panel chairs • ‘Bureaucratically fair’: purely based on an algorithm • Accounting for different panel and proposal sizes • Without prejudice to any possible differences of excellence between panels

  10. The consolidation method (simplified) By convention, all panels scored their serious reserve candidates at 8.9, and sub-ranked them: first, second, …. Nominal budget Res 1 Res 2 Res 3 Res 4 Res 5 Res 6 Res 7 Res 8 Res 9 ….. 1.0 Normalise each panel on its nominal budget Accumulated grant Normalised accumulated grant Ordering by normalised accumulated grant Tagged as inter-disciplinary

  11. The final ranking by the panel chairs • Regarding inter-disciplinary proposals, the panel chairs: • Considered that they did not have the resources to re-examine these in detail • Confirmed their confidence in the prior work of the panels – ‘mainstreaming’ • Regarding the ranked list, the panel chairs unanimously adopted the starting point proposed by the Commission as a fair result • This ranking has not been modified, neither by Commission nor by the Scientific Council

  12. The issue of recommended grant levels - I • Panels’ remit included providing recommendations on award levels • Individual panels faced some difficulties in this respect: • Proposals not overly detailed, over-estimates by PI or host … • Incomplete understanding / interpretation of the Rules of Participation • Contradictory information during interviews • Driven by this, and by different ‘needs of the field’, panels arrived at different solutions: • No grant reductions; reductions across the board; big reductions on some proposals • Some panels have removed the PI salary for PI’s with permanent positions – problematic in view of Rules and of grant mobility • Some panels have calculated the grant ‘bottom-up’

  13. The issue of recommended grant levels - 2 • The ‘reductions issue’ was discussed in the Panel chair meeting • In general, panel chairs confirmed the positions taken by their individual panels as fair and reasonable in the context of the field • A small number of corrections was introduced during the meeting • On the specific issue of the PI’s salary, panel chairs realised the difficulty of the situation and recommended that the Commission applies appropriate corrections • The Commission has applied a correction to all cases where the PI’s salary was explicitly removed • The Scientific Council has strongly endorsed the position of the panel chairs, and requested the Commission to award the grants accordingly, without negotiation │ 13

  14. Towards granting • The Commission has maintained the possibility for successful applicants to seek redress against the level of awarded grant, using the redress procedure • In reality, no such request for redress has been received • The Commission has been cautious in its feedback to applicants • Pending formal decisions on 44.5M€ of third-country contributions • In view of the pending redress • Granting has started on the main list – 201 proposals • About 40 ethical reviews are ongoing • The expectation is that about 300 grants will be awarded

  15. Transparency towards applicants • Four messages to the four groups of applicants: • 201 proposals in main list: granting is imminent, started for 65 • 116 serious reserve candidates; all have score 8.9; probability of a grant varies from 1 to 0 down the ranking • 113 good proposals but clearly outside budget; score between 8.0 and 8.8; clear information given • 117 proposals fail threshold; score < 8.0; clear information given • Subject to individual disclaimers, list of all 430 proposals above threshold is now published on ERC web-site • Most of the 116 serious reserve candidates can make a reasonable assessment of their probability

  16. Status of redress • The ‘Redress’ procedure has worked well • For Stage-1, 245 requests for redress received, three main areas: • Eligibility – none sustained • Factual errors by reviewers – 15 sustained • Scientific judgement – none sustained 15 stage-1 proposals were re-reviewed by panels • One applicant has been invited to submit a stage-2 proposal Stage-2 redress: has just started • 27 requests were received

  17. Details of stage-1 redress • Scientific judgement of panels 150 • The review process 20 • Wrong panel 6 • Conflict of interest 14 • Very near to threshold 53 • Negative, offensive 25 • Due to editing / English 18 • Discriminatory 7 • Eligibility 49 • Factual errors by reviewers 26 • Rebut reviewers / complain size of proposal 6 NOTE: a single redress request may address multiple categories │ 17

  18. Main lessons from stage-2 • Overall, the process worked well: consolidation effective • Interviews successful, to be maintained for StG • Review the framework in which panels operate • Respect autonomy, specificities of scientific fields • But more need for coherent decision-making • The ‘mainstreaming’ approach to inter-disciplinarity has worked, but reflection needed: • It does not give much visibility • Difficult to achieve inter-panel coherence • Reviewers comments in feedback to applicants: • Reveals some flawed judgements – redress • Transparency inevitably drives improvement │ 18

More Related