PROPOSAL EVALUATION: Guidance for evaluators for Calls in FP7 (Framework Programme 7)

1. PROPOSAL EVALUATION: Guidance for evaluators for Calls in FP7 (Framework Programme 7) Sergey Mikhalovsky, Nazarbayev University

2. My recent experience with research funding Co-ordinator, Principal Investigator or Team leader: European Union: FP6-MATISS (2006-2010) FP7-MONACO-EXTRA (2008-2012) FP7-OncoNanoBBB (2012-2015) FP7-ABREM (2010-2014), FP7-FRESP (2009-2012) FP7-Greenland (2009-2012) Interreg IIIA Stent (2005-2008) Interreg IVA Flax (2009-2012) TEMPUS III and TEMPUS IV (2005-2008, 2009-2012) UK: Technology Strategy Board - FullFlush (2010-2013), Department of Health (2008-2013), Medical Research Council (2012-2015)

3. My experience with proposal evaluation FP5, FP6, and FP7 in nanotechnologies, environmental sciences and Marie Curie Programme Erasmus Mundus I and II, Lead Expert in Life Sciences TEMPUS II and III, INTAS, EPSRC (UK), BBSRC (UK), national programmes for Russia, Austria, Cyprus, France, USA and Kazakhstan

4. FP7 Specific Programmes Co-operation – Collaborative Research European Research Council Marie Curie Actions Capacities – Research Capacity JRC EURATOM

5. Co-operation People Ideas Capacities JRC Budget Split 2007-2013 Values in € Millions €1 751 €4 097 €7 460 €4 700 €32 413

6. What about Kazakhstan? Kazakhstan is on the list of International Cooperation Partner Countries (ICPC) And eligible for most FP7 projects!

7. Rules on submission & evaluationBasic principles • Excellence. Projects selected for funding must demonstrate a high quality in the context of the topics and criteria set out in the calls. • Transparency. Funding decisions must be based on clearly described rules and procedures, and applicants should receive adequate feedback on the outcome of the evaluation of their proposals. • Fairness and impartiality. All proposals submitted to a call are treated equally. They are evaluated impartially on their merits, irrespective of their origin or the identity of the applicants.

8. Rules on submission & evaluationBasic principles • Confidentiality. All proposals and related data, knowledge and documents communicated to the Commission are treated in confidence. • Efficiency and speed. Evaluation, award and grant preparation should be as rapid as possible, commensurate with maintaining the quality of the evaluation, and respecting the legal framework. • Ethical and security considerations: Any proposal which contravenes fundamental ethical principles, or which fails to comply with the relevant security procedures may be excluded at any time from the process of evaluation, selection and award

9. The Three MainReference Documents 1 - Rules on submission and evaluation • This is the common and official reference for FP7 rules for submission of proposals, and the related evaluation, selection and award procedures. 2 - Guide for applicants • The Guide for Applicants contains the essential information to guide proposers through the mechanics of preparing and submitting a proposal. • All proposals shall contain a Part A (administrative forms) and a Part B (proposal description). Indications about the content and issues to be addressed are described in the Guides for Applicants • Please make sure that you read the “Guide for applicants” that corresponds to the funding scheme for the topic.

10. The Three PrincipalReference Documents • 3 - The work programme: three complementary documents: • Work Programme - General Introduction • 2012 Work Programme: includes the topic’s description and criteria against which the proposals will be assessed. • Cooperation" Work Programme - General annexes • Annex 1: List of International Co-operation Partner Countries (ICPC) • Annex 2: Eligibility and Evaluation Criteria for Proposals • Annex 3: Forms of Grant and Maximum Reimbursement Rates for Projects Funded • Annex 4: General activities

11. Example: Joint call ENV-NMPWork Programme – Topic description • Nanotechnologies for water treatment Nanotechnology presents many benefits for environmental technology applications, such as remediation, treatment or sensor development and monitoring purposes. In the field of water, nanotechnology has the potential to contribute to long-term water quality, availability, and viability of water resources such as through advance filtration that enables sustainable water reuse, recycling or desalination. The aim of this action is to support research and technological development in the field of water treatment by applying developed or adapted nano-engineered materials to promising separation, purification and/ or detoxification technologies. Proposals should focus on process intensification aiming at improving selectivity, robustness, stability and performance while reducing energy requirements and by-product generation. Specific monitoring issues, as well as safety, environmental and health aspects, should be included if directly associated with the new technological solution proposed. Priority will be given to novel ideas and emerging technologies promising major advances and a large potential impact in the long-term, including cost-effectiveness

12. Joint call ENV-NMPWork Programme – Expected Impacts Development and uptake of innovative and cost-efficient water treatment technologies benefiting from progress and advances made in nanosciences, materials and technologies. This initiative should deliver step-change advances in water treatment technologies, including validation and verification of arising prospects in terms of improving treatment performance and reducing energy requirements. By fostering the knowledge base in this area, the projects addressing this topic will contribute to strengthening European competitiveness in the water sector and the implementation of the Environmental Technologies Action Plan and the Nanotechnology Action Plan.

13. Independentexperts • Expert evaluators are at the heart of the FP7 system • Provide independent, impartial and objective advice to the Commission • they represent neither their employer, nor their country! • Significant funding decisions will be made on the basis of their advice • They can also add value to projects through your comments and suggestions • The integrity of the process is crucial • They should follow the Code of Conduct annexed to the appointment letter

14. Who evaluates? • Selected from a wide pool from a database, on the basis of keywords • Minimum 3 evaluators per project • Selected per call • Replace about ¼ in any given area annually • Sign confidentiality and conflict of interest declarations • Names published after evaluation (though not at call or proposal level) • Target at least 40% female • Mix of geographical location and background • To register: https://cordis.europa.eu/emmfp7 FP7 – How to apply

15. Full Proposal Proposal forms Overview of the Evaluation Process “remote” may be “remote” Submission Individual reading Consensus Panel Finalisation Final ranking list Evaluators Evaluators Evaluators Rejection list Criteria Criteria Criteria Proposals in suggested priority order Eligibility COMMISSION COMMISSION Role of experts

16. ProcessFor each proposal: May be “remote” and / or central “remote” Proposal X copy 1 IER expert 1 Consensus meeting Proposal X copy 2 CR 3 experts IER expert 2 Proposal X copy 3 IER expert 3 • Note: There may be more than 3 evaluators • IER=Individual evaluation report • CR=Consensus Report

17. ProcessEthical Issues • The Consortium is asked to submit drafts of Information Sheet and Consent Form but does not need to submit copies of legislation • Proposals should comply with • fundamental ethical principles • relevant security procedures • … or be excluded from the process • Check if the proposal has in fact ethical issues • If yes: tick ethical issues box in the CR • Prepare Ethical Issues Report

18. ProcessEvaluating a proposal • Three guiding principles: • Objectivity • each proposal is evaluated as it is written • Accuracy • The judgment is made against the official evaluation criteria, and nothing else • Consistency • The same standard of judgment applies to each proposal

19. ProcessThe evaluation criteria • Criteria adapted to each funding scheme and each thematic area • specified in the work programme • Three main criteria: • S&T Quality (relevant to the topic of the call) • Concept, objective, work-plan • Implementation • Individual participants and consortium as a whole • Allocation of resources • Impact • Contribution to expected impacts listed in workprogramme • Plans for dissemination/exploitation

20. ProcessFP7 Evaluation Criteria Applicable to ALL funding schemes

21. ProcessFP7 Evaluation Criteria Collaborative projects

22. Process Funding schemes • Collaborative projects Support to research projects carried out by consortia with participants from different countries, aiming at developing new knowledge, new technology, products, demonstration activities or common resources for research. The size, scope and internal organisation of projects can vary from field to field and from topic to topic.Projects can range from small or medium-scale focused research actions to large-scale integratingprojects for achieving a defined objective Projects may also be targeted to special groups such as SMEs, Specific International Co-operation Actions, etc.

23. International cooperation • Some topics in work programme seek participation from third countries • International Cooperation Partner Countries (ICPC) • industrialised countries • research partners from ICPC may be financed

24. ProcessProposals that are onlypartly in scope • Note: The “S/T quality” of a proposal (first criterion) is evaluated to the extent that the content is relevant to the topic(s) addressed by the call • E.g. If a proposal is only marginally relevant, or if only one work package is relevant, the evaluator must downgrade the score – no matter how excellent is the science! • Relevance to the objectives of the call is also considered under “Impact” (third criterion)  • In relation to the sub-criterion “contribution to expected impacts listed in the work programme”

25. ProcessProposal scoring:Interpretation of the scores • 0: The proposal fails to address the criterion under examination orcannot be judged due to missing or incomplete information • 1: Very poor. The criterion is addressed in a cursory and unsatisfactory manner. • 2: Poor. There are serious inherent weaknesses in relation to the criterion in question. • 3: Fair. While the proposal broadly addresses the criterion, there are significant weaknesses that would need correcting. • 4: Good. The proposal addresses the criterion well, although certain improvements are possible. • 5: Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor.

26. ProcessProposal scoring • Each criterion is scored 0-5 • Marks can go from 0 – 5 in steps of 0.5, i.e half-marks are allowed • Experts are encouraged to use the whole range • Scores must pass thresholds if a proposal is to be considered for funding • Thresholds apply to individual criteria… • Threshold is 3 • …and to the total score • higher than the sum of the individual thresholds • Threshold is 10 • Note that to receive a mark of 5, a proposal does not have to be perfect. An excellent proposal can have minor shortcomings. • When writing comments in the IERs and Consensus Report, the severity of any weakness should be clearly stated, i.e. are they minor, moderate or significant

27. ProcessProposal scoring • Evaluate the proposal and conclude whether the proposal is • Excellent • Good • Fair • Poor • Very poor • Failing to address the criterion • Score the proposal accordingly

28. ProcessCommission Follow-up • Evaluation summary reports sent to applicants • “initial information letter” • Redress procedure • Draw up final ranking lists • Information to the Programme Committee • Contract negotiation • Formal consultation of Programme Committee (when required) • Commission decisions • Survey of evaluators • Independent Observers’ reports New for FP7

29. ProcessRedress • Proposers can complain if they believe there have been shortcomings in the handling of their proposal, and that these shortcomings have jeopardised the outcome of the evaluation process. • The quality and consistency of the evaluation reports (ESRs), derived directly from the CRs, is paramount to minimise the redress procedures

30. Tips on writing a successful proposal • An interesting and innovative idea • Strong consortium • Experienced co-ordinator • Complementarity of skills and expertise • Perfect matching between the proposal and the call • At least 6 months to prepare a proposal • European added value • Sustainability after the end of the project • Dissemination strategy FP7 – Tips

31. The EC Funding Process –A Rough Guide to :Why, What and How Sergey Mikhalovsky

32. Why? • Good Money – typically 200,000-300,000 € per participating organisation for 2-3-4 years • More if you are the Lead Partner – all the money is distributed through you, which counts for your organisation – good for you and for your organisation – typically 1-2-3 million € for 2-3-4 years • Good Value for Money – typically 60% overheads

33. Why? (continued) • Fun – experience of different countries, cuisines and cultures, meeting new people • Knowledge transfer – opportunity to work in different labs, access to unique equipment and instruments, methods and techniques • Human mobility – you visit others and others visit you to exchange knowledge and experience • Fair competition and useful feedback

34. Why? (cont.) • Networking – you build up your own network • There may be unique expertise somewhere in Europe unavailable in this country • Career enhancement • Publicity – EC loves it and gives you lots of opportunities to do that (even if you don’t want it...) • Recruitment of high quality researchers • And you may even get a result...

35. What? • Incredibly diverse formats and degrees of participation: (i) individual fellowships – No age limit, very well paid! (ii) participation as a team – you can be paid or you can employ someone (iii) co-ordinator – the same as (i) and (ii) plus management costs plus permanent headache.

36. What? (cont.) Funding is available for salaries, overheads, travel, consumables, management, equipment and organisation of networking events.

37. How? Regularly monitor the funding opportunities Most obvious source of funding for regular monitoring are: (i) FP7 - become a regular visitor on this site: http://ec.europa.eu/research/participants/ portal/page/home and become familiar with the structure of this programme. It is the main but not the only source of R&D funding in Europe

38. How? (cont.) (i) Be flexible with your idea of participation – sometimes there may be a Call which matches exactly what you want to do but that is unlikely. (ii) Identify potential areas of interest by monitoring previous Calls and - important! – analysing success rate in previous Calls. If the success rate is less than 10% - don’t go for it, unless you are a winner in a National Lottery or somebody else is prepared to do the main job of writing for you.

39. How? (cont.) • Try to get information about forthcoming Calls – it is possible to do so by attending consultations in Brussels or elsewhere, contacting your NCP (National Contact Point), getting information from an insider, etc. The earlier you know the contents of the future Call, the better your chances for success. When the Call has been announced, it is too late! You will have 2-3 months to submit a proposal from the date when the Call opens.

40. How ? (cont.) • There are also bottom-up Calls – these are my favourite – you can write about anything you like and the date of the new Call announcement is usually known well ahead, so you have plenty of time to think and prepare for the Call.

41. How ? (cont.) • There are also bottom-up Calls – these are my favourite – you can write about anything you like and the date of the new Call announcement is usually known well ahead, so you have plenty of time to think and prepare for the Call. • Erasmus Mundus and TEMPUS programmes offer in essence bottom-up Calls in Research&Education.

42. What to start with? • Formulate the idea, which fits a particular Call (probably several ideas to have a choice) • Identify potential partners – do a lot of networking, and do it all the time regardless of Calls. • Learn the Brussels speak – it takes time, but once you understand it – your chances for success are much higher.

43. What to do next? • You have to understand how to write a proposal • Become an expert evaluator for FP7 https://cordis.europa.eu/emmfp7/ ?fuseaction=wel.welcome It will help you to understand the evaluation process and ultimately improve your proposal writing skills

44. What to do next? • Write a proposal – preferably not by yourself if it is your first one. • Write a proposal yourself if you have done this before • Seek advice and assistance of people with experience: colleagues who have got such grants, Research Office • Interact with other partners • Notify your line managers and finances about your intention to submit a proposal in advance

45. S&T Quality of the proposal • Concept and objectives. What are the main ideas that led you to propose this work? Provide sufficient elements on the planned S&T methodology and describe in detail the S&T objectives. Show how they relate to the topics addressed by the call, which you should explicitly identify. The objectives should be those achievable within the project, not through subsequent development. They should be stated in a measurable and verifiable form, including through the milestones and deliverables.

46. S&T Quality (cont.) • Progress beyond the state-of-the-art. • Describe the state-of-the-art in the area concerned, and the advance that the proposed project would bring about. If applicable, refer to the results of any patent search you might have carried out. • S/T methodology and associated work plan. A detailed work plan should be presented, which should follow the logical phases of the implementation of the project.

47. S&T Quality (cont.) • Describe the overall strategy of the work plan. • Show the timing of the different WPs and their components • Deliverables and milestones • Describe any significant risks, and associated contingency plans • Milestones are control points where decisions are needed with regard to the next stage of the project. For example, a milestone may occur when a major result has been achieved. Another example would be a point when the consortium must decide which of several technologies to adopt for further development. Intangible – something which cannot be touched. • Deliverables: report, prototype, material, etc. Tangible - something which can be touched.

48. Implementation • Management structure and procedures Describe the organisational structure and decision-making mechanisms of the project. Show how they are matched to the complexity and scale of the project. Individual participants. provide a brief description of the legal entity, the main tasks they have been attributed, and the previous experience relevant to those tasks. Provide also a short profile of the staff members who will be undertaking the work.

49. Implementation • Consortium as a whole. Describe how the participants collectively constitute a consortium capable of achieving the project objectives, and how they are suited and are committed to the tasks assigned to them. Show the complementarity between participants. Explain how the composition of the consortium is well-balanced in relation to the objectives of the project. Resources to be committed. Show how the resources will be integrated in a coherent way, and show how the overall financial plan for the project is adequate. Infrastructure.

50. Impact What does your project bring to: • Individuals involved in the project; • Teams involved in the project; • Consortium as a whole; • Region, country as a whole; international dimension • New IP – profit • Socio-economic benefits, environmental significance, health • Development, dissemination, use of results