Evaluator’s view

1. Evaluator’s view Borka Jerman-Blažič University of Ljubljana and Jožef Stefan Institute SLOVENIA

2. Acting as an evaluator • Evaluation process is very demanding procedure which involves: • commission officers and staff and • number of evaluators

3. Acting as an evaluator • Evaluators are usually people with expertise in the subject addressed by the call and are coming from different environments: • Academia, industry, professional societies, research institutes, national institutions dealing with research etc

4. Invitation • The selection of evaluators is based on the data in CORDIS data base • If you want to become an evaluator you must register in the respective data base of evaluators for particular program • Sometimes, the selection process for evaluators is supported through additional channels

5. Invitation • Selection: you are selected according to your expertise and data records • You are selected because you have entered information about you in the CORDIS data base and because your expertise is in line with the scientific/technical objectives of the relevant call • So, be careful when entering data about your expertise, level of education and references • Every call has different objectives and as a consequence the selection of the evaluators follows their expertise and in each call is different • After the selection an invitation is launched usually via e-mail

6. The evaluator appointment start with: • An invitation • Your agreement • Your correct response • Your on time provision of the requested data by the Commission • Final selection/invitation comes later when all proposals are known, the call is closed and the number of needed evaluators is finally decided • The number of evaluators depends on the number of received proposals

7. The information requested by the evaluator • Availability • Legal entity form (each time you are asked to act as an evaluator) • Bank information (if the procedures do not change one signed document by the bank and yourself is sufficient for at least one year) • Copy of your ID document

8. Information requested • You have to agree to the conditions of work as stated in the Commission documents sent to you • Confidentiality agreement/commitment of non-disclosure and • Conflict of interest declaration • Reimbursement of your personal cost and fee

9. There are two options for an evaluator • You can act as independent expert or • as employee of an organisation (if you are employed) • The conditions regarding payment are slightly different but the fee per day is same • There are two forms of reimbursement: the fee per working day and reimbursement of travel and accommodation cost, both are specified in the accompanying documents

10. The selection process • After the decision is taken for your engagement as an evaluator the appointment procedure follows: • you receive appointment letter where all conditions of work are specified as well your duties and commitments (confidentiality/non disclosure and conflict of interest declaration)

11. Appointment • The appointment letter is a sort of contract with Annexes • You must sign the letter and the enclosed documents and send them back to the Commission officer • Instructions for your work follow as well

12. On line and on site evaluation • Travel and accommodation is organized by yourself • The place of evaluation is known (for FP6 this the building on Square Frere Orban) • The date of evaluation is known as well • Some programs (e.g. Marie Curie) practice individual evaluation on distance at your site • You receive evaluation guidelines for specific call that includes guidelines for your work, evaluation forms and guidelines how to fill them

13. Commitments and duties • If you are evaluating on distance then you are provided with the proposal on line via secured access to the network • You have restricted time to do the evaluation (1-2 weeks) but you are more free in planning your time • You may influence on the selection of proposals (by reading the abstracts) to be evaluated but this is not always possible as other evaluators are also pointing to the same proposals • All your evaluation data are entered as well on line via secure route • Consensus is achieved at site – in Brussels

14. Evaluation in FP6 • All proposal are receipt, opened, acknowledged and their content entered into a data base by the Commission officials to support the evaluation process • Basic eligibility criteria for each proposal are also checked by the Commission staff before evaluation begins • Commission officials assign particular number of proposals to the panel of experts and to particular evaluator • Before you start to evaluate a briefing meeting is organized for all evaluators involved

15. Evaluation in FP6: your role • Commission staff do not influence the opinion of an independent expert, if asked they provide to you only additional information or assistance • Each independent expert is assigned cca 5 to 6 proposals if they are IP or NoE, in case of STREPs that may be slightly more • Your working time is from 9 to 17 but usually the evaluators work much more as the evaluation is very demanding process

16. The evaluation process in FP6 • The working conditions are good and the staff is supporting and kind • You are supposed NOT to talk to other experts about the proposals you are evaluating • The place to do so are the consensus meetings where all appointed evaluators for particular proposal discuss for common evaluation report

17. The evaluation process in FP6 • Your task is to read carefully the proposal document and to prepare an IAR – individual assessment report • The report has several fields where you put your scores and justification

18. Confidentiality/conflict of interest • It is forbidden the evaluator to work closely with the proposer or to be involved in the preparation, the evaluator should not attend the panel meeting where such proposal is discussed • In case of indirect conflict of interest (proposal is coming from the institution the evaluator is employed) the evaluator should not be involved in evaluation of such proposal and have to indicate such case

19. Confidentiality/conflict of interest • All data, documents, writings, comments etc must stay in the place of evaluation. • Under no circumstances may an evaluator attempt to contact proposal submitter on his own account, either during the evaluation or afterwards. • It is also NOT recommended to visit WEB references even if they are mentioned in the proposal especially if the information on the WEB are placed after the date of proposal submission

20. Evaluation process: your role • Assess and mark the proposal exactly as it is described and presented, do not make any assumptions or interpretations about the project in addition to what the proposers have written in their proposal • Keep to the evaluation criteria as stated in the evaluation forms • Give marks and comments and write them in readable shape • You should care the comments to be written in line with the the marks • Maintain consistency in your scoring through the work

21. Provide a brief but explicit justification of your marks, be honest, but correct. The language especially in case of low scores must be correct as the comments will be sent to the proposers, you may use short extracts from the proposal text to justify your opinion. • Where justified you may give recommendations, but ensure that the scores given reflect the proposal as presented by the applicant • It is recommendable before the start of the evaluation you to examine a number of proposals before signing off their first individual assessment forms. This will help to calibrate their scoring

22. Evaluating: the good and the bad • Good proposals usually have clearly written text, not excessive wording, but text that provide good information for the evaluator to understand the objectives and the presented work • The objectives must be well set up and justified, very clearly • The methods to achieve the objectives should be sound, from technical and from management point of view • There must be a red line along the proposal from the beginning to the end • The proposal should look as compact package that is oriented to achieve common joint goals

23. Evaluating: the good and the bad • The idea of the proposal (what exactly is planned to be done) described must innovative. The innovationmay have different aspects • Evaluators are not always very deeply involved in particular specific field, so good proposal contain short but convincing state of the art of the field or subject background that justify the need for specified and identified problem to be solved • Good proposal contain text or description that justify the project funding from public money: the proposer must present enough evidence why the proposed way of research or proposed research topic is carried out on European level and why deserve public funding

24. Evaluating: the good and the bad • Each criteria from the IAR is important, so the proposal must show more than average merit in all subjects • This implies that the proposal must be well thaught and well prepared in all aspects • Scientific and technical value are most important, meaning that innovation must be present. Beside that, the constitution of the conzortium must be balanced, the proposal must show good balance of all relevant institutions, from industry, academia, SMEs, professional societies ect. • Distribution of the resources is also very important. The proposal must convince the evaluator that the allocation of the work and the resources are well prepared and that they give or show enough guarantee that the conzortium is capable to carry out the project up to the accomplishment of the goals

25. Evaluating: the good and the bad • All criteria from the IAR contribute to the overall ranking of the proposal, in that context • management part must be also prepared in line with the type of the proposed work • Information about involved institution and the major staff must be provided as well. This information contribute to the trust building among the evaluators that there is enough capacity, knowledge and experience for successful accomplishment of the project objectives • The proposer must convince you during text reading that his idea is innovative, sound and the methods proposed for work are the most appropriate for the presented problem to be solved or idea implemented • This implies enough information about the conzortium expertise, skills, knowledge, previous engagement, published references or similar data. The project usually benefits from CVs of involved people and from good description of the institutions consisting the conzortium

26. Evaluation in FP6: Consensus building • One among the evaluators is appointed as a raporteur • NoE and IP have 5 evaluators + a raporteur • STREPs have 3 evaluators • The task of the raporteur is to accept and record the majority view of the other evaluators, together with the Commission officer may bring additional evaluator(s) (up to 3) in case no consensus is achieved • The face to face consensus meeting is usually convened by the officer

27. Evaluation FP6:consensus meeting • The evaluators are usually presenting their scores and the justifications then they discuss in order to achieve common scores and comments • If this is achieved then the raporteur write the report and all evaluators must agree with it and sign it, the raporteur is responsible for collecting the signatures • Then the consensus report goes to the panel • The panel consist of all experts from particular area of the call e.g. BBAll, Security etc

28. Evaluation in FP6: Consensus meeting • If consensus has not been reached, the report sets out the majority view of the evaluators, but also records any dissenting views. In case any such disagreement has not been resolved a threshold score or average score given by the evaluators will be awarded for the proposal. • All consensus reports are entered in the data base and joint list of proposals ranked according the overall scores is generated • This list is discussed by the panel • The panel sometimes may agree to do some changes of the overall listing if this is explained and justified e.g. for some proposals covering missing area in the call etc.

29. Hearings and your role • The best ranked NoEs and IPs by the panel are called after few weeks to hearings • Most of the experts panels are present at the hearings. Experts that have evaluated proposals presenting on the panel must be present at the hearings • These meetings are convened by the Commission official • Evaluators after the presentation of the proposal that include mainly answers on the questions set up by the evaluators can set additional questions • Questions are prepared in written form and passed to the convenor • The presenters have no knowledge who was asking particular questions • Hearings are very helpful meetings as they clarify many aspects that are not always evident from the written text

30. Decision making • Outcomes of the hearings may somehow change the ordering of the highly ranked proposal on the list • After the hearings, the panel convene and decide about final ranking • With this your task as evaluator is almost finished • The further actions are connected with the reimbursement forms and the final payment for your services

31. The panel • It is important the evaluators to have a look on the monitoring statistics before they discuss the proposals at the panel meeting • This will help to take in account the evaluators marking profile and that the other experts before revising the scores and the ranking • The evaluators may agree on some recommendations regarding the negotiation process

32. CONCLUSION • If you are part of the new ERA or if you preparing yourself to take part then being an evaluator is exciting experience • You meet with your colleagues and you contribute to the success of the FP6 • It is overall an exciting experience