Introduction

1. Information and Communication Technologies (ICT) in the Seventh Framework ProgrammeHow to fill in the IER formICT Calls Dec 2010-April 2011

2. Introduction This tutorial gives guidelines on how to carry out the “remote” stage of an ICT evaluation • How to assess the proposal • Scoring the proposal • Filling in and returning the Individual Evaluation Form (IER)

3. But first, remember your responsibilities You have signed a non-disclosure agreement • Before, during and after this evaluation you do not disclose any information about the proposals which you have seen • Keep any printed copies secure when not in use; bring them with you to Brussels You are the evaluator • Do not pass this responsibility to anybody else • Do not discuss the proposals with anybody else

4. Your responsibilities Do not contact the proposers • Evaluate only the proposal as it was submitted to us at the call deadline, without any later additions or clarifications from the proposer • The identities of evaluating experts are never revealed to the proposers concerned Declare any potential conflict of interest • If there is a situation which might prevent you evaluating a proposal impartially, click on “I cannot evaluate this proposal” (see Rivet manual page 9)

5. The Rivet tool For the remote stage of this evaluation we are using a software tool “borrowed” from elsewhere in the Framework programme – Rivet • In Rivet the Individual Evaluation Report (IER) is called the Individual Assessment Report (IAR), but the meaning is the same • Save your work regularly, unsaved work is lost if you are idle too long and Rivet times you out. If you are reading, thinking or even typing Rivet believes you are idle. Only saving is considered to be an activity. Save your form at least every 20 minutes !

6. The Evaluation Criteria Proposals are evaluated on three criteria only • Scientific and technical quality • Implementation • Impact Assess the proposal in terms of all three criteria • Each criterion is more fully defined by descriptive “bullet points”, adapted to the project type. These are shown on the evaluation form • Provide a comment for each of the “bullet points” (but the bullet points themselves are not scored individually or separately)

7. The scoring scale First develop your comments on each criterion ……then select scores accordingly • Each criterion is given a score out of five, corresponding to the explanatory comments • A threshold of 3/5 is applied on each criterion • An overall score is calculated for each proposal • A threshold of 10/15 is applied on the overall score* Out of scope proposals are given low scores on Criterion 1 “Scientific and technical quality” (*not in FET objectives or the FET Flagships call)

8. The scoring scale Use the full scale! Half marks may be given 0 -The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information 1 -Poor. The criterion is addressed in an inadequate manner, or there are serious inherent weaknesses 2 -Fair. While the proposal broadly addresses the criterion, there are significant weaknesses 1 3 -Good. The proposal addresses the criterion well, although improvements would be necessary 2 4 -Very good. The proposal addresses the criterion very well, although certain improvements are still possible 2 5 -Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor 1 Problems which can’t be solved in grant agreement negotiations 2 Your comments must identify the required improvements

9. How to fill in IER - General Your comments • Comments are confined only to the criterion concerned • Comments describe only your final view of the proposal • Comments are clear and unambiguous. Try to avoid obscure acronyms and technical terms • Comments are of adequate length and cover all the bullet points under each criterion • Comments provide full justification for the score given

10. IER – Factual Evidence Comments are substantial; do not write generic criticisms; be specific, explain Comments are facts not opinions, don’t show doubt or indecision • not “I don’t understand why....” • but “The proposers do not make clear why.. .” Poor comments include words like: • “Perhaps, think, seems, assume, probably, …” Good comments include words like: • “Because, percent, specifically, for example, …”

11. IER – Give Clear Messages Poor comments are vague - Good ones are precise • “I think the consortium management plan is probably inadequate.” • “The consortium management plan is inadequate. It does not include clear overall responsibility for the demonstration activities; it omits a problem-solving mechanism in the event of disputes between partners.” • “The resources for the project seem unrealistic.” • “The resources in Workpackages 4 and 6 are seriously underestimated given the complexity of the activities involved.”

12. IER – Avoiding Conflicts Poor comments provide an opening for a debate - Good comments close the question • “There is no discussion of dissemination activities.” • “Dissemination activities are not adequately discussed.” • “There is only one end-user organisation in the consortium.” • “The consortium lacks a sufficient participation of end-users.” • “The proposal coordinator is not adequately experienced.” • “The proposal coordinator does not demonstrate in this proposal an adequate level of experience of work in this field.”

13. IER – Varying the Vocabulary Why say “Poor” when you can say: • Insufficient, minimal, fails to describe, unacceptable, inadequate, very generic, not evident, unfocused, very weak, bad, does not meet requirements, no information, inappropriate, limited, unclear, not sound enough, not specified, no significant impact, not been followed, unjustified, overestimated, does not fit profile… Why say “Excellent” when you can say: • Extremely relevant, credible, very clear, precisely specified, realistic, very innovative, extremely well suited, very good, timely, convincing, comprehensive, high quality, justified, very well identified, strong, highly effective, thoughtful, very promising, evidence, well-formulated, carefully-prepared, very professionally prepared, fully in line, looks great, very profound, sound, very convincingly integrated, clearly articulated, coherent, well balanced, very plausible, ambitious, clear advances, well above average …

14. IER – Final Check • Have you fully explained the proposal’s strengths and weaknesses on each of the criteria ? • Do your scores match your comments (high scores = positive comments, low scores = negative comments)? • Have you double-checked any matters-of-fact which you have quoted? • Have you written at adequate length? • If this was your proposal, would you find this report fair, accurate, clear and complete?

15. ICT Evaluations Submitting your IERs

16. Submitting your IERs • First, read over all your proposals to get an idea of the general standard and content • When you have completed the evaluation of each proposal, submit your IER (IAR) for that proposal in the Rivet tool - Don’t wait until you have evaluated all your proposals before starting to submit • You will see that the Rivet tool has the facility to create Consensus Reports CRs and Evaluation Summary Reports ESRs. We will not use this in this evaluation; you stop at the creation of IERs (IARs)

17. Submitting your IERs Don’t worry if, after submitting your IER, you would like to add to or modify your comments. You will anyway get a chance to discuss your opinions with the other experts in the meetings in Brussels Bring your copies of the proposals and a copy of your IERs with you to Brussels. In the meantime, keep them safe!

18. Finally, Give a fair and clear opinion on each proposal. You are: • Independent : you represent yourself, not your employer, not your country…..) • Objective:you evaluate the proposal as written • Accurate : you use the official evaluation criteria only • Consistent : you apply the same standard of judgment to each proposal • Incommunicado : you do not disclose to anybody the contents of the proposals which you see

19. Finally, Thank you very much for your help! - The ICT evaluation team