IFI8109 Planning and Running Research and Development Projects Reviewing development projects

1. IFI8109 Planning and Running Research and Development ProjectsReviewing development projects • Peeter Normak

2. Plan • Homework: Discussion of the L-Mix project application. • Innovativeness of objectives • Project management / implementation • Impact • General rating • Additional conditions and recommendations • Organization of reviewing of R&D projects. • Prerequisites for success in getting and completing R&D projects.

3. Why this – L-Mix – Example? • It is interdisciplinary (ICT + education). • The scope of the project was very broad (learning takes in today’s society mostly place in an augmented learning environment). • The threshold for being financed was exactly 4,0. • Access to the decision-making processes. • Characterizes different focuses/understanding of different types of reviewers. • Example of how the reviews/scores can be manipulated.

4. Discussion • What we learned from this example?

5. Organization of peer-reviewing of R&D projects

6. The purpose of a review • Provide as objective as possible assessment of the project proposal, based on the criteria set by respective decision-making body/institution. • Consequently: • Reviewers will be selected by the institution that announces the call for proposals. Why? • The reviewer remains anonymous to the applicants. Why? • Public calls are fairly regulated (example: http://ec.europa.eu/research/participants/portal/doc/call/h2020/common/1595113-h2020-rules-participation_oj_en.pdf).

7. Prevalent practice • Reviews are normally form-based. Why? • A reviewer gets normally more than one project application to be reviewed. Why? • Reviewing project applications may be remunerated or not. • Prior to the submission of the project application, a peer-reviewing (for example, by colleagues) is often used.

8. Reviewing – recommendations • Be benevolent and constructive in composing the reviews, rather than suspicious and critical. • Contribute to a review as it is an essential tool in forming the reviewer's image. • Consider the possibility of reviewer’s identity disclosure.

9. Prerequisites for success in getting and completing R&D projects – TDTTD • Being aware of what are the topical and currently heavily discussed problems. Why? • Being aware of what already has been done before in the problem area, and what are the strengths/weaknesses/limitations . Why? • Availability of necessary time resource. Why? • Availability of necessary work environment and tools. • Presence of a work organization and motivation system that support devotion. Why?

10. Home assignment • Team work: • compose a full-scale development project proposal based on the structure that is described in the file Development_project_form.doc. • Prepare a 20 minute presentation of the development project plan. The projects are: • P1 • P2 • P3 • P4 • P5

11. Next class: Tuesday, March 11, at 12:00Topics: Seminar: Presentation of development project plansRunning R&D projectsCase study – Learnmix (Arman)Reflection to the course

12. Innovativeness of Objectives I (Est. ministry official) • The goal is to develop a learner-centered e-textbook concept for three school levels, a business model for publishing and distribution of professionally developed content, guidelines for professional content development, technological infrastructure, the choice of digital devices and tools that enable publishers to expand their publishing media and distribution channels for the new e-textbook. The goals are ambitious but sometimes quite broadly described. What is unclear is the relationship and specificity compared to other similar initiatives in the world, and what e-textbook is planned to create for piloting. • The subject is topical and interesting. • Score: 4

13. Innovativeness of Objectives II (Est. entrepreneur) • E-textbooks based learning is gaining momentum in Estonia and more research needs to be done on that area in order to cover all aspects of it and find the right model/approach for our educational system. This project is contributing that goal and could also be internationally very valuable for other researchers working on that field. • Score: 4

14. Innovativeness of Objectives III (foreign university professor) • The project aims at the generalization and re-conceptualization of the notion ofelectronic textbook, in order to accommodate also learner-developed content and beavailable on a variety of digital devices.The project is timely, well documented, and compliant with the goals of the call.The project addresses the full span of the problem, from the current perception of the etextbookby teachers and learners, to the appropriate business models to be used to makesuccessful the new concept (see, e.g., Sect 2.5.2).The proposal calls for a wide array of methodologies to be used in the development andthe evaluation of the new concepts, all of them present in the involved research groups.It is less convincing on the part related to actual technologies to be used to make the e-textbookavailable on a large span of digital devices, with very different availability ofresources (screen surface, battery consumption, bandwidth availability, computationalpower, etc.). On this account, and on the metrics to be used to measure success on this,the proposal doesn’t say much. • Score: 4,5

15. Innovativeness of Objectives (summary) • The goal is to develop a learner-centered e-textbook concept for three school levels, a business model for publishing and distribution of professionally developed content, guidelines for professional content development, technological infrastructure, the choice of digital devices and tools that enable publishers to expand their publishing media and distribution channels for the new e-textbook. The goals are ambitious but sometimes quite broadly described. What is unclear is the relationship and specificity compared to other similar initiatives in the world; noe-textbook is planned to be created for piloting.The subject is topical and may have international resonance. • The scope seems to be too ambitious – the wish is to solve too many problems (for example, e-textbook can be used for different screen surface, battery consumptioncomputationalpower, etc.). On this account, the proposal doesn’t say much about the criteria. • Score: 4, out of which 40% = 1,6.

16. Project management / implementation I • One goal is to provide a business model - it is not clear whether business specialists will be involved, to which criteria it should conform and how it differs from the existing ones. • The project will be governed by management and research team, the project participants have previously worked together. • It is not clear whether the outcomes will pass the verification tests. • Which criteria and / or the functionality of the project results must meet? • Score: 3,5

17. Project management / implementation II • Project has milestones, but is lacking measurability. Project organization has needed competence and experience to achieve the project goals. Named activities are correlating quite well with the budget. • Score: 3,5

18. Project management / implementation III • Project goals and objectives are realistic and in the possible reach of the proponents.The work plan and the budget are detailed, realistic, motivated and closely connected tothe project goals.The proponent does have the necessary experience and competence to carry out theproject. • Requested resources are reasonable and adequate. • Score: 5

19. Project management / implementation (summary) • One goal is to provide a business model - it is not clear whether business specialists will be involved, to which criteria it should conform and how it differs from the existing ones. The project will be governed by management and research team, the project participants have previously worked together. It is not clear whether the outcomes will pass the verification tests. Which criteria and / or the functionality of the project results must meet? What criteria are followed in project management?It would make sense to cooperate with other organizations acting in the field. Project has milestones, but is lacking measurability. • Score: 3.5, out of which 30% = 1,05

20. Impact I • Creation of a new type of e-textbook is a long-term process and therefore sustainability and future development directions have significant weight. The project application could describe this more thoroughly. Relation to the IT Academy and IKTP programmes is modest. There are no plans to create and protect intellectual property. • Score: 3,5

21. Impact II • Expected results are well documented and reporting of those will be impressive. Direct connection with IT Academy could be questioned. Results could have international importance. • Score: 3,5

22. Impact III • The use of e-textbook is stalling not because of hard limitations of actual technologies(either HW or SW), but for lack of vision on re-inventing this instrument to be used insymbiosis with the new technologies. Under this respect the project could have goodand lasting impact.The proposal is careful in proposing to study also actual business viability of theconcept. Good integration with the goal of IT Academy.Realistic plans for use and informing of project results are set out in the proposal. • Score: 5

23. Impact (summary) Creation of a new type of e-textbook is a long-term process and therefore sustainability and future development directions have significant weight. The project application could describe this more thoroughly. Relation to the IT Academy and IKTP programmes is modest. There are no plans to create and protect intellectual property Score: 4, out of which 30% = 1,2

24. General rating I • Can be satisfied, provided additional conditions are fulfilled.

25. General rating II • Traditional education system is not fulfilling current needs and research around use of e-Textbook is very needed.

26. General rating III • A well documented, timely and innovative proposal, with a possible impact on IT society and itsperception by the general public. • More attention should be put on the technologies to be used to make the new concept of etextbookavailable on a large span of digital devices (how can we automate the porting on thesedevices in such a way to maximize user’s experience and minimize porting costs?)

27. General rating (summary) The subject is innovative and important not only because of usage of new IT applications, but also in terms of providing better education. The application is well written, but the measurability of objectives is weak. It makes sense to spend less resources for extending usability of a wide range of devices that change over time, and target more resources for development, testing and evaluation of a pilot e-textbook that will be developed. Due to the objectives and sub-objectives of the measure, the reviews, the discussions on the evaluation committee, it is proposed to finance the project with a reduced budget. Total score: 3,85

28. Additional conditions and recommendations • Additional conditions: • Specify the budget in accordance with the grant • Specify/refine the objectives • Reduce the number of compatible devices for piloting • Issue and evaluate an e-textbook, based on the elaborated principles • Due to the relevance and potential impact, the Commission decided to increase the total score by 0.2 points, assuming the implementation of the recommendations listed above. • Final score: 4,05

29. Being aware of what are the topical and currently heavily discussed problems • Purpose: • Defining its own research problem and ensure its topicality. Why? • Achieving reputation in the international research community. Why? • Activities: • Active participation in (international) professional events. • Following professional forums.

30. Being aware of what already has been done before in the problem area, and what are the strengths/weaknesses • Purpose: • Prevent “inventing a wheel”.. • Prevent accusations in plagiarism. • Activities: • Reading scientific literature (journals, collections, monographs). • Participation in systematic training (summer and winter schools, scientific seminars, lectures and lecture courses, etc.).

31. Availability of necessary time resource Purpose: • Reliance on extensive knowledge base. • Systemic (covering all major aspects) and deep (non-trivial) elaboration of the research problem. Activities: • Long-term breaking from other duties (for example, working in a foreign university for some months). • Revision of monthly, weekly and daily plans with the purpose to reserve some periods of time only for research.

32. Availability of necessary work environment and tools • Purpose: • To ensure effectiveness of R&D activities. • To ensure wide acceptance of the R&D results. • Activities: • Discussion with the colleagues and acquisition of necessary tools. • Using infrastructure of partner institutions.

33. Presence of a work organization and motivation system that support devotion • Purpose: • Release the mind for R&D activities. • Focus on R&D activities. • Activities: • Awareness of and solving problems that inhibit R&D activities. • Awareness of and taking into account the motivating factors that support devotion.