Peeter Normak

1. Planning and Running Research and Development Projects(IFI8109.DT)Reviewing development projectsPractice of running R&D projects • Peeter Normak

2. Plan for 30.03 • Kaija Kirch, Cyber security narratives. • Homework: Discussion of the LearnMix project application. • Case analysis: Learning Layers (Tobias Ley). • Organization of reviewing of R&D projects. • Practice of running R&D-projects: a view of a research group. • General project management practices. • Artificial measures to improve the performance indicators.

3. Homework • Homework: Discussion of the LearnMix project application. • Innovativeness of objectives • Project management / implementation • Impact • General rating • Additional conditions and recommendations

4. Example of manipulation: LearnMix versus CALLCORP • Grades from the experts: • In fact for LeanMix (12,5*0,4 + 12*0,3 + 12*0,3)/3 = 4,07 • In fact for CALLCORP (13*0,4 + 12,5*0,3 + 12*0,3)/3 = 4,18 • New grades for CALLCORP from the first reviewer (marked red): • 3 + 3,5 + 3 • Final total score (12*0,4 + 11,5*0,3 + 11*0,3)/3 = 3,85 • Example of a financed proposal: (11*0,4+11*0,3+11,5*3)/3 = 3,72.

5. Why this (LearnMix) Example? • It is interdisciplinary (ICT + education). • The scope of the project was very broad (learning takes in today’s society mostly place in an augmented learning environment). • The threshold for being financed was exactly 4,0. • Involvement in the decision-making process. • Characterizes different focuses/understanding of different types of reviewers. • Example of how the reviews/scores can be manipulated. • Another example of the same programme (cf. CALLCORP).

6. Discussion • What we learned from this example?

7. Specifics of evaluation of development projects • The lack of objective quantitative (numerically measurable) criteria. • Local internal decision-making (including international projects). Example – Tempus JEP 12418 (Development of Master Programme in Multimedia and Learning Systems). • Possible incompetence of reviewers. • Wide variation in evaluation process, from non-existent (examples – innovation vouchers, or IKTP) to an extremely rigid and regulated (example – public procurements). • Multitude of possible sources of funding. Example: LMS IVA.

8. Organization of peer-reviewing of R&D projects

9. The purpose of a review • Provide as objective as possible assessment of the project proposal, based on the criteria set by respective decision-making body/institution. • Consequently: • Reviewers will be selected by the institution that announces the call for proposals. Why? • The reviewers remain anonymous to the applicants. Why? • Public calls are fairly regulated (example: http://ec.europa.eu/research/participants/portal/doc/call/h2020/common/1595113-h2020-rules-participation_oj_en.pdf).

10. Prevalent practice • Reviews are normally form-based. • A reviewer gets normally more than one project application to be reviewed. • Reviewing project applications may be remunerated or not. • Prior to the submission of the project application, a peer-reviewing (for example, by colleagues) is often used.

11. Reviewing – recommendations • Agree to became a reviewer if proposed. • Be benevolent and constructive in composing the reviews, rather than suspicious and critical. • Contribute to a review as it is an essential tool in forming the reviewer's image. • Consider the possibility of reviewer’s identity disclosure.

12. Practice of running R&D-projects:a view of a research group

13. Success factors in completing R&D projects • Being aware of what are the topical and currently heavily discussed problems. Why? How? • Being aware of what other colleagues in other research institutions do in the problem area, and what are the strengths/weaknesses/limitations . Why? • Availability of necessary time resource. Why? • Using necessary work environment and tools. Why? • Presence of a work organization and motivation system that support devotion. Why?

14. General project management practices

15. Project management – general methodology General methodology – structural approach (what? instead of how?). Basic structures: • Project management knowledge areas • Project management process groups • Project management activities • Project management artifacts Reason: universality, widely applicable.

16. Project management – knowledge areas • PMBOK Guide: A Guide to the Project Management Body of Knowledge. • The knowledge areas: • Project integration management • Project scope management • Project time management • Project cost management • Project quality management • Project human resource management • Project communications management • Project risk management • Project procurement management • Project stakeholders management. These knowledge areas are applicable to all stages of the project. • PMBOK Guide became in fact a standard in project management; for the 2004 version see http://engineering.queensu.ca/Outreach/EngineeringStudents/files/PMBOK3rdEnglish.pdf ).

17. Project management – process groups • Process is defined as a coherent system of activities that results in a certain outcome. • Project management process groups: • Initiating processes • Planning processes • Executing processes • Closing processes • Controlling processes

18. Project management – activities • Project management activities can be divided into general activities and product specific activities. • A systematic approach to general project management activities is presented in Project Management Maturity Model (PMMM). • Examples: • Planning, organizing and coordinating the work of the project team. • Acquiring and allocation of human and other resources. • Create necessary work environment. • Encourage devotion, excitement and creativity inside the project team. • Solving problems/conflicts both inside the project team as well with other parties. • Informing the project team and other parties involved about the state of the art of the project, as well as about success and problems.

19. Project management – artifacts • Project management artifacts are documents that regulate and support the project execution. • Examples: • Needs analysis and/or feasibility study. • Project charter. • Work breakdown structure and/or project schedule. • Change control plan. • Risk management plan and/or table/database of risks. • Communications plan. • Lessons learned document. The artifacts form a project’s portfolio (NB! The concepts project’s portfolio and portfolio of projects are different).

20. Measures to “improve” the performance indicators(“juriidiliselt on kõik korrektne” – legally is everything correct)

21. Increasing the number of publications • Big number of colleagues are included among the authors of publications. Example: https://cdsweb.cern.ch/record/1129811/files/jinst8_08_s08003.pdf • Articles are as short as possible. Compare: http://www.mat.univie.ac.at/~neum/contrib/fullkepler-1.pdf • Presentation of a research will be repeated in many articles (with small variations).

22. Increasing the number of citations • Possibly many of his own earlier publications are cited (the consortia should be different, for avoiding self-citations). • Articles are published only in ISI journals. • Research is conducted in an area where a big number of researchers are active.

23. Dilemma of big research projects/groups • Top-down versusbottom-up. • Bottom-up approach (individual freedom): successful research presumes academic freedom in all aspects – choice of the research topic, methods to be used, structuring the research etc. • Top-down approach (certain aspects are prescribed): research should be focused on a given topic. • Solution: 1) decide collectively on the research problems; 2) form (on a voluntary basis) subgroups for dealing with the problems.

24. Dilemma of a balance between quality and quantity • Extreme 1: only quality is valued (publication in journals of Thomson Reuters’ Master Journal List). • Problems: • Long publication cycles • No journals for emerging research areas/topics – the most suitable journal may not belong to the ISI WoS • Difficult to publish articles dealing with problems only locally important. • Extreme 2: only quantity is valued (the number of publications). • Problems: • Publishing in local journals/proceedings – no international visibility • Low quality, repetitions – production of “scientific noise” • Low competitiveness in receiving international grants.

25. Home assignment • Individually: study the document Guide for procedure for evaluation and selection of applications for the operation “Support for applied research in smart specialisation growth areas” (http://adm.archimedes.ee/str/files/2016/01/AV-NUTIKAS-hindamiskomisjoni-hindamisjuhend_ENG.pdf). • In teams: compose an initial proposal of a development project based on the structure that is described in the file Development_project_form.doc.

26. Next class: Wednesday, April 6, at 14:15Topics: Case studies – Running R&D projectsDiscussion on the topics raised by the students

27. Innovativeness of Objectives I (Est. ministry official) • The goal is to develop a learner-centered e-textbook concept for three school levels, a business model for publishing and distribution of professionally developed content, guidelines for professional content development, technological infrastructure, the choice of digital devices and tools that enable publishers to expand their publishing media and distribution channels for the new e-textbook. The goals are ambitious but sometimes quite broadly described. What is unclear is the relationship and specificity compared to other similar initiatives in the world, and what e-textbook is planned to create for piloting. • The subject is topical and interesting. • Score: 4

28. Innovativeness of Objectives II (Est. entrepreneur) • E-textbooks based learning is gaining momentum in Estonia and more research needs to be done on that area in order to cover all aspects of it and find the right model/approach for our educational system. This project is contributing that goal and could also be internationally very valuable for other researchers working on that field. • Score: 4

29. Innovativeness of Objectives III (foreign university professor) • The project aims at the generalization and re-conceptualization of the notion ofelectronic textbook, in order to accommodate also learner-developed content and beavailable on a variety of digital devices.The project is timely, well documented, and compliant with the goals of the call.The project addresses the full span of the problem, from the current perception of the etextbookby teachers and learners, to the appropriate business models to be used to makesuccessful the new concept (see, e.g., Sect 2.5.2).The proposal calls for a wide array of methodologies to be used in the development andthe evaluation of the new concepts, all of them present in the involved research groups.It is less convincing on the part related to actual technologies to be used to make the e-textbookavailable on a large span of digital devices, with very different availability ofresources (screen surface, battery consumption, bandwidth availability, computationalpower, etc.). On this account, and on the metrics to be used to measure success on this,the proposal doesn’t say much. • Score: 4,5

30. Innovativeness of Objectives (summary) • The goal is to develop a learner-centered e-textbook concept for three school levels, a business model for publishing and distribution of professionally developed content, guidelines for professional content development, technological infrastructure, the choice of digital devices and tools that enable publishers to expand their publishing media and distribution channels for the new e-textbook. The goals are ambitious but sometimes quite broadly described. What is unclear is the relationship and specificity compared to other similar initiatives in the world; noe-textbook is planned to be created for piloting.The subject is topical and may have international resonance. • The scope seems to be too ambitious – the wish is to solve too many problems (for example, e-textbook can be used for different screen surface, battery consumptioncomputationalpower, etc.). On this account, the proposal doesn’t say much about the criteria. • Score: 4, out of which 40% = 1,6.

31. Project management / implementation I • One goal is to provide a business model - it is not clear whether business specialists will be involved, to which criteria it should conform and how it differs from the existing ones. • The project will be governed by management and research team, the project participants have previously worked together. • It is not clear whether the outcomes will pass the verification tests. • Which criteria and / or the functionality of the project results must meet? • Score: 3,5

32. Project management / implementation II • Project has milestones, but is lacking measurability. Project organization has needed competence and experience to achieve the project goals. Named activities are correlating quite well with the budget. • Score: 3,5

33. Project management / implementation III • Project goals and objectives are realistic and in the possible reach of the proponents.The work plan and the budget are detailed, realistic, motivated and closely connected tothe project goals.The proponent does have the necessary experience and competence to carry out theproject. • Requested resources are reasonable and adequate. • Score: 5

34. Project management / implementation (summary) • One goal is to provide a business model - it is not clear whether business specialists will be involved, to which criteria it should conform and how it differs from the existing ones. The project will be governed by management and research team, the project participants have previously worked together. It is not clear whether the outcomes will pass the verification tests. Which criteria and / or the functionality of the project results must meet? What criteria are followed in project management?It would make sense to cooperate with other organizations acting in the field. Project has milestones, but is lacking measurability. • Score: 3.5, out of which 30% = 1,05

35. Impact I • Creation of a new type of e-textbook is a long-term process and therefore sustainability and future development directions have significant weight. The project application could describe this more thoroughly. Relation to the IT Academy and IKTP programmes is modest. There are no plans to create and protect intellectual property. • Score: 3,5

36. Impact II • Expected results are well documented and reporting of those will be impressive. Direct connection with IT Academy could be questioned. Results could have international importance. • Score: 3,5

37. Impact III • The use of e-textbook is stalling not because of hard limitations of actual technologies(either HW or SW), but for lack of vision on re-inventing this instrument to be used insymbiosis with the new technologies. Under this respect the project could have goodand lasting impact.The proposal is careful in proposing to study also actual business viability of theconcept. Good integration with the goal of IT Academy.Realistic plans for use and informing of project results are set out in the proposal. • Score: 5

38. Impact (summary) Creation of a new type of e-textbook is a long-term process and therefore sustainability and future development directions have significant weight. The project application could describe this more thoroughly. Relation to the IT Academy and IKTP programmes is modest. There are no plans to create and protect intellectual property. Score: 4, out of which 30% = 1,2

39. General rating I • Can be satisfied, provided additional conditions are fulfilled.

40. General rating II • Traditional education system is not fulfilling current needs and research around use of e-Textbook is very needed.

41. General rating III • A well documented, timely and innovative proposal, with a possible impact on IT society and itsperception by the general public. • More attention should be put on the technologies to be used to make the new concept of etextbookavailable on a large span of digital devices (how can we automate the porting on thesedevices in such a way to maximize user’s experience and minimize porting costs?)

42. General rating (summary) The subject is innovative and important not only because of usage of new IT applications, but also in terms of providing better education. The application is well written, but the measurability of objectives is weak. It makes sense to spend less resources for extending usability of a wide range of devices that change over time, and target more resources for development, testing and evaluation of a pilot e-textbook that will be developed. Due to the objectives and sub-objectives of the measure, the reviews, the discussions on the evaluation committee, it is proposed to finance the project with a reduced budget. Total score: 3,85

43. Additional conditions and recommendations • Additional conditions: • Specify the budget in accordance with the grant • Specify/refine the objectives • Reduce the number of compatible devices for piloting • Issue and evaluate an e-textbook, based on the elaborated principles • Due to the relevance and potential impact, the Commission decided to increase the total score by 0.2 points, assuming the implementation of the recommendations listed above. • Final score: 4,05

44. Being aware of topical and currently discussed problems • Purpose: • Defining its own research problem and ensure its topicality. • Achieving reputation in the international research community. Why? • Activities: • Active participation in (international) professional events. • Following professional forums. • Inviting active scholars for presentations and discussions about topical research problems and cooperation opportunities. • Creation and maintaining of discussions inside the research group (research seminars, meetings, blogs, ...). Example: http://tihane.wordpress.com/ (Blog of senior researcher Kai Pata). • Application of funds for direct communication with the international academic community (for attending conferences, network meetings, ...). • Example: COST actions (http://www.cost.eu/domains_actions/ict).

45. Being aware of what colleagues do in the problem area • Purpose: • Prevent “inventing a wheel”. • Prevent accusations in plagiarism. • Activities: • Reading scientific literature (journals, collections, monographs). • Participation in systematic training (summer and winter schools, scientific seminars, lectures and lecture courses, etc.). • Ensuring access to the scientific literature (including electronic access to full-text databases). Example: ELNET consortium. • Promoting information flow (including sharing each other publications) between the research group members and other colleagues. Possible tools: https://www.researchgate.net, http://www.mendeley.com, … • Application of funds for attending and organising systematic training of project team members (courses, series of lectures, for attending conferences, summer schools, ...).

46. Reserving necessary time resource • Purpose: • Reliance on extensive knowledge base. • Systemic and deep elaboration of the research problem. • Activities: • Long-term breaking from other duties (for example, working in a foreign university for some months). Examples: KN, VT. • Revision of monthly, weekly and daily plans with the purpose to reserve some periods of time only for research. • Finding resources for conducting long-term (2-6 months) research in leading research centers. • Hiring necessary support staff (secretaries, project coordinators, web managers, ...) for freeing researchers’ time for professional activities. Prevent assigning tasks to the researchers that are not related to their professional competence. • Establishing work organisation and procedures that take into account preferences and habits of each team member.

47. Using necessary work environment and tools • Purpose: • To ensure effectiveness of R&D activities. • To ensure wide acceptance of the R&D results. • Activities: • Discussion with the colleagues and acquisition of necessary tools. • Using infrastructure of partner institutions. • Establishing permanent personal work places for researchers. • Acquisition of the necessary tools (hard- and software, licenses, …) for research. • Creation of a common space for discussion and other activities (e.g. lunches) with colleagues and partners.

48. Work organization and motivation that support devotion • Purpose: • Release the mind for R&D activities. • Focus on R&D activities. • Activities: • Awareness of and solving problems that inhibit R&D activities. • Awareness of and taking into account the motivating factors that support devotion. • Equal opportunities to attend on scientific discussions, not depending on the academic position of a researcher (junior or senior researcher). • Providing adequate academic support to the team members, and involvement them in decision making processes. E: Research Fund in SDT. • Consider interpersonal compatibility in work organization and assigning tasks. • Recognition of the significance of the research conducted by the team members and stimulating them through scientific discussion and generating new ideas and issues.