Planning and Running Research and Development Projects (IFI8109.DT) Evaluation of R&D-projects

1. Planning and Running Research and Development Projects(IFI8109.DT) Evaluation of R&D-projects • Peeter Normak

2. Plan for 10.02 • Homework: evaluation of the project proposal “The framework for supporting and analysing self-directed learning in augmented learning environment”. • Evaluation (reviewing) of R&D project proposals. • Critical aspects in evaluation of R&D projects: • The purpose (objective) of the project (What?) • Methodology (How?) • Available resources (Who?) • Sustainability

3. Homework: evaluation of the project proposal “The framework for supporting and analysing self-directed learning in augmented learning environment”

4. Review of the project proposal • Scientific quality and innovativenss of the problem setting. • Competency of the Principal Investigator and the team. • Actuality of the subject. • Project elaboration and justification of the methodology. • The involvement of graduate students. • The strengths of the application. • The weaknesses of the application. • Comments. • Overall assessment of the application (is not necessarily the arithmetic average of the previous scores).

5. Discussion • What aspects had to be discussed more thoroughly in the project proposal and what aspects could be presented more briefly? • What aspects are not covered in the review form you consider important to be assessed? • What were your biggest difficulties in assessing the project proposal?

6. Scientific quality and innovativenss of the problem setting • Scientifically acceptable in every sense, an international level project proposal. • Importance in education could actually transcend national boundaries. • Essentially, it is a design-based research in a topical research area – a physical learning environment that is augmented by a virtual one. • Score: 4,5.

7. Competency of the Principal Investigator and the team • Highly competent team. • There exist relevant international co-operation. • Score: 3,5.

8. Actuality (topicality) of the subject • The subject is clearly highly topical. • Score: 4,5

9. Project elaboration and justification of the methodology • In spite of some generality of the description, the methodology seems to be appropriate and reasonable. • The research tasks are well presented and are able to direct adequately the work. • Score: 3,5

10. The involvement of graduate students • Involvement of graduate students in the project is a strengths of the proposal. • Score: 5

11. The strengths of the application • The international level, an important topic. • Design-based research has a strong performance potential in the field of technological and methodological innovation. • Graduate students' involvement and good potential for sustainable development.

12. The weaknesses of the application • There were not enough hypotheses-type research tasks. • Contingent with whom the tests are conducted, is not specified. • The locations where the research will be conducted is not specified.

13. Comments • The foreign reviewer is favorably optimistic. • The local reviewer proposes to fund the proposal in full (as requested).

14. Overall assessment of the application (is not necessarily the arithmetic average of the previous scores) • 5 - Outstanding (is among the best 10% of applications); • 4 - Very Good (falls among the best 25% of applications); • 3 - Good (among the best 50% of applications); • 2 - Satisfactory (is among the best 80% of applications, the funding is not feasible); • 1 - Weak (belongs to the 20% weakest applications, certainly not to finance). • Score: 4,0.

15. Why this example? • It is interdisciplinary (ICT + education). • The scope of the project was very broad (learning takes in today’s society mostly place in an augmented learning environment). • The threshold for being financed was exactly 4,0.

16. Discussion • What we learned from this example?

17. Evaluation (reviewing) of R&D projects:evaluation of proposalsevaluation of final reports

18. My personal experience in evaluating projects • Reviewer of national research grants in Estonia and Georgia. • Reviewer of project proposals to Estonian national IT programme. • Assessing R&D project proposals as a member of the Council of Estonian Science Foundation. • Member of Expert Committee of Information Technology Foundation for Education (HITSA). • Occasionally evaluating other R&D projects.

19. (Peer-)Reviewing of research proposals • The objective of peer-reviewing is to evaluate and rank the project proposals, that would allow the decision-makers to conclude on funding. • Method of rating: parametric (fixed or non-fixed weights). • The basic budgeting methods: • Formula based • Application-based (with possible corrections to individual items or activities) • Proportional (applying the same or decreasing coefficients) • A combined method.

20. Reviewing of the final report • The objective: • Evaluation of the quality and quantity of the outcome • Assessment of the compliance to initial project proposal. • Method of evaluation: parametric (fixed or non-fixed weights) as for evaluation of the project proposals. • Indirect objective (targeted mainly to the applicant): to provide input for composing possible follow-up project proposals.

21. Critical aspects in evaluation of R&D projects

22. Critical aspect 1 – the objective of a project Project objective is the key factor that determines planning (and execution) of a project. General SMART requirements are applicable for the objectives of R&D projects as well: • Specific – states the exact purpose of the project. • Measurable – concrete criteria for measuring progress/outcome. • Agreed (attainable) – stakeholders recognize its importance. • Realistic (relevant) – enough resources are available. • Time-bound (timed) – grounding within a time frame. • The weights of these requirements are quite different for R&D projects.

23. Critical aspect 1 – the objective of a project • Target level (compared to the projects done elsewhere). Example: • Measurability (clearly defined) and realistic. Examples: • Justification (importance), including: • Relevance of the literature cited in justifying the research objectives • Clarity of research questions and/or hypotheses • Description of the theoretical framework • What the study supports and for whom it is important • To what extent the study raises new questions, or opens up new research directions.

24. Analysis of the project objective Forward looking! Analysis of possible trends. Taking into account current and emerging factors, requirements and limitations, for example: • Institutional and national strategies and development plans. • Changes of technological solutions and/or methodological approaches ( Example: increase of computing power). • Emerging/changing social demand. • Legislative constraints (Example: ethical issues, usage of sensitive data). • Possible activities of the competing research groups/schools. Image of a sandglass: absorb as much knowledge as possible (upper part) for having impact as broad as possible (lower part).

25. Critical aspect 2 – the methodology • The clarity of description and adequacy of the methodology, including • unambiguous definitions and concepts • well structured research • compliance with the objectives of the research. • Methodological Innovation. • Adequacy of the methods used. • The methodology must be described in detail, so that it would be possible to decide to what extent achieving the objectives of the project is realistic. Example:

26. Critical aspect 3 – availability and quality of resources • Formal competency indicators of the Principal Investigator (Project manager’s) and the project team members, including: • Existence of degrees and relevant certifications. • Existence of high-quality publications and/or patents. • Non-formal competency indicators of the Principal Investigator and the project team members, including: • Experience in completing similar projects. • Previous cooperation experience between the team members. • Existence of necessary infrastructure, equipment and other tools.

27. Critical aspect 4 – sustainability • Definition 1: sustainability = long-term survival capability. • Definition 2: sustainability = conformity to present needs without compromising the ability to conform the future needs. • Ensuring continuity of people, particularly through inclusion of graduate students in the project implementation. • Maintaining/improving the academic environment (i.e. the project should not exhaust the resources). • Growth of reputation, particularly through applicability and high impact of the project outcomes. • ?

28. Possible additional aspects • Priority of the topic • Widening/deepening of existing expertise • Development of new expertise • Budget • Risks • Applicability of expected outcomes • …

29. Home assignment No 2 • Identify and formulate the objective of a research project in the area of your thesis and perform the following tasks: • Describe the expected outcome of the project. • Explain the necessity (topicality) of achieving the objective of the project. • Describe the innovativeness of the expected outcome. • Analyze the possible applications of the expected outcomes. • Compose a 4-minutes presentation about the items described above (additional 2 minutes will be reserved for discussion).

30. Next class: Wednesday, February 17, at 14:15Seminar: Identifying the objective and outcomes of a research project

31. Example: fixed weights in rating project proposals • The scientific and/or technological level 40% • The level of project objectives • Correspondence of the project’s concept to the objectives • Innovativeness with respect to the current international level • Implementation of the project 30% • Research/technological development methodology • Compliance of activities and budget with the objectives • The partners involved and the level of their experience • Project management structures, adequacy and thoroughness • The adequacy and reasonableness of resources involved • Impact of the project 30% • The project's impact on science and technology development • Communication plan of the results • The use of the intellectual property created in the course of the project

32. Example: non-fixed weights in rating project proposals • Scientific justification of the project - 3,5 • The competence and expertise of the PI and research staff - 3 • The importance of the research for Estonian and EU society, economy and culture - 4 • State-of-art of research environment - 4 • Correspondence to national strategic development plans and to the institution’s development plans and strategies - 4,5 • Justification for accommodating other available R&D resources into the budget - 4 • Overall assessment - 3,5

33. Example: a formula based budget • The simplest case: the formula contains three parameters only: • X – Number of researchers’ positions • Y – Number of positions of other academic staff • P – proportion of experimental research, 0 ≤ P ≤ 1. • The formula for calculating Estonian target financing grant (yearly, €): • S = 15050X + (X+Y)(4215P+1690(1-P)) • Additional possible parameters for more complex formula: • Number of graduate students • Number of technical staff • Equipment • …

34. Application-based budget calculation

35. Proportional budget calculation

36. Combined budget calculation

37. Objective – good examples • Learning Layers: Development of a set of modular and flexible technological layers for supporting workplace practices in SMEs that unlock peer production and scaffold learning in networks of SMEs, thereby bridging the gap between scaling and adaptation to personal needs. • Development of change-aware collaborative decision-making algorithms and tools based on dynamically integrated big data. • Design and validation of methods and web based software tools for language learning and testing the language proficiency level A1-A2-B1-B2-C1-C2 on the basis of Estonian Inter-language Corpus.

38. Example: improper research objective • Development of a web-based and competence-driven learning environment for personal learning which is open to all educational institutions in Croatia. What is wrong with this objective? • What is “competence-driven learning environment ”? • R & D should be problem-based, not country-based. • Openness to institutions is an administrative, not a research issue. • “to all educational institutions” refers to the universality of the proposed learning environment. Why only for educational institutions (LLL, workplace learning, …)? • Back …

39. Example: rated as not feasible (unrealistic) • The main objective: to create theoretical foundations and artificial intelligence support systems prototype for (1) describing situations and (2) deducing decisions based on the written texts.Sub-goals: • To model artificial intelligence support systems for representing the situation descriptions and situation constraints in natural language with the help of formal structures. • To model artificial intelligence support systems for representing arguments in natural language with the help of formal structures. • To create a web-based software prototype which allows to transform the descriptions of the situations in Estonian language to the logic formulas automatically. • To model artificial intelligence support systems for deriving decisions and checking for derivability based on situation descriptions and situation constraints. • To validate the created systems by imitating various management processes.

40. Example: description of a research methodology • The research methodology bases on the principles of design research (The Design-based Research Collective, 2003; Herrington & Oliver, 2000; McMahon & Oliver, 2004): intertwined development of learning environments and learning design theories; repeated and spiral cycles of design, developmentbased on analysis and evaluation; development of learning design theories that support practical applications, authentic design. Secondly, the principles of action research (Whitehead & McNiff, 2006) will be used to inform our methodology which call for involving the target group to the study as researchers and interpreters of their own learning processes. The study is divided into three sequential cycles with the following phases: 1) needs analysis and the analysis of current practices; 2) planning of design process; 3) performing the design, evaluating the results and 4) implementing the outcomes. The results of the first cycle are used in executing the second and third cycles.The soft ontology methods for multi-perspective analysis and corresponding software will be used that are developed by the research group [Kaipainen et al., 2007]. We would like to emphasize the new opportunities that the use of modified Petri nets will offer for describing and analysis of activity theory based learning processes.

41. Example: description of a research cycle • Goal of the cycle: Developing the pedagogical pattern language.

42. Example: extract from the review • Yet, while this general framework is well-presented, the specific research design and practical research tasks remain disappointingly vague. The field of pedagogical pattern language is broad and well-developed, but the proposal gives little indication as to even the kinds of patterns that might be suitable for the language they wish to create. The project cycles are also extremely nebulous, giving no clue as to the time frame of each cycle or the practical activities involved (for instance, where will the focus groups among teachers be conducted? how many?). Likewise, it is not clear whether the project plans to develop a prototype of a distributed learning environment (proposed for the 3 phase of the III cycle) or a distributed learning environment itself (to be tested in the 4. phase of the IV cycle).