Aims of the (integration) activity

1. WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

2. WP4:Pilots with the Integrated System Validation of the ProjectUPF 8, ALTRAN 7, LogicaCMG 3, OUNL 3, GIUNTI 5, CERTH 4, SU 4, SYN 2Dai Griffiths, Josep Blat, UPF WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

3. Aims of the (integration) activity • To provide realistic scenarios and corresponding pilots which constitute an effective environment for evaluation of the project, whereby the systems resulting from the research and development can be challenged and validated • To provide the overall project evaluation strategy WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

4. Tasks, deliverables, timetable (I) • T1. Define initial pilot scenarios (m6) • T2. Define the pilot evaluation plan (m6) • T5. Develop an overall evaluation plan (m6) • D4.1: Report (m 6) on: • scenarios and pilots definition • user requirements and validation criteria for the overall project • evaluation strategies and initial evaluation plan implementation WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

5. Tasks, deliverables, timetable (II) • T3. Set up the initial pilots (m6-12) • T4. Validate the initial system through the evaluation of the Digital Cinema pilot (m12-18) • T6. Generate user requirements for cycle-2 pilots (m12-18) • D4.2: Report on results of first evaluation of pilots (month 18) • Suggested interim report on m12 to submit for the review WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

6. Pilots envisaged • Pilot 1 Training for digital movie production (UPF) • Pilot 2 Training for continuous education of health workers (UPF) • Pilot 3 Unesco-IHE – Institute for Water Education (?) • Pilot 4 The Antwerp Lifelong Learning City (?) • Request an initial description WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

7. Pilots (I) • T1. Define initial pilot scenarios (up to m6) • Pilot 1: Digital movie production • Partners (?) • Visual effects and other UPF postgraduate courses • Others from possible partners: Mediapro, Brainstorm, Thomson, Barco, XDC, Filmlight, DVS, Pandora (MediaSalles, ...) • Commitments definition: elaborate on association document • User requirements, validation criteria, evaluation • Target groups definition: learning providers, learning designers, content, tutors, learners ... availability: STRESS THE IMPORTANCE OF A COMMUNITY OF USERS • Plan for the implementation: define competencies based syllabi, ... INSERT HERE MATERIAL FROM IP-RACINE – DISCUSS TRANSFORMATION INTO COMPETENCE ORIENTED – DISCUSS THE CHALLENGES POSED BY THE PILOT • Establish a model for the other pilots WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

8. Pilots (II) • T1. Define initial pilot scenarios (up to m6) • Pilot 2: Training for continuous education of health workers • Partners (?) Hospital Clínic, Doyma (others) • Gastroenterology/hepatology, Pediatrics, Colo-rectal cancer screening, ... • User requirements, validation criteria, evaluation • Target groups definition: learning providers, learning designers, content, tutors, learners ... availability WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

9. Pilots (III) • T1. Define initial pilot scenarios (up to m6) • Pilot 3: Unesco-IHE – Institute for Water Education • ? WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

10. Pilots (IV) • T1. Define initial pilot scenarios (up to m6) • Pilot 4 The Antwerp Lifelong Learning City • ? WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

11. Evaluation planning and initial implementation • T2. Define the pilot evaluation plan, specifying the validation criteria and the strategies for evaluation (up to m6) • T5. Develop an overall evaluation plan (up to m6) • Validation focus along cycles • 'proof of concept' during the first cycle through the Digital Cinema pilot • 'usability' during the second cycle through Health Workers pilot, the UNESCO-IHE pilot, the Antwerp Lifelong Learning City pilot, and an extension of the Digital Cinema pilot. • ' wider applicability and sustainability' during the final cycle. WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

12. Evaluation planning (I) • T5. Develop an overall evaluation plan (up to m6) • objectives addressed (including gender issues), • criteria and indicators: • quality criteria • effectiveness in context • technical criteria • components and functions to be assessed • methodologies in context of the project • the different actors and “reference users” in the process and their tasks • the coordination and procedures, defining a timeline for evaluation WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

13. Activities Aspect RTD activities Technical Design & Implementation activities Pilots & Validation activities Pilots & Validation activities experiments Experiments pilots demonstrators Objective Technical verification and validation of the functional and usability aspects of the tools Verification of the integration and the interoperability between the components Validation of the environment integrated and of the individual components. Focus: - functionality - operationally - usability Validation of the environment integrated and of the individual components. Focus: - sustainability Feeds into The concerning activities Technical Design & Implementation + Aspect RTD activities Technical Design & Implementation + Aspect RTD activities Requirements & Analysis activities Period ‘continuously’ after completion of each integration loop: 3 times after the conclusion of the experiments of the first two integration loops: 2 times. Round 1: small scale Round 2: large scale after the conclusion of the experiment of the final integration loop: 1 time Sample evaluation grid from TA WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

14. On users/actors and methodologies • Users / actors • learning providers, learning designers, tutors, learners ... (different from different areas?) • Methodologies to be used: • Ethnographic study, Interviews, Focus groups, Questionnaires, Observations, Experiment, Peer review, Think-alouds, User diaries, System log data, Textual data, Cost analysis, Reviews, Usability inspection, ... • Components / areas (to be defined for a good plan) WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

15. Evaluation planning in detail • T5. Develop an overall evaluation plan (continued) • more detail needed for planning and initial implementation • evaluation contact in each organisation – not only this WP partners • define components to be evaluated; provide initial evaluation sheet for each component (model provided) • Initial scenarios for pilots probably needed to plan for the whole period • UPF in charge of coordination and assistance, every partner responsible of carrying out their own evaluation, evaluation committee to supervise overall evaluation and validation • Everybody to send a list of components, and an evaluation sheet for each component WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

16. Component to be evaluated Areas to be assessed Parameters to be assessed Actors / partners Role of the actors Evaluation procedure / material Starting date Immediate action (and date) Intermediate action (and date) End date Conclusion (general / for each area) Overallcomments Evaluationcommittee comments Evaluation sheet model WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

17. Pilot evaluation plan and implementation • T2. Define the pilot evaluation plan, specifying the validation criteria and the strategies for evaluation (up to m6) • Define an implementation plan for the first pilot • Model for the coming plans (which have to be initially defined) WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

18. Deliverable reminder; contributions • D4.1: Report (m 6) on: • scenarios and pilots definition • user requirements (?) and validation criteria for the overall project • evaluation strategies and initial evaluation plan implementation • Contributions of the different partners (to be defined now for the first 6 months of the project) • OUNL ? • ALTRAN ? • LogicaCMG ? • GIUNTI ? • CERTH ? • SU ? • SYN ? • Anybody missing? Any contribution missing? WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

19. Evaluation representatives (all partners) • OUNL ? • ALTRAN ? • LogicaCMG ? • GIUNTI ? • CERTH ? • SU ? • SYN ? • L3S ? • INSEAD ? • UB ? • UvA ? • SURF ? • Evaluation committee? WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

20. Reminder of coming tasks • T3. Set up the initial pilots (m6-12) • Initial pilot on Digital Cinema; start earlier if possible to be able to validate; otherwise, make it a phased pilot, where validation can start early • T4. Validate the initial system through the evaluation of the Digital Cinema pilot (m12-18) • Phased work or earlier start • T6. Generate user requirements for cycle-2 pilots (m12-18) • On the basis of early work on pilots definition • D4.2: Report on results of first evaluation of pilots (month 18) • Suggested interim report on m12 to submit for the review WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

21. Ourselves • Dai Griffiths (WP manager) dgriffiths@iua.upf.edu • Josep Blat (in charge of DC pilot) josep.blat@upf.edu • Sergio Sayago (evaluation specialist) sergio.sayago@upf.edu • and others! WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005

22. By way of summary • Evaluation, validation, user requirement activities are probably everybody’s task • Components and overall evaluation/validation strategies to be defined: contributions expected from everybody; not an easy task • Pilot 1 (Digital Cinema) to be planned and carried out during first period • Pilots 2, 3, 4 to be planned during m12-18; but initial scenarios important for the whole user reqs, validation, evaluation planning due by month 6. A second updated deliverable by month 12? • And don’t forget the periphery of the project! Possible extra pilots, associates feedback ... WP4 Presentation, TenCompetence Kick-off, Dec 8th – 9th 2005