Episode 3 / CAATS II joint dissemination event

1. Episode 3 / CAATS II joint dissemination event Lessons Learnt Episode 3 - CAATS II Final Dissemination Event Philippe Leplae EUROCONTROL Episode 3 Brussels, 13 & 14 Oct 2009

2. Content • Project objectives • Lessons learnt method • Lessons learnt in: • management • validation • concept clarification • result integration • validation techniques • Conclusion Episode 3 - CAATS II Final Dissemination Event

3. Project objectives • Provide detail on key concept elements in SESAR • Initial operability and performance assessment • Initial technical impact assessment • Analysis of available tools for SESAR validation • Report on validation methodology …… • Hence specific emphasis on lessons learnt Episode 3 - CAATS II Final Dissemination Event

4. Lessons learnt method • 2 workshops, Madrid and Brétigny • 9 different sessions on 5 main topics involving more than 30 participants • Validation strategy and project DOW • Project management • Concept clarification • Integration and Consolidation • EP3 validation techniques and levels of maturity • Most project members also working in SESAR • Strong desire to provide feedback to SESAR Episode 3 - CAATS II Final Dissemination Event

5. High level lessons learnt • System view should drive validation activities • Need to adapt DOW and management to required flexibility • Need time to develop system driven project • We have developed artefacts to drive it • Many different techniques tried on Episode 3 • Suited to low maturity level concept elements • Guidelines on when and how to use it are produced • Importance of support tools • Good strategy useless without tools to support it • Tools also require investment in training • This provides harmonized working methods, key to make distributed project work Episode 3 - CAATS II Final Dissemination Event

6. Lessons learntin management • Our challenges: • Wide range of disseminated partners • Crossed responsibilities and thinly spread effort • Innovative tasks • High quality requirements • Tight deadlines • Our answers: • Development of tools for distributed work • Training on the tools, and specific delivery process • A lot of co-ordination activities • Follow-up actions: • Build more flexibility in the programme (DOW and contract) Episode 3 - CAATS II Final Dissemination Event

7. Lessons learntin validation • Our challenges: • Maturity lower than expected at programme onset • E-OCVM V2 not well adapted to concept wide validation • Lack of E-OCVM expertise inside team • Wide range of results • Our answers: • We built a framework with guidelines, principles, templates. • Developed assumptions management • Performance framework and DOD’s provided at start • Guidance provided through system WP and co-ordination cell • Thorough review of experimental plans • Open issue: • Maturity assessment at start of each validation activity Episode 3 - CAATS II Final Dissemination Event

8. Lessons learntin concept clarification • Our challenges: • Multiple sources from SESAR (CONOPS, OI’s,etc…) • Lack of details for validation • Multiple sources to update DOD’s • Our solutions: • Propose a structure, around process model and DOD’s • Develop different modes of descriptions • DOD’s • Operational scenarios • Story boards • Develop collaborative working methods to update documents • Follow-up: • Establish a recognised authority to select among options • Use hot topics to drive validation Episode 3 - CAATS II Final Dissemination Event

9. Lessons learntin result integration • Our challenges: • Exercises not providing consistent results • Different granularity of results • Various degree of validity of quantitative data • Our solutions: • Influence diagrams help understanding benefit mechanism and identify where performance is improved • Influence model has been developed ready to be fed with results • Management of assumptions to interpret results • For subsequent coverage: • Steer validation activities from the performance framework • Use of models developed in EP3 to provide more quantitative data to performance model Episode 3 - CAATS II Final Dissemination Event

10. Lessons learnt in validation techniques • Successful used of different techniques • Prototyping • Macro-modelling • Various sorts or gaming • Expert groups • Work packages integrated different techniques • Combine results of very different nature • Various sequences of activities tried • Expert groups key to success • Benefits of industry assessment: • “Reality check” • Should run as a complementary process to operational validation Episode 3 - CAATS II Final Dissemination Event

11. Validation techniques:Expert Groups • Lessons learnt • Useful at all maturity levels • Need good facilitation techniques by somebody with OPS expertise • Need current operation experts + concept experts • Storyboards good method to structure EG debates • Our input to SESAR • Storyboards technique • Various facilitation techniques • Hot topics • Guidelines document Episode 3 - CAATS II Final Dissemination Event

12. Validation techniques:Gaming • Lessons learnt • To be associated with Expert Groups and modelling • Suited to clarify roles and responsibilities for all maturity levels • More flexible than prototyping as not in real time (interruptions feasible) • Progression from paper-based gaming to platform gaming • Our input to SESAR • Different platforms • CHILL • DARTIS • ACCES platform • Paper based gaming and associated web based tool • Guidelines document Episode 3 - CAATS II Final Dissemination Event

13. Validation techniques:Prototyping • Lessons learnt: • Not a cheap RTS! • Allowed to fail • Concept clarification-no quantitative data • Successive runs with same team of OPS staff • Our input to SESAR • First results on: • PRNAV, CDA in descent phase. • Acceptability of 4D trajectory to controllers • Guidelines document Episode 3 - CAATS II Final Dissemination Event

14. Validation techniques:Modelling • Lessons learnt: • Use of process modelling and network modelling • Process modelling complement gaming • Need to work on precise concept elements • Macro-modelling • Our input to SESAR: • 3 innovative tools developed in EP3 • One abandoned (unsatisfactory results) • One providing results when applied to mature concept elements • One promising for network wide assessments • Guidelines document Episode 3 - CAATS II Final Dissemination Event

15. Conclusion • Episode 3 is the first ATM-wide concept validation programme • We had to develop tools and methods adapted to this new challenge • Another 2 months to complete our work (WP and final report) • We provide to SESAR: • New tools • New methods • An integrated approach • First validation results… • See www.episode3.aero • And our most precious output: • A team with the expertise and concrete experience already working on the SESAR projects: Episode 3 - CAATS II Final Dissemination Event

16. Episode 3 - CAATS II Final Dissemination Event

17. Thank you