Huub Scholten and Adrie J.M. Beulens Wageningen University - Netherlands

1. Testing ontological support for multidisciplinary model-based problem-solving for water management Huub Scholten and Adrie J.M. Beulens Wageningen University - Netherlands Huub Scholten - CSM/KSS2005 - test MoST

2. Index • Introduction • Problems to solve • HarmoniQuA - MoST • Project • Ontological KB and Knowledge Editor • Modelling Support Tool • Verification - validation • Mental Modelling Model • Testing: criteria and results • Conclusions Huub Scholten - CSM/KSS2005 - test MoST

3. Introduction • Model-based problem solving is seen as a process • Integrated problem-solving (with socio-economics aspects)  • multidisciplinary problems • higher demands to QA • QA through • Process Knowledge Base (here for modelling, instantiated for water management) • Process support tool Huub Scholten - CSM/KSS2005 - test MoST

4. Problems to solve • Growing interest for QA in modelling with a multitude of reasons: • ambiguous terminology; • a lack of mutual understanding between key-players, • malpractice in regard to input data; • inadequate model set-up; • insufficient calibration/validation; • model use outside of its intended scope; • insufficient knowledge on some processes; • miscommunication of the modeller to the end-user; • overselling of model capabilities; • confusion on how to use model results in decision making; • a lack of documentation and transparency of modelling processes. Huub Scholten - CSM/KSS2005 - test MoST

5. HarmoniQuA - the project • Period 1-1-2002 – 31-12-2005 (4 years) • 2.57 million Euro (1.65 million Euro financed by EC) • 12 partners in 10 countries with expertise in 7 domains of model based water management • www.HarmoniQuA.org Huub Scholten - CSM/KSS2005 - test MoST

6. WU, The Netherlands GEUS, Denmark NTUA, Athens, Greece CEH-Wallingford, UK WL | Delft Hydraulics,The Netherlands Cemagref, France BfG, Germany SMHI, Sweden VITUKI Plc, Hungary Uni Dortmund, Germany LNEC, Portugal DHI Hydroinform a.s., Czech Republic HarmoniQuA - partners Huub Scholten - CSM/KSS2005 - test MoST

7. HarmoniQuA - aims Developing a methodology and support for modelling in water management according to the Water Framework Directive consisting of: • Knowledge Base (KB): • Guidance on how to model for water management • Glossary of terms • Modelling Support Tool (MoST): • Providing guidance from the KB • Monitoring what modelling teams do • Helping generating fancy reports • Training website for professionals and students: • Screen-recording movies on MoST • Presentations, exercises, background information Huub Scholten - CSM/KSS2005 - test MoST

8. MoST (KB + tool) overview

9. Knowledge Base • Content (what to do and how to do it) in increasing specialization • Knowledge on processes (co-operating in teams) • Specialised for modelling • Instantiated for water management • Implemented in an ontology(e.g. smart database; Protégé2000, (http://protege.stanford.edu) • Protégé2000 is for knowledge engineering experts and too difficult for domain experts • HarmoniQuA’s Knowledge Base Editor act as front-end for Protégé2000 and deals with authorisation issues • Provided to users by MoST (Modelling Support Tool) Huub Scholten - CSM/KSS2005 - test MoST

10. Modelling Support Tool (MoST) • Provides guidance from the KB to users • Can set-up and support the execution of multidisciplinary modelling projects • Monitors the work of all persons involved in a modelling project • Reports to specific audiences Huub Scholten - CSM/KSS2005 - test MoST

11. Verification – validation of KBS • Verification: determining that the KBS is built according to its specifications. • Validation: determining that the KBS actually fulfils the purpose for which it was intended. • Testing: more generic term, referring to verification and validation. Huub Scholten - CSM/KSS2005 - test MoST

12. Mental Modelling Model • If problem owner has a problem, he/she can organize a multidisciplinaryteam, consisting of team members, who have different disciplinary backgrounds and fulfil different roles in that team. • The multidisciplinary team ‘solve’ the problem at hand in a processwith best professional practices. • The process is carried out in a modelling project, consisting of one or more subprojects, belonging to one or more domains/disciplines. • Subprojects: • are single domain subproject or multidomain subproject; • consist of steps. • Steps consist of tasks. • Tasks consists of activities. • Tasks and activities can use methods and tools. • What teams have to do can be made explicit in a knowledge base. Huub Scholten - CSM/KSS2005 - test MoST

13. Structured testing • Implemented ontological KBS (1) • KB structure (11) • KB content (related to water management) (12) • Process decomposition (121) • Content of decomposition elements (122) • MoST (13) • Training material (14) • KBS technology (2) Huub Scholten - CSM/KSS2005 - test MoST

14. KB structure (11) Criteria Correctness, completeness, consistency, granularity. Method(s) • Discussions with (1) internal experts, (2) internal testers and (3) external reviewers. • Using the structure to fill KB with the knowledge. Result(s) Initially discussions were fruitful, later not. Huub Scholten - CSM/KSS2005 - test MoST

15. KB content: process decomposition (121) Criteria Correctness, completeness, redundancy, consistency, transparency, granularity. Method(s) • Discussions with (1) internal experts, (2) internal testers and (3) external reviewers. • Using the KB in 2x10 test cases. Result(s) • Demands for changing the process decomposition came from project partners and from the wider modelling society. These were related to (1) order of tasks, (2) dependencies of tasks and (3) is some piece of work a task or an activity. • Achieving agreement was difficult, but implementing changes was easy and not time consuming. Huub Scholten - CSM/KSS2005 - test MoST

16. KB content: decomposition elements (122) Criteria Correctness, completeness, redundancy, consistency, meaningfulness. Method(s) Reading and using by (1) internal testers (not involved in filling KB, (2) professional modellers in test cases, (3) external reviewers. Result(s) Long list of errors, wishes, comments, but all appreciated the guidance from the KB. Huub Scholten - CSM/KSS2005 - test MoST

17. Modelling Support Tool (13) Criteria Correctness, reliability, according to requirements, adequate for professionals, appropriate for students. Method(s) Using MoST by (1) internal testers (not involved in filling KB), (2) professional modellers in test cases, (3) external reviewers. Result(s) Incremental (prototyping) development of MoST in 3 phases, with a test after phase 1 and phase 2  • long list of ‘bugs’ • more support needed for multidisciplinary teams + more flexibility (working on 2 decomposition levels), because activity-level is often too detailed. Huub Scholten - CSM/KSS2005 - test MoST

18. Testing training material (14) Criteria Correctness, usefulness professionals, usefulness students. Method(s) Using MoST in (internal and external) workshops/seminars and university courses. Result(s) Many small changes, but training website and especially screen-recording movies on MoST appeared to flatten the (steep) learning curve. Huub Scholten - CSM/KSS2005 - test MoST

19. Testing re-use of technology (2) Criteria Is the KB ontological structure reusable and is the tool MoST useful for other processes? Method(s) Applying the presented technology for other processes, i.e. within the EC-funded project Aquastress (aiming to mitigate water stress problems). Result(s) • Applying the ontological KB structure for the Aquastress project, did not show any technical problem. • MoST could show and use the new KB without a problem. Huub Scholten - CSM/KSS2005 - test MoST

20. Conclusion • Testing is a continuous process. • Peer reviewed scientific output = testing. • Applying our technology for other processes in which multidisciplinary teams have to perform a complex job, shows that the technology has a wide field of application • Public website www.HarmoniQuA.org • free software • training material • etc. Huub Scholten - CSM/KSS2005 - test MoST