1 / 32

Semantic models in healthcare education What is it and how it can improve formative assessments

Semantic models in healthcare education What is it and how it can improve formative assessments. MedBiquitous Annual Conference 2012 May 2-4 2012 - Baltimore, MD Muriel Foulonneau Younes Djaghloul Raynald Jadoul Nabil Zary. 1. Context. 2. OAT approach. 3. Experimentation. Challenges

zeph-hill
Download Presentation

Semantic models in healthcare education What is it and how it can improve formative assessments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Semantic modelsin healthcare educationWhat is it and how it can improve formative assessments MedBiquitous Annual Conference 2012 May 2-4 2012 - Baltimore, MD Muriel FoulonneauYounes DjaghloulRaynald JadoulNabil Zary

  2. 1 Context 2 OAT approach 3 Experimentation • Challenges • Research questions • OVACS • AIGLE • TAO • The ontology of carieswith Karolinska institutet

  3. 1 Context 2 OAT approach 3 Experimentation • Challenges • Research questions • OVACS • AIGLE • TAO • The ontology of carieswith Karolinska institutet

  4. Challenges • Two main challenges: • Item variability in an assessment • generate items from a model in order to avoid repeating items • save time and resources, as assessment resource creation is a time and resource consuming activity • Learning adaptivity • adapting question forms or assessment path in formative assessment according to candidate answers or profile • We strive toward: Efficient Approach to Automate/Assist the generation of assessment resources. OVACS-AIGLE-TAO

  5. How the challenges are addressed • Knowledge sources • Expert • Social / Crowd sourcing • Repository • Textual How ? • Question generation • Keep the initial semantic • Semantic Inference • Adaptivity Assessment resources OVACS-AIGLE-TAO

  6. In summary the goal was to • Enable the automatic generation assessment questions based on formal models of knowledge • Knowledge oriented approach based on semantic technologies: • The creation of a streamline exploring the use of semantic technologies for e-assessment • Semantic for model checking • Semantic for inference ( to discover knowledge) • Needs to have models with formal representation (such as RDF) • Four questions • How to build a domain model? • How to validate the proposed model by non IT expert ? • How to generate assessment questions from the refined model? • How to build a flexible delivery environment for these questions? OVACS-AIGLE-TAO

  7. 1 Context 2 OAT approach 3 Experimentation • Problem statement • Research questions • OVACS • AIGLE • TAO • The ontology of carieswith Karolinska institutet

  8. The vision OVACS-AIGLE-TAO

  9. Overview on approach Experts, repositories, social media The Final test Knowledge: Informal models • 4.Delivery strategy • TAO Delivery Module • TAO QTI viewer 1.Model building: data mining, human methodology Formal but not validated Validate questions Experts for question validation Knowledge: Formal models List of assessment questions • 2.Model validation • Experts for model validation • OVACS : to assist experts and to hide the complexity of he formalism (OWL, description logic ) Final Ontology • 3.Question generation • AIGLE tool, Automatic QTI based questions generation • Semantic similarity techniques OVACS-AIGLE-TAO

  10. The process OVACS-AIGLE-TAO

  11. Origins of OAT OVACS-AIGLE-TAO

  12. OVACS • Ontology VAlidation for Common uSers • How to validate formal knowledge model by questions OVACS-AIGLE-TAO

  13. OVACS: what ? • Question based strategy for validation • Question to for the validation of the domain not for the assessment • Generate question based on existed knowledge element ( automatic) • More simple for the expert than modifying formal model ( OWL ) • Four ontological components (OC) to validate (RDF schema) • Instance of • All property value • Sub class • Property of a class • 12 types of feedback • For each OC Accept, remove, don’t Know • Templates for textual question • Generic (Subject, Predicate, Object) • Dedicated OVACS-AIGLE-TAO

  14. OVACS architecture Source Ontology OWL Validated ontology OVACS Engine (Semantic web technologies) Evaluated ontology Ontology of management feedback Generated Question (Web based) • Manage history • Get past questions Expert feedbacks OVACS-AIGLE-TAO

  15. OVACS interface http://crpovacscaries.elasticbeanstalk.com/ OVACS-AIGLE-TAO

  16. AIGLE • AssessmentItemGeneratorinLearningEnvironment OVACS-AIGLE-TAO

  17. AIGLE – Assessment item generator • Security issue (variability) • Adding variability to an item • no expected variation of the construct • Model-based learning (adaptivity) • Generating items from knowledge represented as a model • the construct is modified for each item Stem variables Auxiliary information Options Key OVACS-AIGLE-TAO

  18. IMS-QTI item generation process • Generating items from Web data sources OVACS-AIGLE-TAO

  19. Calculating the semantic similarity between distractors and the correct answer Gabon -- Libreville No SemSim With SemSim Adapted 3 semantic similarity strategies to large scale semantic graphs OVACS-AIGLE-TAO

  20. Results of user test • Clear decrease of performance in the population when using SemSim (optimizing the similarity between the correct answer and the distractors) OVACS-AIGLE-TAO

  21. User testing with countries and their capital OVACS-AIGLE-TAO

  22. TAO • TestingAssisté parOrdinateur • (Computer-Aided Testing) OVACS-AIGLE-TAO

  23. TAO – assessment and feedback loop • The TAO platform is based on semantic web paradigm, i.e. it manages question items decorated with any needed ad-hoc properties • The TAO platform delivers questionnaires that can also be featured with any desiderated extra semantic properties • The TAO collects all answers and behaviors of the test-takers • If extra properties like the “provenance” (i.e. the source model built with OVACS and used by AIGLE) are attached to the question items or to the questionnaire, these properties are stored in tests results • The analysis of the tests results will enforce be used by as feedback loop for a validation process impacting the AIGLE & OVACS phases. OVACS AIGLE TAO OVACS-AIGLE-TAO

  24. 1 Context 2 OAT approach 3 Experimentation • Problem statement • Research questions • OVACS • AIGLE • TAO • The ontology of carieswith Karolinska institutet

  25. Experiment with a dentistry teacher OVACS-AIGLE-TAO

  26. Original hypothesis • The creation of the domain ontology can use semi-automatic strategies, or third party encoders, or a collaborative work: can we ask an expert to validate the assertions in the ontology? • What is lost in the expert’s speech when creating the ontology? • Does the expert understand automatically generation questions? • Does the expert flag the errors? OVACS-AIGLE-TAO

  27. Creating the ontology • An ontology of the caries • A one hour interview where the teacher explained the caries, their description, their causes, how to handle them, how to prevent them, how to set a diagnostic • Definition of a list of concepts / keywords • Creation of classes, instances, and properties • Creation of the OWL ontology OVACS-AIGLE-TAO

  28. Test set up • Labels on stand alone • Selected a subset of the ontology to keep the test short:instanceOf (13 items) and subClassOf (11 items) • Only Boolean questions + “I do not know” option • 24 questions • 2 intentional mistakes: on the content (causes of caries) and spelling (emanel instead of enamel) • Objective: • verify whether the teacher would find the validation mechanism usable • Verify whether errors would be detected and corrected • Video recording of the teacher OVACS-AIGLE-TAO

  29. OVACS interface http://crpovacscaries.elasticbeanstalk.com/ OVACS-AIGLE-TAO

  30. Test conclusions • Confusion between • the role of domain expert validating knowledgeandthe role of teacher who prepares questions for students •  Objective was not well understood  rework experiment conditions • According the comments of our expert: • “Difficulty level of the generated questions is generally low” • “But with very different variations in the difficulty level” • The OVACS validation questionnaire led to: • 6 removals (2 subClassOf, 4 instanceOf) • 16 accept (9 for subclassOf, 7 for instanceOf) • 2 answers “I do not know” for subclassOf meant not relevant OVACS-AIGLE-TAO

  31. Next steps • OVACS • Enrich collaborative features • AIGLE • Ensure a validation / feedback on the generated items • AIGLE generates distractors from an open model (large dataset from the Web) using semantic similarity, but needs to identify relevant distractors in the case of a bounded model (in this case a model for caries) • Predicting item difficulty? Initial test for general culture questions using a Web mining approach. Would need to be tested for medical knowledge. OVACS-AIGLE-TAO

  32. Thanks a lot! http://tao.lu OVACS-AIGLE-TAO

More Related