1 / 27

Danica Damljanovi ć University of Sheffield danica@dcs.shef.ac.uk

Usability Enhancement Methods in Natural Language Interfaces for Querying Ontologies Birmingham, 12 April, 2011. Danica Damljanovi ć University of Sheffield danica@dcs.shef.ac.uk. Outline. Background: What are Ontologies ? What are Natural Language Interfaces (NLIs)?

uriel-oneil
Download Presentation

Danica Damljanovi ć University of Sheffield danica@dcs.shef.ac.uk

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Usability Enhancement Methods in Natural Language Interfaces for Querying OntologiesBirmingham, 12 April, 2011 DanicaDamljanović University of Sheffield danica@dcs.shef.ac.uk

  2. Outline • Background: • What are Ontologies? • What are Natural Language Interfaces (NLIs)? • What are Usability Enhancement Methods? • Objective • Improve NLIs to Ontologies with usability enhancement methods • Our approach • Two NLI systems for querying ontologies: • QuestIO • FREyA • Two usability studies to test the usability enhancement methods • Findings • Demo • Conclusion

  3. MARY <is a> PERSON UNIVERSITY OF SHEFFIELD <is an> ORGANISATION MARY <works for> UNIVERSITY OF SHEFFIELD SHEFFIELD <is a> CITY UNIVERSITY OF SHEFFIELD <is located in> SHEFFIELD UNITED KINGDOM <is a> COUNTRY SHEFFIELD <is located in> UNITED KINGDOM MARY <lives in> SHEFFIELD Mary works for University of Sheffield, which is located in Sheffield. Sheffield is located in the United Kingdom. Mary lives in Sheffield. SELECT ?country WHERE { ?person <lives in> ?city ?city <located in> ?country • FILTER ?person = MARY }

  4. In which country does Mary live?

  5. What are Usability Enhancement Methods? • Who are the users? • application developers • end users

  6. The Objective • Increase usability of Natural Language Interfaces to ontologies • For end users: increase precision and recall • For application developers: decrease the time for customisation

  7. Our Approach

  8. QuestIO 1.15 1.19 compare

  9. QuestIO prototype

  10. QuestIO: User Evaluation • Usability testing: • effectiveness: could the tasks could be finished using QuestIO • efficiency: how quickly? • user satisfaction • System Usability Scale (SUS) • subjective (was it easy to formulate a query?, etc.) • Experimental setup: • a complete counterbalanced repeated measures, task-based evaluation design • Baseline (search engines) vs. QuestIO • 12 subjects familiar with the domain (GATE software) • four tasks: • three defined, e.g. ...find parameters of Cebuano gazetteer... • one undefined task, ...find anything you want about GATE software...

  11. QuestIO User Evaluation: Results • Effectiveness: • the scale from 0 (easy) to 2 (impossible) • 0.355 for QuestIO in comparison to 0.895 for baseline, p = 0 .001 • Efficiency: • the subjects significantly slower when using baseline (157s) in comparison to QuestIO(107s), p=0.001 User satisfaction: SUS score satisfactory (69.38) • Tasks: • defined tasks: user satisfaction reaching 90% • undefined tasks: user satisfaction low (~44%)

  12. QuestIO: weaknesses • Lexical failures: Tokenizer vs. Tokeniser • Conceptual failures: • missing concepts, relations, or both • The users not being aware of why the failures happened • Can this be improved with usability enhancement methods such as feedback and clarification dialogs?

  13. FREyA - Feedback, Refinement, Extended VocabularyAggregator • Feedback: showing the user system interpretation of the query • Refinement: • resolving ambiguity: generating dialog whenever one term refers to more than one concept in the ontology (precision) • Extended Vocabulary: • expressiveness: generating dialog whenever an “unknown” term appears in the question (recall) • portability: no need for customisation from application developers • The dialog: • generated by combining the syntactic parsing and ontology-based lookup • the system learns from the user’s selections

  14. Feedback: answer is found

  15. Feedback: No answer is found

  16. Feedback: User Evaluation • Usability testing: • effectiveness • efficiency • user satisfaction • System Usability Scale (SUS) • subjective (was it easy to formulate a query?, etc.) • Experimental setup: • 30 subjects outside Sheffield, two domains (GATE software and US geography) • four tasks: • three defined: • two repeated from the previous study • one where the answer was not available, e.g. ...find states bordering hawaii... • one undefined task, ...find anything you want about GATE software or rivers, cities, ... in the United States...

  17. Does the feedback make any difference? • Effectiveness: yes , p=0.01, 0.67 for QuestIO, 0.13 for FREyA • Efficiency: no, although the overall result differs (180.5 seconds for QuestIO, 155.27 seconds for FREyA), 2-tailed independent t-test reveals that this difference is not significant (p=0.852) • Query Formulation: for the defined tasks there is no difference in the perception of the difficulty of the supported language (F=5.255, p=0.071), but for the undefined tasks the users believed that the language supported by FREyA is easier! (F=8.016, p=0.015) • Showing that the system knows about certain concepts, but cannot find any relation between them was not clear. • Interactive features were well accepted.

  18. FREyA Workflow

  19. Demo • http://gate.ac.uk/freya ESWC 2010

  20. Evaluation: correctness • Mooney GeoQuery dataset, 250 questions • 34 no dialog, 14 failed to be answered • Precision=recall=94.4%

  21. Evaluation: Learning • 10-fold cross-validation • 202 Mooney GeoQuery questions that could be correctly mapped into SPARQL and required dialog • improvement from 0.25 to 0.48 • Errors: ambiguity and sparseness

  22. Evaluation: Ranking Mean Reciprocal Rank: 0.76

  23. Learning the Correct Ranking • Randomly selected 103 dialogs from 202 questions (343 dialogs) • MRR increased for 6% from 0.72 to 0.78

  24. Evaluation: Answer Type

  25. Conclusion • Combining syntactic parsing with ontology-based lookup in an interactive process of feedback and query refinement can increase the precision and recall of NLIs to ontologies, • while reducing the time for customisation by shifting some tasksfrom application developers to end users.

  26. Thank You! email: danica@dcs.shef.ac.uk

  27. More information... • D. Damljanovic, M. Agatonovic, H. Cunningham: FREyA: an Interactive Way of Querying Linked Data, 1st Workshop on Question-Answering over Linked Data, in conjunction with ESWC’11, 2011. (to appear) • D. Damljanovic, M. Agatonovic, H. Cunningham: Natural Language Interfaces to Ontologies: Combining Syntactic Analysis and Ontology-based Lookup through the User Interaction. In Proceedings of the 7th Extended Semantic Web Conference (ESWC 2010), Springer Verlag, Heraklion, Greece, May 31-June 3, 2010. PDF • D. Damljanovic, M. Agatonovic, H. Cunningham: Identification of the Question Focus: Combining Syntactic Analysis and Ontology-based Lookup through the User Interaction. In Proceedings of the 7th Language Resources and Evaluation Conference (LREC 2010), ELRA 2010, La Valletta, Malta, May 17-23, 2010. PDFD. Damljanovic. Towards portable controlled natural languages for querying ontologies. In Rosner, M., Fuchs, N., eds.: Proceedings of the 2nd Workshop on Controlled Natural Language. Lecture Notes in Computer Science. Springer Berlin/Heidelberg, Marettimo Island, Sicily (September 2010)

More Related