Natural Language Interfaces to Ontologies: usability and performance (Transfer report)

1. Natural Language Interfaces to Ontologies: usabilityand performance(Transfer report) Student: Danica Damljanović Supervisor: Prof. Dr Hamish Cunningham Danica Damljanović

2. Introduction • Semantic Web • Ontologies and knowledge bases • Semantic search • Semantic Web interfaces

3. Semantic Search Interfaces • Protégé (Noy et al., 2001) • KIM (Kiryakov et al., 2004) • TAP (Guha et al., 2003) • GetData • Reflection • TAPSearch • SemSearch (Lei et al., 2006) • NLIs to KBs Danica Damljanović

4. User preferences • (Kaufmann and Bernstein, 2007) • Natural Language Interfaces preferred to keywords, menu-guided, and graphical interfaces • (Linckels, 2007): • keywords preferred to NL interfaces Danica Damljanović

5. NLIs to KBs: challenges • Robustness • Portability • What to show? • Understanding information need • Habitability • Usability Danica Damljanović

6. Natural Language Interfaces • Usability • Effectiveness • Efficiency • User satisfaction • Who uses NLIs? • Application developers: customisation • End users: search Danica Damljanović

7. Customisation and Retrieval Performance Danica Damljanović

8. Ontology engineer Domain expert Customisation of NLIs to KBs Ontology editing (e.g. using Protege) WordNet … NLI for querying NLI for Ontology authoring Domain lexicon Domain knowledge

9. End users point of view • Can the User and the System speakthe same language? Danica Damljanović

10. Design recommendations • System Vocabulary>>User Vocabulary: • Feedback • Guided interfaces • Personalised vocabulary • User Vocabulary>> System Vocabulary: • Clarification dialogs • Query refinement • Controlling the relevance • Ranking suggestions • Defining similarity • User profiles Danica Damljanović

11. First experiments: Question-based Interface to Ontologies Danica Damljanović

12. NL --> SeRQL query Danica Damljanović

13. An Example 1.15 1.19 compare

14. Evaluation: Initial results We evaluated: • coverage and correctness • scalability and portability Danica Damljanović

15. Comparison with Aqualog • We removed 6 questions that we knew were not supported by Aqualog • 1 conjunction query “What are the run parameters of POS Tagger and Sentence splitter?” • 1 query with brackets “Does GATE have a coreference resolution component (PR)?” • 1 query starting with “How many. . . ” • 3 queries not in a form of a full-blown question, for example “I cannot get Wordnet plugin to work“. Danica Damljanović

16. Evaluation on scalability Danica Damljanović

17. Expected contributions and research plan Addressing challenges without compromising retrieval performance: How existing NLIs to KBs can increase the performance without a significant additional cost for customisation? How methods for assisting the user in formulating the query can influence the retrieval performance? Richer navigation and semantic search interface which extend the concept of faceted search Improving QuestIO 9/15/2014 Danica Damljanović 24

18. Expected contributions and research plan • Evaluation with golden standard dataset which would be one of the contributions of this PhD: • Knowledge base • Set of queries • Set of relevant SPARQL queries • Exploring methods for usability and their impact on retrieval performance

19. Related publications Journals: • Danica Damljanovic, Kalina Bontcheva: Towards Enhanced Usability of Natural Language Interfaces. Annals of IS: Special Issue on “Semantic Web & Web 2.0”. Springer. 2009. (to appear) Workshops: • Damljanovic, D.,Bontcheva, K.. Enhanced Semantic Access to Software Artefacts. In Workshop on Semantic Web Enabled Software Engineering (SWESE) held in conjunction with ISWC'08, Karlsruhe, Germany, October 2008. Conferences: • Tablan, V., Damljanovic, D., Bontcheva, K.. A natural language query interface to structured information. In Proceedings of the 5h European Semantic Web Conference (ESWC 2008), Tenerife, Spain, June, 2008. • Damljanovic, D., Tablan, V., Bontcheva, K.: A text-based query interface to owl ontologies. In: 6th Language Resources and Evaluation Conference (LREC), Mar- rakech, Morocco, ELRA (May 2008). Posters: • Natural Language Queries for Enhanced Knowledge Access presented at Summer School on Multimedia Semantics Analysis, Annotation, Retrieval and Applications (SSMS07) , Glasgow, UK, July 15-21, 2007.

20. Thanks • …you for your attention! • Hamish Cunningham and Kalina Bontcheva for continuous support • Valentin Tablan for the help with QuestIO • Vanesa Lopez from Open University for allowing me to play with AquaLog • Professor Abraham Bernstein and Esther Kaufmann from the University of Zurich for sharing with me the Mooney dataset in owl format. • Milan Agatonovic for continuous support

21. References • (Noy et al., 2001)N. Noy, M. Sintek, S. Decker, M. Crubezy, R. Fergerson, and M. Musen. Creating Semantic Web Contents with Protege-2000. IEEE Intelligent Systems, 16(2):60-71, 2001. • (Kiryakov et al., 2004) A. Kiryakov, B. Popov, D. Ognyano, D. Manov, A. Kirilov, and M. Goranov. Semantic annotation, indexing and retrieval. Journal of Web Semantics, ISWC 2003 Special Issue, 1(2):671-680, 2004. • (Guha et al., 2003) R. Guha, R. McCool, and E. Miller. Semantic search. In WWW '03: Proceedings of the 12th international conference on World Wide Web, pages 700-709, New York, NY, USA, 2003. ACM. • (Lei et al., 2006) Y. Lei, V. Uren, and E. Motta. Semsearch: a search engine for the semantic web. In Managing Knowledge in a World of Networks, pages 238{245. Springer Berlin /Heidelberg, 2006. • (Kaufmann & Bernstein, 2007) E. Kaufmann and A. Bernstein. How useful are natural language interfaces to the semantic web for casual end-users? In Proceedings of the Forth European Semantic Web Conference (ESWC 2007), Innsbruck, Austria, June 2007. • (Serge Linckels, 2007) C. M. Serge Linckels. Semantic interpretation of natural language user input to improve search in multimedia knowledge base. it - Information Technologies, 49(1):40-48, 2007.