Natural Language Interfaces to conceptual models: usability and performance

1. Natural Language Interfaces to conceptual models: usabilityand performance Danica Damljanović Danica Damljanović

2. Outline • NLIs to KBs and their usability • QuestIO – Question-based Interface to Ontologies • Demo and evaluation • Towards better usability using FREYA • Conclusion Danica Damljanović

3. Natural Language Interfaces • Allow users to interact with a system using written or spoken language to perform tasks which require knowledge of a formal query language • NLIs to structured data: • NLIs to DBs: TEAM, PRECISE • NLIs to KBs • NLIs to semi-structured data • Open-domain question answering systems • Other: • E.g. NLC: replacement for a programming language • Dialog and tutoring systems

4. NLIs to knowledge bases • (Kaufmann and Bernstein, 2007) • Natural Language Interfaces preferred to keywords, menu-guided, and graphical interfaces • (Linckels, 2007): • keywords preferred to NL interfaces Danica Damljanović

5. NLIs to KBs Danica Damljanović

6. Usability of NLIs • Who uses NLIs? • Application developers: customisation • End users: search • Usability • Effectiveness • Efficiency • User satisfaction Danica Damljanović

7. Usable NLIs to KBs: challenges • Robustness • Portability • What to show? • Understanding information need • Habitability Danica Damljanović

8. Habitability • Can the User and the System speakthe same language? Danica Damljanović

9. Question-based Interface to Ontologies Danica Damljanović

10. NL --> SPARQL query Query Execution Filteringconcepts Ranking concepts Query Creator Danica Damljanović

11. An Example 1.15 1.19 compare

12. Demo • http://www.tao-project.eu/researchanddevelopment/demosanddownloads/movies/questio/questio.html • http://www.tao-project.eu/researchanddevelopment/demosanddownloads/movies/prototype-tutorial/prototype-tutorial.html Danica Damljanović

13. User-centric task-based evaluation • Training: using video tutorials • 12 participants, 4 tasks: • 3 defined, e.g. “Find runtime parameters of Cebuano gazetteer.” • 1 free task: “Think of any task that you would like to perform using this prototype.” • Measured: • Efficiency, Effectiveness and User satisfaction Danica Damljanović

14. Efficiency: average time per task

15. Effectiveness: how successfully the tasks were finished?

16. Post –task survey: question 1

17. Post –task survey: question 2

18. Post –task survey: question 3

19. Post –task survey: question 4

20. Evaluation: conclusion • Browsing ontology mostly helpful – but only for users who are familiar with ontologies • Refinement pane: primitive, still favourable for defined tasks • Tasks not finished: • in cases when the user information need was not precisely expressed (undefined tasks) • When the answer was not in the ontology Danica Damljanović

21. “find out which are the runtime parameters of Cebuano Gazetteer” • “cebuano gazetter parameters” • “ What are the runtime parameters of cebuano gazetteer?“ • “what are the parameters of cebuano gazetteer?” • “Cebuano gazetteer runtime parameters“ • “Runtime parameters of cebuano gazetteer“ • “Cebuano runtime parameters“ • “Cebuano gazeteer“>>“Cebuano gazetteer“ • Find parameters by browsing the ontology Danica Damljanović

22. “cities in Europe” “cities in Europe“ Europe city <LocatedIn> <LocatedIn> <PartOf> Countries

23. Undefined tasks • “Developer of Tokeniser” • ”Projects about GATE ”, ”GATE web site” • ” Tokenizer ” vs. ” Tokeniser ” • ” Gazetteer ” vs. ”Gazeteer ” • ”horacio saggion publications” vs. ”horacio saggion articles” • ”Author of morphological analyser ” >> ”Developer of morphological analyser ” >> ”Developer and morphological analyser ” Danica Damljanović

24. Next steps: FREyA FREyA (Feedback, Refinement, Extended vocabulary, Agregation) Danica Damljanović

25. Example • the user's query: • Cities in Europe • FREyA: • Cities>>locatedIn >>Countries >> locatedIn >> Europe • Cities >>locatedIn >> Countries >> partOf >> Europe • Attempto: • which cities are located in countries that are part of Europe? Danica Damljanović

26. Challenges • Return answers in real time: • which rivers flow through Germany >> which rivers flow through (cities in) Germany • Refinement model - an ontology? • with concepts such as query (hasAnswer) answer, refinement, feedback, subject of query (main subject), domain, context. • find out what the user wants and instantiate the refinement ontology. Danica Damljanović

27. Main subject: Cities in Europe • Main subject based on several heuristic rules: • first noun in the query, but common noun? Jonny depp date of birth and when johny depp was born? • If it starts with when then the main subject is some date • If it starts with where the main subject is a location • If it starts with who the main subject is a person/organization, unless otherwise specified • If it starts with How many look at the first noun to be the main subject but the count of results should be an answer

28. Examples • Johnny Depp date of birth • Feedback: • Actor Johnny Depp date of birth or • What is the date of birth of the actor Johnny Depp? • When was Johnny Depp born? • Feedback: • When was the actor Johnny Depp born? • Main subject is the date but it is tied with the first noun • When was the actor who played the main role in What’s eating Gilbert Grape born?

29. Examples • Constraints: • <Main subject> hasOldName continent Europe • <Main subject> locatedIn continent Europe • NLG: • Output: • <Main subject> is/are: list of results.

30. Expected contribution Usability improvement of NLIs: Eliminate training Improve the performance by implementing user-system interaction using FREYA: help the user to easily familiarize himself with the system capabilities express his need more precisely, in a way which is understandable by the system 9/4/2014 Danica Damljanović 44

31. Thank you • Questions? • Natural Language Processing for Serbian: • PMF • GoranNenadic • Stasa Danica Damljanović

32. References • (Noy et al., 2001)N. Noy, M. Sintek, S. Decker, M. Crubezy, R. Fergerson, and M. Musen. Creating Semantic Web Contents with Protege-2000. IEEE Intelligent Systems, 16(2):60-71, 2001. • (Kiryakov et al., 2004) A. Kiryakov, B. Popov, D. Ognyano, D. Manov, A. Kirilov, and M. Goranov. Semantic annotation, indexing and retrieval. Journal of Web Semantics, ISWC 2003 Special Issue, 1(2):671-680, 2004. • (Guha et al., 2003) R. Guha, R. McCool, and E. Miller. Semantic search. In WWW '03: Proceedings of the 12th international conference on World Wide Web, pages 700-709, New York, NY, USA, 2003. ACM. • (Lei et al., 2006) Y. Lei, V. Uren, and E. Motta. Semsearch: a search engine for the semantic web. In Managing Knowledge in a World of Networks, pages 238{245. Springer Berlin /Heidelberg, 2006. • (Kaufmann & Bernstein, 2007) E. Kaufmann and A. Bernstein. How useful are natural language interfaces to the semantic web for casual end-users? In Proceedings of the Forth European Semantic Web Conference (ESWC 2007), Innsbruck, Austria, June 2007. • (Serge Linckels, 2007) C. M. Serge Linckels. Semantic interpretation of natural language user input to improve search in multimedia knowledge base. it - Information Technologies, 49(1):40-48, 2007.