Learning To Understand Web Site Update Requests

1. Learning To Understand Web Site Update Requests William W. Cohen, Einat Minkov, Anthony Tomasic

3. Directory Info • Courses • Projects • Publications • Sponsors • Events • …

4. Web site update Database Not interested (and better not) interact directly with the database. Dislike active effort.Happy with natural language.

5. Web site update Database Not interested (and better not) interact directly with the database. Dislike active effort.Happy with natural language. Natural language Formal language

6. Web site update Database Not interested (and better not) interact directly with the database. Dislike active effort.Happy with natural language. a BOTTLENECK Natural language Formal language

7. Web site update Database Not interested (and better not) interact directly with the database. Dislike active effort.Happy with natural language. a BOTTLENECK Natural language Formal language Can we automate the webMaster? • The Problem(s): • Natural Language Processing • Email text • The domain is fixed, however, database schema changes over time.

8. Assume DB-backed website, where schema changes over time Address changes in factual content of website Assume update of one tuple The system should interact with the user, where the goal is to avoid errors in website update: User requests some change, via NL email System analyzes request System presents preview page and editable form version of request user can verify correctness (vs case for DB queries, Q/A,...) => source of training data Partial correctness is useful. A learning system. Message understanding is decomposed into entity recognition and classification tasks. Framework and motivation SCOPE THE HUMAN FACTOR FRAMEWORK

9. LEARNER offline training data Shallow NLP Feature Building Classification C requestType POS tags email msg NP chunks targetRelation C words, ... targetAttrib C features Information Extraction newEntity1,... oldEntity1,... entity1, entity2, .... C keyEntity1,... otherEntity1,... preview page user-editable form version of request Update Request Construction User confirm? web page templates database

10. “Mr.Web” [Lockerd et-al, CHI-03] NL interfaces to DBs, to facilitate queries. Learning commonly used as tool to develop non-adaptive NLP components. Semantics learning systems: CHILL [Zelle & Mooney, 96], [Miller et-al, 96] Many non-learning NLP systems incorporate domain knowledge (hand-tuned). What’s new: Understanding update requests, rather than queries or statements: partially correct analysis still useful here. Deep analysis in limited but evolving domain Email text Related work

11. Outline • The experimental corpus • Request decomposition • Sub-task evaluation • End-to-end evaluation • Conclusions • Future directions

12. Experimental corpus

13. Experimental corpus Mike Roborts should be Micheal Roberts in the staff listing, pls fix it. Thanks - W User1

14. Experimental corpus Mike Roborts should be Micheal Roberts in the staff listing, pls fix it. Thanks - W User1 On the staff page, change Mike to Michael in the listing for “Mike Roberts”. User2 User3 ....

15. Experimental corpus Add this as Greg Johnson’s phone number: 412 281 2000 User1 Please add “412-281-2000” to greg johnson’s listing on the staff page. User2 User3 ....

16. Experimental corpus Add this as Greg Johnson’s phone number: 412 281 2000 User1 Please add “412-281-2000” to greg johnson’s listing on the staff page. User2 User3 .... 617 examples: ~20 subjects x ~30 tasks

17. Preprocessing – entity names are made distinct Add this as Greg Johnson’s phone number: 412 281 2000 User1 Please add “543-341-8999” to fred flintstone’s listing on the staff page. User2 User3 .... Modification: to make entity-extraction reasonable, remove duplicate entities by replacing them with alternatives (preserving case, typos, etc)

18. Experimental corpus • 617 examples • Factual updates (with some exceptions) • Each update request refers to a single tuple in the database. • In the underlying database, a relation does not contain two attributes or more of the same “type” (entity). • Text is ungrammatical and noisy!

19. Outline • The experimental corpus • Request decomposition • Sub-task evaluation • End-to-end evaluation • Conclusions • Future directions

20. Entity recognition Entity role Request type Relation Attribute

21. Shallow NLP Feature Building C requestType POS tags email msg NP chunks targetRelation C words, ... targetAttrib C features Information Extraction newEntity1,... oldEntity1,... entity1, entity2, .... C keyEntity1,... otherEntity1,...

22. Outline • The experimental corpus • Request decomposition • Sub-task evaluation • End-to-end evaluation • Conclusions • Future directions

23. Shallow NLP Feature Building C requestType POS tags email msg NP chunks targetRelation C words, ... targetAttrib C features Information Extraction newEntity1,... oldEntity1,... entity1, entity2, .... C keyEntity1,... otherEntity1,... hi webmaster- on the Education Technology Personnel page under "staff", please change [RobertPaul]PERSON's Phone number from “x[365]PHONE" to “x[3655]PHONE". Thanks, [Martha]PERSON

24. 1. Entity recognition • ‘Offline’ training data. • Experiments: • hand-coded rules (cascaded FST in “Mixup” language) • Used CRF for learning • A standard feature set vs. “tuned” feature set • results are in entity-level F1 (harmonic avg of recall and precision) • Good performance, also for a small dataset.Users tend to use the terminology and formats of the website, resulting in reduced variability.

25. Robustness Evaluation System robustness: how robust is the learned model to new user styles or new requests? message (user 1,req 1) test message (user 2,req 1) .. message (user 1,req 2) train message (user 2,req 2) .. message (user 1,req 3) train message (user 2,req 3) ..

26. Robustness Evaluation System robustness: how robust is the learned model to new user styles or new requests? message (user 1,req 1) train message (user 2,req 1) .. message (user 1,req 2) test message (user 2,req 2) .. message (user 1,req 3) train message (user 2,req 3) ..

27. Entity recognition

28. Shallow NLP Feature Building C requestType POS tags email msg NP chunks targetRelation C words, ... targetAttrib C features Information Extraction newEntity1,... oldEntity1,... entity1, entity2, .... C keyEntity1,... otherEntity1,... hi webmaster- on the Education Technology Personnel page under "staff", please change [RobertPaul]key's Phone number from “x[365]OLD" to “x[3655]NEW".Thanks, [Martha]

29. 2. Role-based entity classification • Entity “roles”: • keyEntity: value used to retrieve a tuple that will be updated (“delete greg’s phone number”) • newEntity: value to be added to database (“William’s new office # is 5307 WH”). • oldEntity: value to be overwritten or deleted (“change mike to Michael in the listing for ...”) • irrelevantEntity: not needed to build the request (“please add .... – thanks, William”) .. change [RobertPaul] 's .... change .. from “x[365]" .... change .. to “x[3655]". • Features: • closest preceding “action verb” (add, change, delete, remove, ...) • closest preceding preposition • is the entity part of a determined NP • presence of a possession marker

30. Role-based classification results • Task of semantic nature • The text is semi-ungrammatical. • However, good results with a small, simple set of features • Semi-finite set of language patterns: ‘change’, ‘correction’, ‘update’, ‘replace’, ‘shouldbe’, ‘wrong’, ‘needs to be’ ..

31. Shallow NLP Feature Building C requestType POS tags email msg NP chunks targetRelation C words, ... targetAttrib C features Information Extraction newEntity1,... oldEntity1,... entity1, entity2, .... C keyEntity1,... otherEntity1,...

32. 3. Target relation classification Good results with “bag of words” features. Even better results when given the included entity types. (e.g, phone number  people relation)

33. Shallow NLP Feature Building C requestType POS tags email msg NP chunks targetRelation C words, ... targetAttrib C features Information Extraction newEntity1,... oldEntity1,... entity1, entity2, .... C keyEntity1,... otherEntity1,...

34. 4. Request type classification Can be determined from entity roles and action verb, except for deleteTuple and deleteValue. • “Delete the phone number for Scott” • “Pls delete the whole entry for Scott” Features: • counts of entity role types • action verbs • nouns in NPs which are (probably) objects of action verb • A small set of nouns, tagged with a dictionary Performance is waybetter with 12-words of schema-specific knowledge: dictionary of terms like phone, extension, room, office, ...Can be learnedfrom data.

35. Shallow NLP Feature Building C requestType POS tags email msg NP chunks targetRelation C words, ... targetAttrib C features Information Extraction newEntity1,... oldEntity1,... entity1, entity2, .... C keyEntity1,... otherEntity1,...

36. 5. Target attribute classification • “Delete the phone number for Scott”  phone • “Delete Scott’s extension”  phone • Additional features: • small dictionaries (BOW) e.g., phone, extension, line. • Can be learned from data due to redundancy • The vocabularies used in the corpus are small. • Conjecture: tendency to use the terminology of the website. Also, the vocabularies are naturally small.

37. Outline • The experimental corpus • Request Decomposition • Sub-task evaluation • End-to-end evaluation • Conclusions • Future directions

38. End-to-end performance Entity Recognition • Tasks are not independent • Consider noisy inputs • Evaluation criterion: % of completely correct messages. • Results in ~40% of perfectly processed messages. Relation Classification Entity Role Classification Request Type Classification Target Attr. Classification

39. End-to-end: Individual tasks Entity Recognition 85% Noisy inputs Relation Classification % of perfectly processed messages Entity Role Classification 99% 67% Request Type Classification 80% Target Attr. Classification

40. End-to-end: Composite tasks • The user would get the correct form in 79.2%of messages. Entity Recognition 85% Relation Classification Entity Role Classification 99% 67% Request Type Classification 80% Target Attr. Classification

41. End-to-end: Composite tasks • The user would get the correct form in 79.2%of messages. • The correct form, with all entities extracted correctly: 53.4% of messages. Entity Recognition 85% Relation Classification Entity Role Classification 99% 67% Request Type Classification 80% Target Attr. Classification

42. End-to-end: Composite tasks • The user would get the correct form in 79.2%of messages. • The correct form, with all entities extracted correctly: 53.4% of messages. • Fully correctly processed messages: 39.2% Entity Recognition 85% Relation Classification Entity Role Classification 99% 67% Request Type Classification 80% Target Attr. Classification

43. Conclusions • A promising rate of 40% messages processed correctly. This rate is expected to improve as data accumulates. • System architecture means allschema-dependent knowledge can be learned • Potential to adapt to changes in schema • Data needed for learning can be collected from user • Learning appears to be possible on reasonable time-scales (10s or 100s of relevant examples, not thousands) • The noisy informal email text can be successfully processed applying a learning approach, using small sets of syntactic features. • The system is robust to user style, and request variation. • Human subject experiments show partially correct results to be useful. Thus, a realistic adaptive, automatic webmaster assistant.

44. Future directions • Relax the restriction that each request concerns the update of one tuple per email. • Evaluate more complex entity types for the entity recognition component (coverage). • Entity recognition may be improved by database lookups. • Collective classification to improve on message-based classification performance (e.g., entity roles) as well as pipeline processing.

45. Thank You.  Questions?