Eliciting Formal Models From Informal Requirements

1. Eliciting Formal Models From Informal Requirements Some issues and an approach. Nikhil Dinesh, David E. Arney, Aravind K. Joshi, Owen Rambow, Martha Palmer and Insup Lee University of Pennsylvania Chicago, September 24 2003

2. Outline • Introduction • Example • Specification Language for Requirements • The approach • Future work

3. Issues • What is the right specification language for requirements ? • Natural Language • Formal Specification Language • Formal Specification Language with an NL-looking restricted language interface • Evaluation • What metrics can be applied in the evaluation of such a system

4. Overall Approach NL requirements NL-based Finite State Machine Corrections to requirements in NL Requirements Engineer Policy Specifier Extended Finite State Machine Verification & Validation Errors

5. Approach to elicit an EFSM* • Work is preliminary, no implementation • Outline the stages in eliciting an EFSM • Desired output representation at each stage • How to compute from the NL requirements in terms of : • What is available • What is needed • What is achievable with existing tools and what research is needed for the long range *An Extended Finite State Machine is an FSM with variables. (as in the HASTEN project)

6. Outline • Introduction • Example • Specification Language for Requirements • The approach • Future work

7. Example from FDA CFR* 610.40 (a) Except as specified in paragraphs (c) and (d) of this section, you must test all samples for evidence of infection due to the following communicable disease agents: (i) Human immunodeficiency virus (ii) Hepatitis-B virus * A policy document from Food and Drug Administration, Code of Federal Regulations. There are several volumes each of which is updated once each calendar year and issued on a quarterly basis.

8. Example from FDA CFR 610.40 (a) Except as specified in paragraphs (c) and (d) of this section, you must test all samples for evidence of infection due to the following communicable disease agents: (i) Human immunodeficiency virus (ii) Hepatitis-B virus (b) To test for evidence of infection due to communicable disease agents in paragraph (a), you must use a screening test approved by the FDA.

9. Example from FDA CFR 610.40 (a) Except as specified in paragraphs (c) and (d) of this section, you must test all samples for evidence of infection due to the following communicable disease agents: (i) Human immunodeficiency virus (ii) Hepatitis-B virus (b) To test for evidence of infection due to communicable disease agents in paragraph (a), you must use a screening test approved by the FDA. You must perform one or more such test, as necessary, to reduce adequately and appropriately the risk of transmission of communicable disease.

10. Outline • Goals • Example • Specification Language for requirements • The approach • Future work

11. Specification Language for Requirements • Natural Language (NL) • Specification is accessible to people • Properties should correspond to the requirements • Hard to compute properties • Formal Specification Language (FSL) • Allows specification to be easily verified • Not easily accessible to people such as policy users • Application should motivate the choice • What is the right choice for the policy domain ?

12. Policy • Large number of policy documents in NL • Hand conversion to FSL is expensive and error prone. • Human readability • Policy is interpreted by humans. • Is this just an interface problem ?

13. Interfaces to Specification Languages • Suppose policy requirements were in a formal specification language • Interface provides NL-looking, restricted language • How easily can it be read? • We examine one such interface

14. Specifying a Property • PROPEL – An approach supporting property elucidation [Smith et al], 2002. • For our example, • Core phrase : Exactly one occurrence of arrival of a sampleeventually leads to one or more occurrences of test for diseases. • Repetition Phrase : The above behavior is repeatable • Scope Phrase : This property must hold before the first occurrence of P (where P is a state corresponding to “Exceptions in paragraphs (c) and (d)”)

15. How do properties relate to the NL document ? • Core phrase : Exactly one occurrence of arrival of a sample eventually leads to one or more occurrences of test for diseases • Repetition Phrase : The above behavior is repeatable • Scope Phrase : This property must hold before the first occurrence of P Test for disease Sentence 1 Exceptions in (c) and (d)

16. How do properties relate to the NL document ? • Core phrase : Exactly one occurrence of arrival of a sample eventually leads to one or more occurrences of test for diseases • Repetition Phrase : The above behavior is repeatable • Scope Phrase : This property must hold before the first occurrence of P One or more tests Sentence 3

17. How do properties relate to the NL document ? • Core phrase : Exactly one occurrence of arrival of a sample eventually leads to one or more occurrences of test for diseases • Repetition Phrase : The above behavior is repeatable • Scope Phrase : This property must hold before the first occurrence of P Arrival of a sample ? Repeatability

18. How do properties relate to the NL document ? • To enforce use of a screening test: • test for disease -> screening test for disease • “test for disease” from Sentence 1 • “screening” from Sentence 2 • Correspondence becomes harder • Interface starts to look like an FSL • Not easily accessible to policy users

19. Outline • Goals • Example • Specification Language for requirements • The approach • Future work

20. NL Document EFSM-like representation from NL document NLFSM EFSM

21. NLFSM Agree to perform test Agree to use screening test Do the test Have you tested ? Have you reduced risk?

22. NLFSM to EFSM Clause a V(a) true V(a) == true Clause b

23. EFSM

24. NL Document Clause Connective Dependency Structure NLFSM Temporal Ordering Of Clauses EFSM

25. Clauses and Connectives (a) Except as specified in paragraphs (c) and (d) of this section, you must test all samples for evidence of infection due to the following communicable disease agents: (i) Human immunodeficiency Connective virus (ii) Hepatitis-B virus as specified in paragraphs (c) and (d) of this section Explicit Connective Except Clauses you must test all samples for evidence of infection due to the following communicable disease agents : (i) Human immunodeficiency (ii) Hepatitis-B virus

26. Clauses and Connectives (b) To test for evidence of communicable disease agents in paragraph (a), you must use a screening test approved by the FDA. To test for evidence of communicable disease agents in (a) Implicit Connective In order to Clauses You must use a screening test approved by the FDA

27. Predicates and Arguments John ate the apple Predicate: ate(Agent,Object) Dependency Structure ate (Object) The apple (Agent) John

28. Arguments of Connectives • Except • Exception • Default

29. Arguments of Connectives • In order to • Action • Purpose

30. Arguments of Connectives • In order to • Action • Purpose

31. Arguments across sentences • If X is true do action A. Otherwisedo action B. Otherwise (Condition) X is true (Action) Do action B

32. Computing the Dependency Structure • Discourse Connectives vs Verbs • arity, Verbs (1 – 3), Connectives (2) • Locating the arguments, easier for verbs • Roles of the arguments, verb-specific (Propbank), connective-specific (our approach)

33. Computing the Dependency Structure • Chunking • [Except] [as specified in paragraphs (c) and (d) of this section] [you must test all samples for evidence of infection due to the communicable disease agents: (1) Human immunodeficiency virus (2) Hepatitis B virus] • Simple for the cases where there is only one connective in the sentence.

34. Computing the Dependency Structure • You should test donation for evidence of all disease, exceptif it is a dedicated donation, you need not test for diseases (a)(5) and (a)(6).

35. Computing the Dependency Structure • Three Generative, Lexicalized Models for Statistical Parsing, Michael Collins, Proceedings of the 35th Annual Meeting of the ACL, Madrid (1997) • Statistical Parsing with an automatically-extracted tree adjoining grammar, David Chiang, Proceedings of the ACL, Hong Kong (2000) • Discriminative Reranking for Natural Language Parsing, Michael Collins, ICML (2000)

36. Computing the Dependency Structure • Resources available • Parsers compute rich dependency structure at the sentence-level • Accuracy of approximately 90% (Collins 2000) • Efforts underway for parsers at the discourse-level (DLTAG, Discourse Treebank) • Resources required • Some annotation of requirements documents • Labeling of connective-specific roles

37. NL Document Clause Connective Dependency Structure NLFSM Temporal Ordering Of Clauses EFSM

38. Temporal Ordering of Clauses • Related to programs • Cannot impose an order from an operational perspective (there may be cycles) • But can an order be imposed from a syntactic perspective – the way we write programs ?

39. Temporal Ordering of Clauses while(i < 9) i<9 while(i < 9){ i = i + 1; } i = i – 1; False True True False True i = i + 1 i = i - 1 i=i+1 i=i-1 Temporal Tree Operational FSM

40. Computing the Temporal Ordering(Between Clauses in a Sentence) Exception before the Default

41. Computing the Temporal Ordering(Between Clauses in a Sentence) Action before the check for Purpose

42. Computing the Temporal Ordering(Between Clauses in a Sentence) Action before the check for Purpose

43. Computing the Temporal Ordering(Between Clauses in Different Sentences) Agreement to use a screening test precedes agreement to test Noun-Verb “link” through the light verb (use) Check that test has taken place should follow the test Finite Verb – Nonfinite Verb “link”

44. Computing the Temporal Ordering(Between Clauses in Different Sentences)

45. Computing the Temporal Ordering(Between Clauses in Different Sentences) “tests” and “screening tests” are related. But the order is not quite clear from these in isolation. However these clauses are both scoped by purpose clauses reduce follows test Desirable to keep scopes nested.

47. Computing the Temporal Ordering • Resources available • Work on noun coreference (Morton) • New Machine Learning algorithms (Maximum Entropy Models, Conditional Random Fields etc) • Research needed • Granularity of annotation • Incorporating world knowledge

48. NL Document Clause Connective Dependency Structure NLFSM Temporal Ordering Of Clauses EFSM

49. From Temporal Trees to NLFSM • Find the phrases/connectives indicative of iterative constructs • while, for (connectives) • “one or more” (phrases) • Identify the scope and add the back-edges in the temporal tree. • Scope given by purpose clauses or relations between verbs