Presentation Transcript

1. Automatically Mapping Natural Language Requirements to Domain-Specific Process Models

   Uthayashanker  Thayasivam, Kunal Verma, Alex Kass, Reymonrod Vasquez
2. Outline Requirements analysis for packaged software Research challenges Matching approach Evaluations
3. requirements analysis for packaged software
4. Requirements Analysis for enterprise software is a complex task… Requirements elicitation for packaged software (e.g., ERP) consists of matching project requirements to software capabilities Classifying each requirement as a fit/gap Over the years, large enterprises have developed reusable process models to capture the key functional capabilities of tools Process models contain: Reusable requirements with fit/gap information Reusable artifacts such as , reusable reports, code and test scripts
5. Some questions to be answered in context of package software implementations Process model based estimator How complete are my requirements? Are there any gaps in my requirements? Which part of my requirements are standard vs. non standard? How much will the project cost? Which parts of the process model are relevant to my project? RFP/Requirements Reusable Requirements Repositories Are there standard requirements that can be reused? High Performance Business Process Model Reusable Code Reusable Asset Repository ReusableReports/Forms Are there any assets that can be reused? ImplementationBest Practices
6. Mapping requirements to process models is non-trivial task Sample Process Model: Typically 3000-4000 process model elements and 1000-2000 reusable requirements Sample Requirements: Typically 5000-10000 requirements for large projects
7. Some examples of mappings
8. Research challenges
9. Research Challenges Unlike web-search, many capabilities are about similar objects Relationship between verb and object is important For example, “invoice” appears in 95/1203 capabilities Tool must be able to use verbs, prepositional phrases and their relationships to distinguish between similar capabilities There is no standardization of syntaxes for requirements Cannot assume RAT-ification (active voice etc.) Tool must be able to deal with different ways of expressing same intent System shall allow the user to create a purchase order. Purchase order shall be created by the user
10. Research Challenges Users have their own vocabulary. May not necessarily follow process model vocabulary User may say “bill”, “debit memo”, “debit note” or “invoice” to mean same thing Tool must have a model of common concepts and their semantic relationships in a domain Lot of implicit details in requirements. Apparent to users, but not to tool “The system shall allow the user to enter discount code” When a user talks about “entering a discount code”, he/she implicitly refers to “creating a purchase order” activity Tool must have a model of how typical role-based actions in the domain
11. Matching approach
12. Overview of Matching Approach between a Requirement and a Capability
13. Similarity Measure 1: Extract Verb-entity-prepositional phrases triples and match them This measure works well for well-formed requirements in active voice and passive syntaxes because it depends output from syntactic parser
14. Example 1 – VOP String Matching R: Sales order shall be created by a user C: Create sales order VOP Term Extractor VOP: {created, sales order} VOP: {create, sales order} Stemmer and String Matcher ProcGap Match Score = 1.0
15. Example 2 – VOP Semantic Matching R: The system shall allow the user to create a debit memo. C: Create invoice VOP Term Extractor VOP: {create, invoice} VOP: {create, debit memo} debit memo subClassOfinvoice Semantic Matcher ProcGap Semantic Graph Match Score = 0.85
16. Example 3 – VOP Semantic Matching R: The system shall allow the user to select transportation mode. C: Create delivery document VOP Term Extractor VOP: {create, delivery document} VOP: {select, transportation mode} Semantic Matcher “Transportation mode “partOf“delivery document” ProcGap Semantic Graph Match Score = 0.72
17. Similarity Measure 2 – Cosine Similarity Requirement R , Capability C This measure does not use the parse tree. It works on extracted entities and verb vector and hence is independent of the structure of the sentence
18. Example 4 – Cosine Similarity R: The system shall allow the user to create contracts and use them as reference for order creation C: Create Purchase Order with Reference to a Contract Create delivery document VOP Term Extractor TV: {purchas=2, order=2, refer=2, creat=2, contract=2} TV: {creation=1, order=2, refer=1, creat=2, contract=2} Cosine Matcher ProcGap Match Score = 0.83
19. Classifying the types of matches
20. Semantic Graph is semi-automatically created for each process model Action Entity Create Bill Invoice Performa Invoice Change Edit Discount Price Type in Add Customer Surcharge Credit Invoice Select Choose Item Total Semi Automatic Graph Population Indexing Phrase extraction (Noun, Verb, Preposition) Rule based relation extraction Manual pruning PartOf SubClass
21. Evaluation
22. Experiments - Objectives Can we do better? Comparison with other techniques Does it significantly help? Case Study
23. Experimental setup Dataset ”ERP Process Model for the Chemical Industry” 3116 Capabilities 189 requirements Gold Standard Voting: Team of 3 project members 334 mappings 147 requirements
24. Comparative Evaluation
25. Experiment – II : Case Study Limited amount of time 6 participants 2-10 years of experience in the IT industry Varying amount of expertise about the chemical industry Randomly grouped into 2 Standard matching process Manual matching aided with M.S. Excel ProcGap matching process Manual mapping on top of ProcGap mapping aided with ProcGap UI Wilcoxon-mannWhiteney test ProcGapoutperformed the manual group, both in terms of precision and recall with the level of significance at 0.05
26. BACK UP SLIDES Back-up slide
27. Example of cosine similarity
28. Most frequently uses entities are used as a starting point
29. RICEF - P Process R- ReportsR stands for Report developments. All the technical objects/developments that deal with programming of SAP Reports. It includes different types of reports; Simple reports where you output using WRITE statements, Simple ALV reports, ALV Grid, ALV Grid with advanced functionality, etc. I - InterfacesI stands for Interface developments. Interfaces are ALE/IDOCs development. Involves not just ABAP programming for IDOCs but also IDOC customization. C - ConversionC stands for conversion objects. Conversion refers to BDC programing. Data upload from legacy system in flat files format to SAP system is done via Conversion Objects. This involves uploading data through BDC, LSMW, BAPI, etc. E - EnhancementsEnhancements are User Exits, Customer Exits, BADI's etc. ABAP code that has to be written in order to enhance SAP system functionality. F - FormsForms include SAP Smartforms, SAP Scripts. Technical development that deals with fetching necessary data from SAP system and displaying in terms of forms for printout are classified as under Forms.