Using Natural Language Program Analysis to Locate and understand Action-Oriented Concerns

1. Using Natural Language Program Analysis to Locate and understand Action-Oriented Concerns David Shepherd, Zachary P. Fry, Emily Hill, Lori Pollock, and K. Vijay-Shanker Presented By: Paul Heintzelman

2. Global Concepts • Concept assignment problem • Hybrid of structural and natural language information • Concern Comprehension • Action-oriented relations between identifiers • Represented by Action-oriented identifier graph model (AOIG)

3. Why Action-Oriented Concerns • In OOP • Code is organized by objects • Objects are nouns • Objects and actions conflict • Code organized by objects causes actions to be scattered • Therefore in OOP action-oriented concerns tend to be scattered and more difficult to locate

4. Paper Contributions • AOIG • Interactive query expansion algorithm • A result graph construction algorithm • An Eclipse plug-in • Evaluation • Comparison of search effectiveness of tools • Per task analysis • Comparison of user effort

5. AOIG

6. State of the Art • Search-based approaches • Lexical searches • Lead to over-generalized searches • Information retrieval • Does not separate verbs and objects • Uses word frequency • Program navigation • Uses structural information e.g. call, inheritance graphs... • Accurate but difficult to seed • Dynamic approaches • Requires test case to enact concept

7. Challenges • Map high level concepts to queries • Aid user in mapping concepts • Inability to search with high precision and recall • Search NLP representation of concern • Understanding large result sets • Return results in an explorable graph

8. Overview of Approach • User formulates a query • Query must include verb-direct object pairings • User expands query • Recommendations based on query words and source code • Searches the AOIG • Interact with result graph

9. Independent Variables • Search Tools • Find-Concept • ELex built in Eclipse search • GES Google Eclipse search (modified) • Search Tasks • Application concept pairing • Human Subjects • 13 professional programmers • 5 grad students

10. Application Concept Pairing • Applications • 4 large open source java projects • 9 concepts taken from bug reports • 1 training application • 2 concepts

11. Forming the Initial Query • User generates abstract initial query • e.g. “automatically finish the word” • User decomposes abstract query into verb-direct object pairs • e.g. “finish” and “word” • Find concept maintains both verb query and direct object query • Initial query expansion • User is presented with alternative forms of words in both queries

12. Query Expansion • Iterative steps • Generate recommended list • Similar semantics is weighted more heavily than similar use • 10 ranked recommendations • User examines recommendations • User selects words to add to queries • User can view a list of methods fitting the current queries • Stop when user is satisfied • Augment user query with get, set, execute, construct • Use AOIG to map verb-direct object pairs to source code • Generate result graph

13. Word Recommendation • Similar semantics • Stemming • Recommends different forms of words in either list • e.g. If “finish” is in verb-query, “finished” will be recommended • Synonyms • Recommend a word if synonym exists in either list • e.g. Recommend “complete” if “finish” is in list • Similar use • Recommend words that occur near words in either query • e.g. Recommend “word” if “complete” is in the verb query and “complete word” is in the AOIG

14. Evolution of a Query

15. Result Graph

16. Find-Concept Process

17. Research Questions • Which search tool is most effective at locating concerns by forming and executing a query? • Which search tool requires the least amount of human effort to form an effective query?

18. Evaluation • Effectiveness • Use the harmonic mean of precision and recall (f-measure) • (2 * precision * recall)/(precision + recall) • Result set is compared to evaluation set • Evaluation set is 90% generated by a member unfamiliar with the work of this paper • Effort • Measured amount of time required to form each query

19. Experimental Setup • Training • Subjects are guided through the use of each tool on the two training tasks • Task setup • Users are presented concepts in a visual form • Users confirm that they understood each task

20. Experimental Procedure • 9 tasks • 18 programmers • 6 groups • 6 of every task tool combination

21. Results • Find-Concept vs. ELex • Consistently outperformed ELex • Find-Concept vs. GES • Outperformed GES on 4 tasks • Outperformed by GES on 2 tasks • AOIG to blame? • Performed equally to GES on 3 tasks

22. Effectiveness

23. Effort • Human Effort was very similar with all tools

24. Threats to Validity • The selected tasks favored one tool • Concerns selected from bug reports • Evaluation sets created for evaluation • 90% generated by member unfamiliar with work • Results may not generalize to all Java applications • Tested on reasonably-sized applications • Results may not generalize to all types of concepts

25. Conclusion • Interactive query expansion algorithm • Graph construction algorithm • Find-Concept performs well against state of the art tools • All evaluated tools required similar human effort

26. Future Work • Create a more effective AOIGBuilder • Evaluate the effect of application’s quality and size on results • Evaluate the effect of incorporating naming conventions • Perform a study on how many tasks focus on actions • Automate query expansion

27. Additional threats to validity • Effort and Effectiveness are not really independent • Relies heavily on unjustified heuristic • Augmenting query • Search tools are often used in conjunction with structural tools