1 / 38

A Framework for Writing Trigger-Action Todo Comments in Executable Format

TrigIt is a framework that allows developers to write trigger-action comments as specifications in an executable format. By utilizing the TrigIt DSL embedded in the host language (Java), developers can easily write triggers and actions and automate the process of checking triggers and executing associated actions. TrigIt removes all its specifications before execution, ensuring that outdated or forgotten trigger-action comments do not cause issues.

petery
Download Presentation

A Framework for Writing Trigger-Action Todo Comments in Executable Format

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Framework for Writing Trigger-Action Todo Comments in Executable Format Pengyu Nie, Rishabh Rai, Junyi Jessy Li, Sarfraz Khurshid, Raymond J. Mooney, and Milos Gligoric August 29, 2019 ESEC/FSE 2019 Tallinn, Estonia supported by

  2. Motivation (1) From Google Guava protected AbstractStreamingHasher(intchunkSize, intbufferSize) {   // TODO(kevinb): check more preconditions   // (bufferSize >= chunkSize) if this is ever public checkArgument(bufferSize % chunkSize == 0); ...} Trigger The method is public  Action checkArgument(bufferSize >= chunkSize); • The comment is written in natural language, thus a developer needs to … • locate the comment • check if the trigger evaluates to true • execute the action • Tediousand error-prone

  3. Motivation (2) 2012/11: The comment was added 2015/5:   The trigger was satisfied 2015/10: The action was executed 2017/10: The comment was removed From Apache Struts void testDynamicAttributesSupport() throws Exception { ... // TODO : remove expectedJDK15 and if() after switching to Java 1.6 String expectedJDK15 = "<input type=\"text\" ..."; String expectedJDK16 = "<input type=\"text\" ..."; if (result.contains("foo=\"bar\" ...")) assertEquals(expectedJDK15, result); else assertEquals(expectedJDK16, result); ...} Trigger switch to Java 1.6 Action remove a variable and if statement

  4. Motivation (3) • Trigger-action comments are frequently used • We have found hundreds of trigger-action comments in 10 open-source projects • Trigger-action comments are 10+% of developer comments • Like any other comments written in natural language, they frequently become outdated or forgotten • We contacted 10 developers from large software companies and asked if they write trigger-action comments; 7 (out of 8 replies) answered yes • They confirmed they have a lot of trigger-action comments, and lack of automation is a big issue

  5. TrigIt • A framework for writing trigger-action comments as specifications in executable format • Developers write triggers and actions in the TrigIt DSL • Embedded in the host language (Java) • Utilizes native syntax (if-statements) with different semantics • TrigIt checks the triggers after compilation and before execution • If a trigger satisfies, TrigIt executes the associated actions • TrigIt removes all its specifications before execution

  6. Example TrigIt Specification (1) From Google Guava protectedAbstractStreamingHasher(intchunkSize, intbufferSize) {   // TODO(kevinb): check more preconditions   // (bufferSize >= chunkSize) if this is ever public if (trigItIsPublic()) checkArgument(bufferSize >= chunkSize); checkArgument(bufferSize % chunkSize == 0); ...} @TrigtItMethod booleantrigItIsPublic() { returnTrigIt.getMethod("<init>", int.class, int.class).isPublic(); }

  7. Example TrigIt Specification (2) From Apache Struts void testDynamicAttributesSupport() throws Exception { ... // TODO : remove expectedJDK15 and if() after switching to Java 1.6 if (TrigIt.getJavaVersion().ge(TrigIt.JAVA6)) { String expectedJDK16 = "<input type=\"text\" ..."; assertEquals(expectedJDK16, result); } else { String expectedJDK15 = "<input type=\"text\" ..."; String expectedJDK16 = "<input type=\"text\" ..."; if (result.contains("foo=\"bar\" ...")) assertEquals(expectedJDK15, result); else assertEquals(expectedJDK16, result); } ...}

  8. Design Choices • Preserves semantics of comments from the execution point • Each TrigIt specification is local to its context similar to inline comments • TrigIt specifications are removed from the compiled code (after TrigIt's processing), thus they do not cause side-effects during execution • Java compiler checks partial correctness of TrigIt specifications • Integrated as a bot that sends a notification to developers or a code review with changes when a trigger evaluates to true

  9. TrigIt DSL • Query Expression / Trigger • Action Statement / Global Action • Local Action • TrigIt Method

  10. TrigIt DSL (1/4) • Query Expression / Trigger • Examples • AST: TrigIt.getClasses().findAny(“C”).isPresent() • Build: TrigIt.getJavaVersion().ge(TrigIt.JAVA8) • Issue: TrigIt.isClosed(“github…issues/1897”) • Time: TirgIt.after(2019, 4, 2) if ( <query expression> ) { // action }

  11. TrigIt DSL (2/4) • Action Statement / Global Action • Examples • TrigIt.getMethod(m1()).remove() • TrigIt.getField(f).setPrivate() @TrigItMethod voidglobalAction() { if (/* trigger */) { <action statement> } }

  12. TrigIt DSL (3/4) • Local Action if (/* trigger */) { /* code executed if trigger evaluates to true */ } else { /* code executed if trigger evaluates to false */ }

  13. TrigIt DSL (4/4) • TrigIt Method • @TrigItMethod annotation • void return type: specifies trigger + global action(s) • boolean return type: specifies a trigger that can be reused @TrigItMethod voidtrigItDeprecateMakeChecked() { if (TrigIt.after(2018, 4, 30)) TrigIt.getMethodByName("immediateCheckedFuture").remove(); } @TrigtItMethod booleantrigItIsPublic() { returnTrigIt.getField(f).isPublic(); } if (trigItIsPublic()) { /* action 1 */ } if (trigItIsPublic()) { /* action 2 */ }

  14. TrigIt Workflow Test/Execution Compiler .class Files .java Files

  15. TrigIt Workflow TrigIt Compiler Test/Execution .class Files .java Files

  16. TrigIt Workflow TrigIt • Rewrite • removes all other code and only keeps TrigIt specifications Compiler Test/Execution rewrite TrigIt specifications .class Files .java Files

  17. TrigIt Workflow TrigIt • Rewrite • removes all other code and only keeps TrigIt specifications • Evaluate • checks all triggers Compiler Test/Execution evaluate rewrite TrigIt specifications .class Files .java Files AST Build Configurations Issue Tracker System Time

  18. TrigIt Workflow TrigIt • Rewrite • removes all other code and only keeps TrigIt specifications • Evaluate • checks all triggers • Apply • executes actions by changing the bytecode, sends warning messages, or changing source code • removes all TrigIt specifications Compiler Test/Execution evaluate rewrite TrigIt specifications .class Files .java Files apply AST Build Configurations Issue Tracker System Time Warning Messages Patch to .java Files

  19. Evaluation • RQ1: Can developers quickly implement correct triggers and actions using TrigIt? • RQ2: What’s the build overhead of TrigIt? • RQ3: What’s the overhead for writing TrigIt specifications over trigger-action comments, measured by number of tokens?

  20. User Study Design • Can developers quickly implement correct triggers and actions using TrigIt? • Tasks • Read a brief tutorial of TrigIt syntax (10min) • Migrate 3 trigger-action comments to TrigIt specifications • The first one is warm-up and not counted in results (without telling the participants) • 2 groups of tasks: • Warm-up, Task A from Google Guava, Task B from Apache Struts • Warm-up, Task C from Apache Ignite, Task D from Jenkinsci Jenkins • Details available on our webpage: cozy.ece.utexas.edu/trigit

  21. User Study Participants • 2 groups of tasks: • Warm-up, Task A from Google Guava, Task B from Apache Struts • Warm-up, Task C from Apache Ignite, Task D from Jenkinsci Jenkins • 20 participants • 8 professionals (6 developers in large software companies, 2 researchers) • 12 students (4 undergrads, 8 PhD students)

  22. User Study Results

  23. User Study Results Participants obtained correct solutions with very little training

  24. Build Overhead • RQ2: What’s the build overheadof TrigIt? • Subjects: We migrated 44 trigger-action comments from 10 projects to TrigIt specifications. • Environment: Intel i7-6700U@3.40GHz, 16GB RAM, Ubuntu 18.04 LTS • Results: average 8.85%, median 6.63%, min 1.91%, max 26.75%

  25. Complexity of the TrigIt DSL • RQ3: What’s the overhead for writing TrigIt specifications over trigger-action comments, measured bynumber of tokens? • Results: 58.74% • Trigger-action comment: average 12.2, median 12.0, min 6, max 26 • TrigIt specifications: average 18.0, median 16.5, min 11, max 36 • Trigger: average 7.3, median 6.0, min 4, max 19 • Action: average 7.4, median 5.0, min 2, max 27 • Boilerplate code: average 3.3, median 3.0, min 3, max 7 Boilerplatecode Trigger Action

  26. Anecdotal Experience • During our inspection of trigger-action comments in open source projects, we observed some comments are forgotten, i.e., the trigger is satisfied but the action is not executed • We reported 6 cases and the developers responded quickly • Some triggers and actions in comments are not specified enough: “Unfortunately Java 8 still causes some issues with some Google internal infrastructure. Java 8 is allowed in tests but not in the main part of the code (yet).” • Knowing the details of the projects, e.g., internal testing reports, will help writing correct TrigIt specifications

  27. Conclusion • A framework for writing trigger-action comments in executable format • Introduces DSL for triggers and actions • Preserves semantics and developers’ workflow • TrigIt is easy to learn and introduces small overhead • TrigIt can run as a bot that sends a notification to developers or a code review with changes when a trigger evaluates to true • Webpage:  cozy.ece.utexas.edu/trigit Pengyu Nie <pynie@utexas.edu>

  28. Backup slides after this point

  29. User Study Participants’ Prior Experience • Programming experience: average 8.4 years • Java skills (on a scale of 1-5): 3.7 • None of the participants has contributed to the open-source projects where we selected the tasks from

  30. User Study Error Analyze • Trigger error: 1 & action error: 1 • Misunderstanding the trigger-action comments • Syntax errors: 3 • Putting the local action in a wrong place • We added additional checks in TrigIt to prevent this kind of errors protectedAbstractStreamingHasher(intchunkSize, intbufferSize) {   // TODO(kevinb): check more preconditions   // (bufferSize >= chunkSize) if this is ever public checkArgument(bufferSize % chunkSize == 0); ...} @TrigtItMethod booleantrigItIsPublic() { if (TrigIt.getMethod("<init>", int.class, int.class).isPublic()) checkArgument(bufferSize >= chunkSize); }

  31. Example TrigIt Specifications (3) From Elastic Elasticsearch publicvoidtestShadowReplicaNaturalRelocation() throws Exception { … // TODO: uncomment the test below when https://github.com/elastic/elasticsearch/issues/17695 // is resolved. if (TrigIt.isClosed("https://github.com/elastic/elasticsearch/issues/17695")) assertIndicesDirsDeleted(nodes); }

  32. Example TrigIt Specifications (4) From Google Guava // TODO(b/72241893): Remove by 2018-04 @Deprecated publicstatic <V, X extends Exception> CheckedFuture<V, X> immediateCheckedFuture( @NullableDeclV value) { returnnewImmediateSuccessfulCheckedFuture<>(value); } @TrigItMethod voidtrigItDeprecateMakeChecked() { if (TrigIt.after(2018, 4, 30)) TrigIt.getCurrentClass().getMethodByName("immediateCheckedFuture").remove(); }

  33. Study of Trigger-Action Comments • We inspected 471 trigger-action comments and classify the trigger/action’s specificity • high: migration feasible • medium: migration could potentially be done • low: hard to understand, migration is not feasible • 28% have both specific trigger and specific action • 70% have either specific trigger or specific action

  34. TrigIt Implementation • Implemented as a standalone Java library, can be easily integrated into existing build process • Command line interface • Maven plugin • Implemented with Java bytecode manipulation, using ASM • Extra checks for the correctness of triggers and actions • TrigIt.getClass("C").getField("f").isPrivate()TrigIt will check if class C and field f exist • Debug options: --assume-true, --no-action

  35. Using Bytecode Analysis & Manipulation • The bytecode format was more stable in recent years than the Java language syntax; this may lower the cost of maintenance. • We wanted to avoid dependencies on IDEs and enable integration of TrigIt with any build process, even if the build is running on a continuous integration service. Future versions of TrigIt may have a tighter connection with IDEs based on the adoption and feedback that we received from the community.

  36. Example Patch (Before Trigger Satisfies) From Google Guava protectedAbstractStreamingHasher(intchunkSize, intbufferSize) { if (trigItIsPublic()) checkArgument(bufferSize >= chunkSize); checkArgument(bufferSize % chunkSize == 0); ...} @TrigtItMethod booleantrigItIsPublic() { returnTrigIt.getMethod("<init>", int.class, int.class).isPublic(); }

  37. Example Patch (The Patch) From Google Guava protectedAbstractStreamingHasher(intchunkSize, intbufferSize) { - if (trigItIsPublic()) - checkArgument(bufferSize >= chunkSize); + checkArgument(bufferSize >= chunkSize); checkArgument(bufferSize % chunkSize == 0); ...} -@TrigtItMethod -booleantrigItIsPublic() { - returnTrigIt.getMethod("<init>", int.class, int.class).isPublic(); -}

  38. Related Work • Pengyu Nie, Junyi Jessy Li, Sarfraz Khurshid, Raymond Mooney, and Milos Gligoric. 2018. Natural language processing and program analysis for supporting todo comments as software evolves. In Workshops of the the AAAI Conference on Artificial Intelligence. 775–778. • Margaret-Anne Storey, Jody Ryall, R. Ian Bull, Del Myers, and Janice Singer. 2008. TODO or to bug. In International Conference on Software Engineering. 251–260. • GiriprasadSridhara. 2016. Automatically detecting the up-to-date status of ToDo comments in Java programs. In India Software Engineering Conference. 16–25. • DorsafHaouari, HouariSahraoui, and Philippe Langlais. 2011. How good is your comment? A study of comments in Java programs. In International Symposium on Empirical Software Engineering and Measurement. 137–146. • Lin Tan, Ding Yuan, Gopal Krishna, and Yuanyuan Zhou. 2007. /*iComment: bugs or bad comments?*/. In Symposium on Operating Systems Principles. 145–158. • Shin Hwei Tan, Darko Marinov, Lin Tan, and Gary T. Leavens. 2012. @tComment: Testing Javadoc comments to detect comment-code inconsistencies. In International Conference on Software Testing, Verification, and Validation. 260–269.

More Related