1 / 77

Exploiting Context in Dealing with Programming Errors and Exceptions

Exploiting Context in Dealing with Programming Errors and Exceptions. Mohammad Masudur Rahman Department of Computer Science University of Saskatchewan. Exception triggering point. Exception: a common experience!!. Exception Debugging. Web search!!.

uma
Download Presentation

Exploiting Context in Dealing with Programming Errors and Exceptions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Exploiting Context in Dealing with Programming Errors and Exceptions Mohammad Masudur Rahman Department of Computer Science University of Saskatchewan

  2. Exception triggering point Software Research Lab, U of S Exception: a common experience!!

  3. Exception Debugging Software Research Lab, U of S Web search!! Not a helpful message for understanding or solving the exception

  4. Solving Exception (Step I: Web Search) Switching is often distracting Software Research Lab, U of S • The browser does not know the context (i.e., details) of the exception. • Not much helpful ranking • Hundreds of search results • Forces the developer to switch back and forth between IDE and browser. • Trial and error in searching

  5. Solving Exception (Step II: Query Selection) Selection of traditional search query Software Research Lab, U of S This query may not be sufficient enough for most of the exceptions Switching to web browser for web search

  6. Solving Exception (Step III: Post-Search Analysis) 4 2 • Only the most relevant section needs to be checked to determine the relevance of a page. • Frequent switching between IDE and web browser for content analysis • Manual analysis of a number of web pages is non-trivial and time-consuming 3 1

  7. Solving Exception (Step IV: Handling Exceptions) • Only adds a generic handler (i.e., printStackTrace()) for the exception. • Not much helpful for effective handling of the exception

  8. Observations on Traditional/Ad-hoc Approach for Exception Solving • Step I: Web searchmay not be much effective or reliable • Step II: Support for search query formulationis not enough • Step III: Support for post-search analysisis insufficient • Step IV: Support for exception handlingis not enough Software Research Lab, U of S

  9. Dealing with Errors & Exceptions:Proposed Approach

  10. Contributions of this Thesis (1) SurfClipse (WCRE 2013, CSMR/WCRE 2014) Web search (2) QueryClipse (ICSME 2014) Query formulation (5) ExcClipse (3) ContentSuggest Post-search analysis User study (4) SurfExample (SCAM 2014) Exception handling

  11. Technical Details & Programming Context of an Exception 3 Programming context (i.e., context code) 1 Error message Software Research Lab, U of S Stack trace 2

  12. SurfClipse: IDE-Based Context-Aware Meta Search Engine

  13. Motivational Experiment • 75 programming exceptions (details later) • Individual engine can provide solutions for at most 58 exceptions & each has some unique results. • Combination of content & context is always better than content only Software Research Lab, U of S

  14. The Key Idea !! Meta Search Engine Software Research Lab, U of S Fig: Meta search engine

  15. Proposed IDE-Based Meta Search Model Software Research Lab, U of S Web page Start Results

  16. Proposed IDE-Based Meta Search Model • Distinguished Features (5) • IDE-Based solution • Web search, search result and web browsing all from IDE • No context-switching needed • Meta search engine • Captures data from multiple search engines • Also applies custom ranking techniques • Context-Aware search • Uses stack traces information • Uses context-code (surroundings of exception locations) • Software As A Service (SAAS) • Search is provided as a web service, and can be leveraged by an IDE. http://srlabg53-2.usask.ca/wssurfclipse/ Software Research Lab, U of S

  17. Proposed IDE-Based Meta Search Model • Two Working Modes • Proactive Mode • Auto-detects the occurrence of an exception • Initiates search for exception by client itself • Aligned withCordeiro et al. (RSSE’ 2012) &Ponzanelli et al. (ICSE 2013) • Interactive Mode • Developer starts search using context menu • Also facilitates keyword-based search • Aligned with traditional web search within the IDE Software Research Lab, U of S

  18. Proposed Metrics & Scores • Content Matching Score (Scms) • Cosine similarity based measurement • Stack trace Matching Score (Sstm) • Structural and lexical similarity measurement of stack traces • Code context Matching Score (Sccx) • Code snippet similarity (code clones) • StackOverflow Vote Score (Sso) • Total votes for all posts in the SO result link Software Research Lab, U of S

  19. Proposed Metrics & Scores • Site Traffic Rank Score (Sstr)-- Alexa and Compete Rank of each link • Search Engine weight (Ssew)---Relative reliability or importance of each search engine. Experiments with 75 programming queries against the search engines. • Heuristic weights of the metrics are determined through controlled experiments. Software Research Lab, U of S

  20. Experiment Overview Eclipse plug-in & Java development 75 exceptions Software Research Lab, U of S Gold set solutions Peers

  21. Results on Different Ranking Aspects Software Research Lab, U of S [ MP = Mean Precision, R = Recall, TEF= Total Exceptions Fixed]

  22. Comparison with Existing Approaches Software Research Lab, U of S [ MP = Mean Precision, R = Recall, TEF= Total Exceptions Fixed]

  23. Comparison with Traditional Search Engines Software Research Lab, U of S

  24. Contributions of this Thesis (1) SurfClipse (WCRE 2013, CSMR/WCRE 2014) Web search (2) QueryClipse (ICSME 2014) Query formulation (5) ExcClipse (3) ContentSuggest Post-search analysis User study (4) SurfExample (SCAM 2014) Exception handling

  25. QueryClipse: Context-Aware Search Query Recommender

  26. Motivating Example 1 2 Context code Stack trace QueryClipse Software Research Lab, U of S Recommended search queries

  27. Proposed Context-Aware Query Recommendation Approach • Distinguishing Features (4) • Context-Aware Query • Exploits both stack trace and context code • Extract search keywords carefully and systematically • Ranked List for Queries • Ranked list based on keyword importance in queries • Automatic suggestion through auto-completion • Custom Search Query • Stack trace graph based on implied relationships • Keyword importance based on network connectivity • Query Length Customization • Number of query keywords customizable • Search-friendly & easily applicable for any search engines Software Research Lab, U of S

  28. ContentSuggest: Context-Aware Page Content Recommender

  29. Motivating Example 1 3 2 Software Research Lab, U of S • Only the most relevant page section displayed • Less information overhead, less effort required • No need to browse the page for relevance checking

  30. Proposed Context-Aware Page Content Suggestion Approach • Distinguishing Features (3) • Relevant section(s) suggestion • Analyzes both quality and relevance of the content • Exploits stack trace & context code for relevance checking • Partial automation in post-search analysis • Less content, less overhead • Need to analyze less content for page relevance checking • Displayed content more useful than meta description • Noise-free version of the page • Removes advertisements, irrelevant widgets and so on • Applies link-based heuristics • Returns a noise-free version of the web page Software Research Lab, U of S

  31. Contributions of this Thesis (1) SurfClipse (WCRE 2013, CSMR/WCRE 2014) Web search (2) QueryClipse (ICSME 2014) Query formulation (5) ExcClipse (3) ContentSuggest Post-search analysis User study (4) SurfExample (SCAM 2014) Exception handling

  32. SurfExample: Context-Aware Code Example Recommender for Exception Handling

  33. Motivating Example Context code Software Research Lab, U of S SurfExample Recommended code example

  34. Proposed Context-Aware Code Example Recommender for Exception Handling • Distinguishing Features (3) • Graph-based structural relevance • Static relationship and data dependency graph • Graph structure matching • Handler quality Paradigm • Novel idea to ensure quality of exception handlers • Based on readability, amount & quality of the handler actions • Seamless integration of dataset • Exploits GitHub API for data collection • Hundreds of popular and mature open source projects from Eclipse, Apache and others Software Research Lab, U of S

  35. Contributions of this Thesis (1) SurfClipse (WCRE 2013, CSMR/WCRE 2014) Web search (2) QueryClipse (ICSME 2014) Query formulation (5) ExcClipse (3) ContentSuggest Post-search analysis User study (4) SurfExample (SCAM 2014) Exception handling

  36. ExcClipse: Comparative Analysis between Proposed Approaches and Traditional ones

  37. User Study Overview Four exceptions Six participants Software Research Lab, U of S Four tasks

  38. User Study Overview Questionnaire Observation checklist Software Research Lab, U of S Evaluation Training Execution

  39. Evaluation Features Software Research Lab, U of S

  40. Feature-wise Ratings (Web Search) Software Research Lab, U of S

  41. Feature-wise Ratings (Code Search) Software Research Lab, U of S

  42. Feature-wise Ratings (Non-Functional) Software Research Lab, U of S

  43. Overall Ratings Software Research Lab, U of S

  44. Threats to Validity • SurfClipse--Search engines constantly evolving, same results may not be produced at later time. • QueryClipse-- Long query generation due to lengthy error message in the stack trace. • ContentSuggest-- Changes in the look and feel of the page due to removal of <style> and <script> tags • SurfExample-- Subjective bias in gold set development • ExcClipse--Limited number of participants Software Research Lab, U of S

  45. Conclusion

  46. Concluding Remarks • SurfClipse – context-aware meta search engine • IDE-based complete web search solution • Outperforms two relevant existing approaches • More recall than three search engines with precision comparable to Google, the best performing engine. • QueryClipse– context-aware query recommender • More effective than traditional queries and queries by existing approaches • Highly applicable in terms of pyramid score Software Research Lab, U of S

  47. Concluding Remarks • ContentSuggest– context-aware page content recommender • Exploits exception details and recommends relevant section(s) from the page • Less information, less overhead for relevance check • Great potential for problem solving • SurfExample– code example recommender for exception handling • Graph-based structural relevance matching • Handler Quality paradigm • Outperforms four existing approaches in all metrics Software Research Lab, U of S

  48. Future Work • SurfClipse – DOM-based element extraction & topic modeling • QueryClipse– Semantic and customized query recommendation • ContentSuggest– More focused content recommendation (e.g., paragraph of interest) for problem solving • SurfExample– More directed support (e.g., applicability of an example) for exception handling Software Research Lab, U of S

  49. Thank You!! Software Research Lab, U of S

  50. References [1] J. Cordeiro, B. Antunes, and P. Gomes. Context-based Recommendation to Support Problem Solving in Software Development. In Proc. RSSE, pages 85 –89, June 2012. [2] L. Ponzanelli, A. Bacchelli, and M. Lanza. Seahawk: Stack Overflow in the IDE. In Proc. ICSE, pages 1295–1298, 2013 [3] J. Brandt, P. J. Guo, J. Lewenstein, M. Dontcheva, and S. R. Klemmer. Two Studies of Opportunistic Programming: Interleaving Web Foraging, Learning, and Writing Code. In Proc. SIGCHI, pages 1589–1598, 2009. [4] F. Sun, D. Song, and L. Liao. DOM Based Content Extraction via Text Density. In Proc. SIGIR, pages 245–254, 2011. [5] T. Gottron. Content Code Blurring: A New Approach to Content Extraction. In Proc. DEXA, pages 29–33, 2008. [6] S. Bajracharya, J. Ossher, and C. Lopes. Sourcerer: An Internet-Scale Software Repository. In Proc. SUITE, pages 1–4, 2009 [7] E. A. Barbosa, A. Garcia, and M. Mezini. Heuristic Strategies for Recommendation of Exception Handling Code. In Proc. SBES, pages 171–180, 2012 [8] R. Holmes and G. C. Murphy. Using Structural Context to Recommend Source Code Examples. In Proc. ICSE, pages 117–125, 2005 [9] W. Takuya and H. Masuhara. A Spontaneous Code Recommendation Tool Based on Associative Search. In Proc. SUITE, pages 17–20, 2011. [10] M. M. Rahman, S. Yeasmin, and C. K. Roy. Towards a Context-Aware IDEBased Meta Search Engine for Recommendation about Programming Errors and Exceptions. In Proc. CSMR-WCRE, pages 194–203, 2014 [11] M. M. Rahman and C.K. Roy. On the Use of Context in Recommending Exception Handling Code Examples. In Proc. SCAM, 10 pp., 2014 (to appear) [12] M. M. Rahman and C.K. Roy. SurfClipse: Context-Aware Meta Search in the IDE. In Proc. ICSME, 4 pp., 2014 (to appear) [13] Slide 13, Meta Search Engine, http://en.wikipedia.org/wiki/Metasearch_engine Software Research Lab, U of S

More Related