Hipikat: A Project Memory for Software Development

1. Hipikat: A Project Memory for Software Development The CISC 864 Analysis By Lionel Marks

2. What’s the Typical Scenario for a New Developer? • A new developer joins the team and knows little of the project • An existing member offers advice and helps newcomer finish first tasks • Information generally about basic ideas on the problem domain and pointers on using tools effectively • Mentors give information on the eccentricities of the system and why things are done that way

3. What is the New Scenario? • Team members are not co-located • Hard for a newcomer to get up-to-speed on the project on the “virtual team” • Using electronic media such as mailing lists, code repositories and work tracking programs, can create a “project memory” with Hipikat

4. What is Hipikat Good for? • With the volume of information being so enormous, sifting through the documents would take too long • Hipikat allows the user to search these archives • Searching CVS for code to clone for your new method • Searching configuration management tools for a history of changes, to help in setup maintenance or to redo a setup.

5. Related Work • Group Memory • Design Assistant and TeamInfo had the information generated by humans specifically for their tool to query • Recommender Systems • Remembrance Agent and Code Broker did not use multiple information sources

6. Related Work • Programming from Examples • Reuse View Matcher and Explainer which has libraries of specifically created code to query, not existing code from the current system. • Mining Software Repositories to help in SW Maintenance • CVSSearch – search comments from CVS commits • Expertise Browser and Expertise Recommender – recommend developers with expertise at certain parts of a project • Zimmerman et al., Ying et al. – found change patterns in repositories to recommend potentially relevant files when working on a change in the code of a project

7. How Hipikat Works • Hipikat forms the project memory from source code, documentation, and other electronic media such as e-mail messages and bug reports. All are put in a central database that can be queried. • It can then be queried to recommend artifacts that are relevant to the user’s current task. • Information sources can be monitored periodically or continuously, and when they are committed to the database, then they can be included in searches

8. How Hipikat Works • “A user makes a query by selecting an artifact in the Eclipse project workspace and choosing ‘Query Hipikat’ from the context menu.” (Interesting!) • User can also search the Hipikat database through “Search Hipikat”

9. The Digg Effect • Training the System (Increasing the rank of certain artifacts in queries) • Just by clicking on the artifact (that it looks useful) • Clicking a “thumbs-up” or “thumbs-down” • Problems (Can anyone see them?)

10. The Digg Effect • Training the System (Increasing the rank of certain artifacts in queries) • Just by clicking on the artifact (that it looks useful) • Clicking a “thumbs-up” or “thumbs-down” • Problems (Can anyone see them?) • Clicking on artifact  What if relevant on first sight, but not really upon inspection? Inc. rank of bad match • Clicking on “thumb”  Asking to verify usefulness before even used the data. Can’t rank like this

11. Evaluation Criteria • Precision vs. Recall • In a perfect world, whatever the system recommends is exactly what you need • Giving recommendations that have “examples of use” for relevant APIs rather than just giving the place in code where you must make your changes (i.e. “The solution”) (Interesting!)

12. Eclipse Newcomer Study • Finding Participants • Require adequate programming experience in Java • Have experience developing large or medium-sized SW • Very small pool of recruit-able computer science undergrads with the above criteria (Good point!) • Also wanted to compare the solutions • Got some of experienced members of the Eclipse Development team • Asked them to perform the same tasks for baseline comparison

13. Eclipse Newcomer Study Results • The first task, implementing the pop-up window when hover over the side of code to suggesting breakpoint properties • Experts performed worst on handling the special cases • 25% of experts got them • 38% of newcomers got them perfectly right • 63% of newcomers got it basically right • Harder task • 75% of experts solved it correctly • 38% of newcomers got it right

14. Use of Hipikat • More queries made on the harder task • Almost all queries made within the first hour • Generally once the person knew what file(s) to change, and had a general plan, no more queries were made. • Criteria for finding a recommendation useful • Is the problem report “interesting” • Similar enough to current task – code re-use • Learn relevant information • For difficult coding problems, searched and found filenames that look potentially relevant • Did not look at source code changes, but rather built their understanding from “scratch” looking at the file in an editor. (easier, did not want a “false lead” and have to continue searching)

15. My General Thoughts • Good paper in general • The paper did a good job at stating its facts and shortcomings. • Would like to see it used in a different way, as a data miner for teaching the person about the system rather than providing assistance when in doubt (too much recall for when trying to perform a single task)

16. Likes and Dislikes of this Paper • Likes • Its study of how people thought about the problem • The tool itself is an intriguing idea • The style of the paper – very readable! • Dislikes • Asking that people take into account special cases when first coding something in a system – a bit of a reach • Saying that the participants must have worked on a medium to large SW system before. Many undergrads will go into industry working on their first large system ever. Would be interesting for your tool to get someone who is really new to these kinds of SW systems – see if they use Hipikat more