Onboarding, Finding experts, maintaining awareness

1. Onboarding, Finding experts, maintaining awareness Kun Niu Spring, 2011 Feb. 22nd 05-899D Human Aspects of Software Development (HASD) The Information Networking Institute is a cooperative endeavor of: College of Engineering School of Computer Science Tepper School of Business Heinz College

2. Agenda • Overview • Paper discussion • Onboarding • Finding experts • Maintaining awareness • Summary

3. Agenda • Overview

4. Concepts • Onboarding - The process of helping new employees become productive members of an organization • Finding expert - The process of finding the person who is expert in a specific part of the software project • Awareness - An understanding of the activities of others to provide a context for one’s own activities

5. Why do people do research in these fields? • Find different facts hindering newcomers in joining the project efficiently – Onboarding • Find experts in the software development group and make best use of them – Find experts • Keep track of what software developers did in order to know what the developers have done to the code and why they did that – Maintaining awareness • All in all, make software development team more productive.

6. Sample target for studies • Students in the university who are ready to work in a software intensive company • Developers who work in a collaborative environment (open source software or commercial software or academic research, etc.)

7. Agenda • Overview • Paper discussion

8. Agenda • Overview • Paper discussion • Onboarding

9. Struggle of New College Graduates in their First Software Development Job • Andrew Begel, 2008 • Discover what occurs during the beginning of the transition period from college graduate to experienced software engineer • Spent 85 hours observing 8 new software developers (NSDs) in their first six months of employment at Microsoft Corporation • Result • Five ways in which recent graduates struggle to be effective: communication, collaboration, technical, cognition, and orientation

10. Moving into a New Software Project Landscape [Barthélémy, 2010] • Grounded Theory • A qualitative research approach that employs theoretical sampling and open coding to formulate a social theory “grounded” in the empirical data (exploratory study) • Three instruments for research • Initial Survey (of people in IBM Quality of Software Engineering) • Project Sketch (Asked them to describe their projects) • Phone interview (One-hour semi-structured) • Participants • 3 women and 15 men, experienced programmers and novices

11. Landscape features related to orientation aids and obstacles

12. Comparison Between Papers • Similarities • Coupled with orientation and collaboration • Struggle in using document to gather information • Difficulty in working in a corporate environment • Differences • Field study vs. multiple stages • Novice vs. Novice mixed with Experienced programmers • Where Novices get stuck • Without feedback vs. with feedback • Microsoft vs. IBM

13. Agenda • Overview • Paper discussion • Onboarding • Finding experts

14. A Degree-of-Knowledge Model to Capture Source Code [Thomas Fritz, 2010] • The degree-of-knowledge (DOK) Model • The DOK values are computed for a developer automatically by combining authorship data from the source revision system and interaction data from monitoring the developer's activity in the development environment. • Goals • Investigate whether DOK values can support finding who is an expert in particular parts of a code. • Investigate whether DOK values can help familiarize (onboard) a new team member onto a particular part of the development project.

15. Exploratory Case Studies • Site 1 involves seven professional developers building a Java client/server system and using IBM Rational Concert as the source repository • Site 2 involves 2 developers building open source frameworks for Eclipse as part of their work and using CVS as source repositories

16. Factors that might correlate with expertise • Degree-of-Authorship (DOA) • Determined by three factors: first authorship (FA), the number of deliveries (DL) and the number of acceptances (AC) • Degree-of-Interaction (DOI) • The amount of interaction — selections and edits — a developer has had with a source code element • Degree-of-Knowledge (DOK) • Formula linearly combines the factors contributing to DOA and DOI: • DOK = α FA * FA+ αDL * DL+ α AC * AC + βDOI * DOI

17. Analysis and Results • Formula factor computation • DOK = 3.293 + 1.098 * FA + 0.164 * DL – 0.321 * ln(1 + AC) + 0.19 * ln(1 + DOI) • Negative values of DOI indicate usage that is not recent

18. Case study results: Finding experts • Method • Two projects from Site 1 • Most members of the team interacted with these two projects • Calculate DOK for class-developer pair • Calculate DOK for package-developer pair • Results • All six developers stated that the result was reasonable, using phrases like it is “close” and it reflects [reality] correctly.

19. Case study results: Onboarding • Method • Three randomly-chosen developer • Interview and survey • Results • Elements with high DOK values are not considered useful • Elements with high DOK may be a good starting point to find useful elements

20. Expert Recommendation with Usage Expertise [David, 2009] • Assess the viability of usage expertise concept • Evaluate expert recommendations for the ASPECTJ and ECLIPSE projects • Collect data according to usage context, depth, breadth….

21. Experiment

22. Results • Context improves accuracy of usage expertise • This means only a handful of developers contribute the bulk of contributions for most methods. • On Recommending Across Projects • There must be sufficient overlap in the calls to external libraries. By extension this means that recommendations are at the very least, possible.

23. Expertise Browser: A Quantitative Approach to Identifying Expertise [Audris, 2002] • Application range • Locate experts in a particular technology or tool • Locate experts in a particular part of product • Interviews with members of distributed project teams • Calculation using EA (“experience atom”) • Elementary units of experience • The simplest unit of experience that could be observed in projects using change management systems is the atomic change (delta) made to source code or to documentation.

24. Learning curve

25. Expertise Browser (Java Applet)

26. Analysis of Expertise Finding • All of the papers try to quantify expertise analysis • Are the selected data really helpful? • Does change of code really reflect the expertise in the code? • Does it apply to other parts of the software development group? (product design, QA, customer service…)

27. Agenda • Overview • Paper discussion • Onboarding • Finding experts • Maintaining awareness

28. Hipikat: A Project Memory for Software Development [Davor, 2005] • Provide developers efficient and effective access to the project memory • Focus on open-source software projects, a subset of virtual teams that typically have extensive public archives of all artifacts relevant to the project • The tool forms a project memory from artifacts that were created during a software development project (source code, documentation, and electronic communication records) • Hipikat recommends to a developer artifacts selected from the project memory that it considers to be relevant to the task being performed

29. HipiKat Server Architecture

30. Eclipse newcomer case study • Goals • If project memory about past modifications helpful to the developers • When and from which artifacts are queried • Evaluate the recommendation result • Sample • Have adequate programming experience in the programming language of the system under study • Have no previous experience as developers on the system itself

31. Eclipse newcomer case study (cont) • Design • Easy task – Display break points property • Difficult task – Interactive modification during version operations on a group of files • Procedure • Training • Programming

33. Results • Less access during the easy task than the difficult task • Focusing on solving task rather than understanding code • Using recommendations to find the code could be reused

34. FASTDash: A Visual Dashboard for Fostering Awareness in Software Teams [Jacob, 2007] • Use electronic tools to operate in a shared virtual environment • Hard to observe activities • Lack adequate tools to support sufficient ongoing awareness • Hard to determine who is working on which file

35. Fastdash Interface

36. Field Study • Observational study of a software team’s collaborative behavior before and after the introduction of FASTDash • Leverage the earlier work to develop a new coding that was a synthesis of the most relevant parts of existing schemes and basic requirements • Coding categories and classificatoin

37. Observation Log

38. Result Communication category distribution for pre- and post- conditions Use of shared artifacts

39. Group Awareness in Distributed Software System [Carl, 2004] • A study of open source teams to determine whether developers need to stay aware of one another • Interviewed fourteen developers on three different well-established OSS projects, examined email and chat archives, and analyzed project artifacts such as source-code repositories, web pages, and official project documentation • Expected that projects would be set up to reduce awareness requirements, with each software module carefully partitioned and protected from others. • Found that official partitioning is limited, and that developers can contribute to any part of the code – an organizational approach that increases awareness requirements • Found that the developers were able to maintain a good general awareness of other developers and their activities, and were able to find more detailed information about people’s activities when they needed to • The main mechanisms for maintaining group awareness were simple text communication tools – developer mailing lists and text chat

40. Situational Awareness Support to Enhance Teamwork in Collaborative Environments • [Olga, 2008] • Situational Awareness (SA) definition • Previous knowledge and understanding of the situation • Detection of relevant information from the environment • Interpretation of these information and reconfiguration • Research Approach (Exploratory study) • Included contextual observations, interviews and task analysis, have been translated into requirements for support of multidisciplinary teamwork in life sciences • Conclusion • Designing systems that support situational awareness is of great importance to ensure that a collaborative environment enables efficient and effective team coordination and decision making.

41. A Comprehensive Evaluation of Workspace Awareness in Software Configuration Management Systems [Anita, 2007] • Workspace tool – Palantir • Tool used to monitor code, which is usually manually merged, change • Method – Experiment • Eclipse & CVS vs. Eclipse & CVS & Palantir • The same as above plus UML diagram • Result • Subjects clearly monitor awareness cues • Filters are useful

42. Palantir

43. Conflict Detection and Resolution for Direct and Indirect Conflicts for Control and Experimental GroupsTime to completion

44. Indirect conflicts detection

45. Empirical Evidence of the Benefits of Workspace Awareness in Software Configuration Management [Anita, 2008] • Method – Experiments ( two-stage) • Edit text file experiment • Edit Java code experiment

49. Results • The tool is helpful in many ways • Threats • How to generalize the result • How to make the experiment closer to real development

50. Comparison of Papers:Awareness & Collaboration • Awareness improves collaboration • Collaboration needs communication interface (Partial awareness?)