Challenges in Software Engineering for Collaborative Development

1. Challenges in Software Engineering for Collaborative Development Thomas Marlowe (Seton Hall) VassilkaKirova (Alcatel-Lucent) MojganMohtashami (Advanced Infrastructure Design)

2. Outline • Why cooperative development? • Motivation, properties, forms • A long list of challenges – many open questions • Process • Information • System organization • Technology • Management and legal • A short list of mitigations • Endless opportunities for innovation and research Seton Hall University

3. Software Engineering Disciplines • Requirements Elicitation • Specification and Analysis • Design • Implementation • Testing and Validation • Maintenance and Evolution • Process Evolution Seton Hall University

4. Traditional Software Engineering • Disciplines are phases • Limited feedback • “Requirements complete at start of project” • “Testing can wait until after implementation” Seton Hall University

5. Agile OO Software Engineering • Disciplines are work areas • Disciplines exercised concurrently during development • Obviously, more requirements at start; more evolution at end; and so on • Testing must start early • Requirements never complete or completely understood but change • Development must follow changes • Evolution is critical Seton Hall University

6. Some Prior Findings • Iterative, continuous communication improves collaboration and decision making • Large, complex, long-lived, or high uncertainty projects benefit from dynamic management and good quality communication • Emphasis on information and knowledge exchange is the key to collaborative success • Traditional software development practices negatively affect collaboration Mojgan Mohtashami - AID, INC. IRMA May 22

7. THE challenge Seton Hall University

8. Software Industry Challenges Market Expectations Up Costs Up Intervals Shrinking Complexity Up Revenues Down Stockholders : ROI Customers: Value Users : Functionality & Quality Need solutions that are flexible, expansible, survivable, sustainable, affordable and secure  Seton Hall University

9. Four Styles of Development • Traditional • Single organization, few teams, coordinated • Open Source • High parallelism, low coordination & control • Contractual • Vendors, subcontractors, services, … • Collaborative • Multiple organizations, negotiated partition Seton Hall University

10. MotivationSoftware Development is Risky • 75% of standalone software development projects fail • Full collaboration increases resources • But also increases complexity and risk • Without attention to collaborative process, failure rate is almost certainly higher Seton Hall University

11. Models • Traditional, single-organizational • Dominant style for small and medium projects • Problems for large and complex projects • Expertise, staffing, resources, economy, … Seton Hall University

12. Models • Open source • Important source for low-security projects, components and tools • Management and intellectual property issues • Contractual model • Uses expertise, resources, … of others • Limited flexibility after early decisions • Fits traditional software engineering model Seton Hall University

13. Models • Collaborative model • Highly flexible, both for resources & design • Better fit to modern software engineering • Distributed, collaborative risk management • Success in restricted domains/environments • Lots of open questions Seton Hall University

14. What to do? • Iterative incremental development • Management contingency policies • Distributed development • Across teams, sites, and organizations • Distributed risk management • Collaborative development • International collaboration Seton Hall University

15. Cooperative-Collaborative Development • Development • Large and complex systems • Ultra-large Scale Systems (DoD, SEI) • Requiring multiple specialties • Mix of skills, knowledge and expertise • Multiple communities of experts • Specialized components, tools & libraries • Diversity of development models and participating organizations • Software production as a socio-technical ecosystem Seton Hall University

16. Cooperative Development • Cooperative development • Multiple teams working toward a common product • Often at multiple sites • Frequently in different organizations • Domain and application experts and specialists—and reusable code base—also widely distributed Seton Hall University

17. International Collaboration • New Benefits • Multiple perspectives and skill sets—both managerial & technical • Time zones extend collaborative work day • New Challenges • Conflicts in perspectives and skill sets • Time differences inhibit communication • Language and cultural differences matter Seton Hall University

18. Collaborative DevelopmentOur Goal • Provide a documented, standardized structure • A foundation for collaborative process definition & alignment • A bridge to management buy-in • A basis for SQA for collaborative systems • A plan for ongoing involvement • A checklist for possible risks and hazards Seton Hall University

19. Modes of Collaborative Software Development

20. Cooperative Development • Inter-organizational Collaborative Software Development (ICSD) • Often international • Management-inspired — outsourcing/offshoring • Technically inspired — expertise matching • Less collaboration  less satisfactory • Important business, process, and project characteristics Seton Hall University

21. ICSD • Business Characteristics • Long or short-term alliances • Ventures with loosely defined relationships with parent company • Distributed decentralized development and management model Seton Hall University

22. ICSD • Process Characteristics • Full-fledged integrated development, with dynamic requirements and flexible design • Flexible component development • Development of modifiable, configurable tools and libraries Seton Hall University

23. ICSD • Project Characteristics • Long-lived projects or high innovation • Components with specific roles, but significant dependences and interaction • Evolvable specification, not completely fixed Seton Hall University

24. Requirements for Collaborative Software Development

25. Overview • ICSD requires • High bandwidth and high diversity communication • Ongoing coordination • Emphasis on information and communication • Collaboration-aware risk management • Collaboration-aware management policies Seton Hall University

26. Examining the Challengesand Requirements • Processes and methods • Technology and infrastructure • Risk Management • Corporate Management • Customer Interaction • Legal • Metrics • Reflection—process flexibility & evolution Seton Hall University

27. Process Challenges • Need to develop a modified process for software development • Start with less common knowledge • Require more information sharing • Constrained by lower information bandwidth • Both data volume and conceptual communication • Face-to-face or even virtual face-to-face interaction necessarily more limited • Constrained by cultural differences Seton Hall University

28. Process ChallengesInformation • Sharing information • Relax need-to-know, information hiding relative to requirements • Share high-level process and design views • Provide support for design decisions • Interacts with management and legal concerns • Be sensitive to cultural differences—what is the role of personal information? Seton Hall University

29. Process ChallengesSystem Organization & Design • Architectures • Adaptable, open architectures, aspects, services • Component structure • Component boundaries and interaction mechanisms • Identify integration and end-to-end constraints • Support flexible interfaces • Levels of abstraction • Use higher-levels of abstraction in system definition and design (e.g., model-driven development, domain specific modeling languages) Seton Hall University

30. Technology Challenges • Integrated (possibly virtual) development environment or views • Coordinated tool support and development • Common tools where appropriate • Identify and extend existing tools and techniques • “Not invented here” is also cultural Seton Hall University

31. Technology ChallengesCommunication • Both synchronous and asynchronous channels • Improved multi-media communication • Live Meeting support including (e.g. LiveMeeting) • Collaborative Environments • Co-authoring (e.g., ColabNet) Seton Hall University

32. Technology ChallengesKnowledge base • Support for lore—a viable group memory • Shared project repositories • Documents, code, specification and design artifacts, … • Multi-layer, distributed risk management archive • With language differences • Establish governing versions of text documents • Create usable translations where feasible Seton Hall University

33. Special Technical Challenges • Handling innovation • Security • Real-time applications • Active applications with multiple active components • Process control, voting, robotics, … • Extensive multi-lingual interfaces • Configurability over multiple components Seton Hall University

34. Risk Management • Uncertainty + loss • Steps in risk management • Identify and analyze • Plan how to recognize, handle and recover • Try to avoid or prevent • Mitigate occurrences • Recover software process • Need communication and reflection Seton Hall University

35. Risk Management ChallengesNew and Modified Risk Factors • Communication • Increased bandwidth increases vulnerability • New modes and artifacts require infrastructure and understanding • Linguistic and cultural differences complicate specification and documentation Seton Hall University

36. Risk Management ChallengesNew and Modified Risk Factors • Cultural differences • Not just social and language • Institutional, site, and team culture • Work norms and at-work interactions • Interactions with outside contacts Seton Hall University

37. Risk Management ChallengesNew and Modified Risk Factors • Differences in processes and practices • Incompatible views of design, code, and data • Leads to difficulties in integration • Leads to misunderstandings • Impact of process and practices on mental map, and conversely • Sapir-Whorf — also language Seton Hall University

38. Risk Management ChallengesNew and Modified Risk Factors • Collaborative impact of problem • Increased risk (of course) at component interfaces • Risk confinement and contagion • Scope limited to single component or organization • Effects primarily on interfaces or agreements • Many components or partners affected Seton Hall University

39. Risk Management ChallengesNew and Modified Risk Factors • Collaborative impact of problem • Trust—Problems can affect ongoing relationships • Current project • Future collaboration • Reputation and ability to form new collaborations • Language and cultural differences exacerbate • May be redressable or recoverable Seton Hall University

40. Risk Management ChallengesWhy is this a new challenge? • Cultural differences • Lots of work on social and management practices • For example, American vs Japanese customs • Little attention to differences in technical or technical management worldviews • Little attention to differences in work norms or importance of work for technical staff Seton Hall University

41. Risk Management ChallengesNew Criteria for Effective Collaboration • Level of trust established • Cultural sensitivity • Collaborative communication • Effective interaction • Focus on quality of relationships • Technical support for communication Seton Hall University

42. Risk Management ChallengesNew Criteria for Effective Collaboration • Aligned management support • Responsibility for risk management • Responsibility for customer interaction • Responsibility for maintenance • Process for product evolution • Process for process evolution, including updating RMMM plan Seton Hall University

43. Risk Management ChallengesSome specific issues • Glossary must account for regional or cultural differences in definitions of words • At least three facets of communication • Culture-neutral or culture-exempt technical communication • Culture-sensitive personal communication at both management and technical levels • Culture-aware documentation—both for development and for customers + users Seton Hall University

44. Management Challenges • Management • Management buy-in is important and often tricky • Need a process for resolving disputes • Arbitration—by whom? Where? • Need a way to judge technical merit in addition to legal rights • Needs to be seen as forging alliances, not as weakness or revealing secrets Seton Hall University

45. Management Challenges • Management metrics • Evaluation of cooperative projects and staff • Cost-benefit needs to include • Immediate benefits from collaboration • Benefits from long-term relationships (encourage!) • Multiple alliances and conflicting loyalties Seton Hall University

46. Management ChallengesManagement Contingency Policies • Management planning ahead • Three major aspects • Response to external stimuli • Management of organizational behavior • Management of processes Seton Hall University

47. Management ChallengesManagement Contingency Policies • Tensions in collaborative development  Need more clearly defined policies and practices   Need more flexible policies and practices  Need to detect changes and problems quickly and robustly   Need to coordinate responses, possibly delaying action  Need to promote information sharing   Need to protect intellectual property, privacy, security, & resources Seton Hall University

48. Management ChallengesCustomer Interaction • Customer interaction with multiple partners • Likely to blur responsibilities • Likely to confuse customer • Customer may provide conflicting information • Partners may have conflicting interpretations • Most importantly, likely to weary customer • Interact in customer’s language & idiom • Need a policy for customer interaction Seton Hall University

49. Legal challenges • Intellectual property rights—general • Application and tools • Can you reuse your component? • Can you use it in a competitor’s application? • Proprietary confidential information • Library source code, proprietary tools, data, … • Test suites and case studies • Performance of partner applications or process • Overlapping alliances and conflicts Seton Hall University

50. Legal Challenges • Intellectual property rights • Right to publish and disseminate • Revealing experimental results • Indirect intellectual property • Insights developed in collaborative process • Process and practices • Whose law governs what? Seton Hall University