Organizational Patterns

1. Organizational Patterns James O. Coplien Bell Laboratories Naperville, Illinois, USA cope@research.bell-labs.com JaCC Software Developers 16 September 1999 11:00 - 12:30

2. Organizational Patterns James O. Coplien Bell Laboratories Naperville, Illinois, USA cope@research.bell-labs.com IIT Colloquium Series Tuesday, February 8, 2000 M-50 Auditorium

3. Overview • What are organizational patterns? • Methodology • Our model of organizational change • Using patterns for change management • The patterns themselves can be found at http://www.bell-labs.com/cgi-user/OrgPatterns/OrgPatterns

4. What is a pattern? • A solution to a problem in a context • Architectural patterns ideas first published by Alexander in 1977 • Look at issues of structure, not just parts • Build on proven practice, not “promising theories” • Adopted by the software community starting in 1990; started gaining critical mass by 1993 • Today, widely used as a design tool, especially in the OO community

5. What are organizational patterns? • Solutions to organizational problems in a context • First appeared in the Alexander + software context at PLoP in 1994 (Coplien, Whitenack); received with some skepticism • Now, a growing body of knowledge • Or, a construct from anthropology, Kroeber: • Universal patterns: transcend cultures • Systemic patterns: have a common root in an ancient culture • Total culture patterns: give a culture its identity • Patterns define culture

6. Why do we care about organizational patterns? • Because process-based approaches have serious limitations that hark back to the era of the industrial revolution • Because experience with project management, technology transfer, etc. show software to be a primarily social activity • Because of the tie to cultures, and the study of cultures is about social activity • Because we should care about human comfort in addition to product quality • Because they work!

7. The Pasteur Process Research Project at Bell Labs • Goal: To know how to design highly productive software development organizations • Premise: Organization drives process, which drives productivity • Strategy: Correlate properties of organizations with order-of-magnitude productivity • Technique: Patterns

8. A Theory of Organizational Change • The idea of an organization as an anthropomorphic entity • Some are intelligent: good reactions, learn from mistakes • Some are stupid: react in insane ways, don’t learn • Organizations can learn! • Organizations can’t be taught • Therefore, learning is experiential • Experience is a hard teacher • Can be helped with empirical foundations and introspection

9. Empirical foundations • “Pattern Mining” • Finding recurring structures and practices in healthy organization • To a degree, seeking the absence of these structures in dysfunctional organizations • A soft science • Akin to anthropology and its many branches

10. Organizational Pattern Mining • Based on social network analysis with informal extensions • Organizational role-play to gather social network data • Dyadic and triadic sociometric data • Uses CRC cards • Light facilitation • Half-day exercise

11. CRC Cards: Classes, Responsibilities, and Collaborators Subsystem coord. Validate MR lists Subsystem coord. Build group products Change committee Administer ENVY Designers System test Resolve logical deps.

12. Work-life Role-Play • Identify project roles • Study subjects play roles • Development scenarios drive role-play • Capture interaction & coupling on CRC cards • Social Network Analysis Tools • Organization Structure Visualization • Organizational Metrics • Capture Trends as Generative Patterns

13. Distilling the Patterns • Sociometric analyses • Pasteur analysis tool • Analytical studies of data • Catalog sociograms, sociomatrices, sociometric data • The pattern must recur • The pattern must solve a problem • Document in pattern form

14. QPW as a Subject • Fourth generation product in one generation • Remarkable press reviews • The most effective organization we have found • Productivity • Quality • Interval • Market • Used to calibrate our “spectrum” • Analysis done at my request • I am not a QPW user

17. QPW Organization Metrics

18. Writers’ Workshops • A forum to review patterns • Borrowed from the creative literature community • Designed to protect the dignity of the author • Several basic steps: • Moderator introduces author • Author reads the work • Author becomes a fly on the wall • Someone summarizes the pattern • The strengths are emphasized • Suggestions for improvement • Author asks for clarification • Thank the author

19. Using the Patterns • Let dysfunctional organizations become exposed to the patterns to see what is possible • Use organizational role-play as an introspection tool • Collect and process sociometric data from the role play • Present the data to the organization for a second round of introspection—a “mirror” • Learning and growth build on the insights

20. The Organizational Role-Play • Same technique as used for pattern mining • Largely self-running • Role-based • Useful in most development organizations • Actor-based • Useful for diagnosing specialized team problems • “Ethnological” studies

21. Facilitation Guidelines • Involve everybody • Don’t try solving other peoples’ problems • Other organizations must solve their own problems • Interface issues can be dealt with in another forum • Interfaces generally involve politics • Don’t look for “the problem” • Problems are systemic • Solutions are generative • Let the patterns cause the “aha”s

22. More principals of organizational change • Happens bottom-up, perhaps with the top-down support of a sponsor or patron • Patterns are closely tied to value systems—which must be elicited bottom-up • Organizational change means building a new culture: reverse anthropology • New language • New normative behaviors • New values • New symbols • New stories (patterns) • New rituals

23. What makes organizations excel? • No silver bullet • Well-known productivity enablers • A little luck • The right balance of techniques to make a working social (and economic and technical and...) system

24. Communication is Key • The “Buffalo Mountain” pattern • Decentralized, yet a graceful distribution • Almost fully connected • Anti-schmisogenetic • No splinter groups • No time-serial sub-processes • No “subroutines”

25. QPW Communication Intensity

26. Size • Easy to socialize the architecture • Self-directing • Correlates to Gerry Weinberg’s SEI/CMM parallel • The true hallmark of a mature organization • Divide and conquer • Essentially large organizations need hierarchy

27. Architect Also Implements Problem: Preserving the architectural vision through to implementation Context: An organization of developers that needs strategic technical direction. Forces: Totalitarian control is viewed by most development teams as a draconian measure. The right information must flow through the right roles. The architect is one of the few people, or may be the only person, with the big picture. Big picture knowledge is important even during implementation, especially when using iterative development. But the architect usually isn’t around during implementation. Solution: Beyond advising and communicating with developers, architects should also participate in implementation. Resulting Context: A development organization that perceives buy-in from the guiding architects, and that can directly avail itself of architectural expertise. The architect learns from implementation constraints and may modify the architecture accordingly. Rationale: Vitruvius notes: “...[A]rchitects who have aimed at acquiring manual skill without scholarship have never been able to reach a position of authority to correspond to their pains, while those who relied only upon theories and scholarship were obviously hunting the shadow, not the substance. But those who have a thorough knowledge of both, like men armed at all points, have the sooner attained their object and carried authority with them.” Larry Constantine emphasizes the value of “street creds” in an architect.

30. Distribute Work Evenly • Like QPW adjacency graph • Sociometric analysis: low graph centrality • Can be remedied by load leveling, staffing, splitting roles, combining roles

31. QPW Organization Metrics for Distribute Work Evenly

32. Personnel Issues • Personal excellence and integrity • Highly specialized • Balance of teamwork (communication) and individuation (specialization, personal sign-offs) • How does one assemble such a group? 1. Magic: A secret that only Philippe knows 2. Outside interests? (Gabriel) 3. Self-selecting Team (another pattern)

33. Engage Quality Assurance • QA is tightly coupled in the social network • Large beta program–customer engagement • Developers reserve early verification to themselves • QA first-in, last-out

34. Reward Excellence • A sense of contribution • Reinforced in project signoffs • Reinforced in reward system • Financial incentives • No first-hand insights into QPW • Controversial, but claims demonstrable results • Two cultures with extravagant rewards: 1. The U. S. West Coast 2. Financial trading • Overcomes fear of change

35. Process? • The ability to introspect well • The ability to self-manage • Organizational structure over task sequencing • Effective meetings, and lots of them

36. Conclusion • Organizational patterns are: • Patterns that capture the “architecture” of organizations • In anthropology, the fabric of social structure • Role-playing data gathering and introspection • Organizational change is generative • Organizations can use patterns as exemplars for improvement • A rich body of literature exists and is growing