Professional Software Development: Shorter Schedules, Higher Quality Products, More Successful Projects

1. Professional Software DevelopmentShorter Schedules, Higher Quality Products, More Successful Projects, Enhanced Careersby Steve McConnell Matt Sawka

2. About the Author • Steve McConnell is has a bachelor's degree Whitman College and a master’s degree in Software Engineering from Seattle University. • He has been working in the Software industry since 1984, including being a consultant for Microsoft. • From 1998-2002, he was Editor and Chief of IEEE Software magazine. • He is currently on a panel of experts that advises the Software Engineering Body of Knowledge project and is a Vice Chair of the IEEE Professional Practices Committee. • Currently the CEO and Chief Software Engineer of Contrux Software. • http://www.stevemcconnell.com/ • http://www.contrux.com/

3. Purpose • What is software engineering? • How does software engineering relate to computer science? • Why isn’t regular computer programming good enough? • Why do we need a profession of software engineering? • Why is engineering the best model for a software development profession? • In what ways do effective practices vary from project to project (or company to company), and in what ways are they usually the same? • What can organizations do to support a professional approach to software development? • What can individual software developers do to become full-fledged professionals? • What can the software industry as a whole do to create a true profession of software engineering?

4. The Software Tarpit Wrestling with Dinosaurs Fool’s Gold Cargo Cult Software Engineering Body of Knowledge Novum Organum Individual Professionalism Orphans Preferred Raising Your Software Consciousness Building the Community      Programmer Writing Organizational Professionalism Software Gold Rushes Business Case for Better Software Practices Ptolemaic Reasoning Quantifying Personnel Factors Construx’s Professional Development Program Industry Professionalism Engineering a Profession Hard Knocks Stinking Badges The Professional’s Code Alchemy Organization

5. Part OneThe Software Tar Pit

6. Chapter 1:Wrestling with Dinosaurs “He that will not apply new remedies must expect new evils, for time is the greatest innovator.” –Francis Bacon

7. Wrestling with Dinosaurs • “No scene from prehistory is quite so vivid as that of the moral struggles of great beasts in the tar pits. In the mind’s eye one sees dinosaurs, mammoths, and sabertoothed tigers struggling against the grip of the tar. The fiercer the struggle, the more entangling the tar, and no beast is so strong and so skillfully but that he ultimately sinks. Large-system programming has over the past decade been such a tar pit, and many great and powerful beasts have thrashed violently in it Most have emerged with running systems – few have met goals, schedules, and budgets Large and small, massive or wiry, team after team has become entangled in the tar No one thing seems to cause the difficulty – any particular paw can be pulled away. But the accumulation of simultaneous and interacting factors brings slower and slower motion. Everyone seems to have been surprised by the stickiness of the problem, and it is hard to discern the nature of it But we must try to understand it if we are to solve it. -The Mythical Man-Month (p. 3), Fred Brooks. 1975

8. Wrestling with Dinosaurs • Has anything changed? • ~75% of all medium-sized projects and 90% of or more of large projects are subject to excessive schedule pressure. • Overtime is the standard. • “In many companies, programmers faced with deadlines have been known to spend nights in the offices.” – Fortune, 1967 • Windows NT was projected to take 1500 staff-years to complete. • OS/360 took more than 3 times this amount in 1966. • The advent of web-development poses the question is delivering software quickly better than delivering quality software? McConnell argues that this is not a new phenomena.

9. Chapter 2:Fool’s Gold “Hope is a good breakfast, but it is a bad supper.” –Francis Bacon

10. Fool’s Gold • During the California Gold Rush of the late 1800s, many were deceived by fool’s gold – iron pyrite, which has the same appearance as Gold. • The last fifty years in software development has produced similar results. Software Engineers seeking solid processes and standards have mostly found fool’s gold, software practices that appear useless on first glance, but are actually flaky, brittle, and virtually valueless.

11. Fool’s Gold • Problem: Looking back further in history, how could we build one of the ancient pyramids from ancient Egypt? • The stone blocks need to be moved 10,000 meters from the river to their final resting place. • You have 100 days to move the blocks. • You have 20 people to move the blocks. • You are allowed to use any method you’d like. • On average, you must move the blocks 100 meters a day (or have a method that will speed up the process later on). • How would you accomplish this?

12. Fool’s Gold • Option 1: • Immediately start pushing the blocks with brute force. • Result: You move the block 10 meters a day and fall 90 meters/day behind.

13. Fool’s Gold • Option 2: • Analyze the problem. You decide to cut down several trees and use them as rollers to move the block. Create a smooth, level roadway to push forward. • Result: Although there is an initial investment to find the trees, it still will pay off.

14. Fool’s Gold • Option 3: • Implement option 2, but balance the pushers with pullers and add log-movers, so that a group is always moving extra logs in front of the block. • Result: An improvement on option 2, the block will be continuously moving.

15. Fool’s Gold • How does software development relate to ancient Egyptian pyramids? • Change the pyramid block to a software development project. • You have 100 days to complete a project. This means you either have to complete 1/100 of the source code each day or you need to schedule some parts taking less time than others. • Avoid “last-minute” syndrome (Option 1) • The development team has little concern near the beginning of the project, but has a frenzied push at the end of the development cycle.

16. Fool’s Gold • How does software development relate to ancient Egyptian pyramids? • Also avoid “code-and-fix” development (Option 2) • Put a brute force of programmers on the project without properly planning or design of the software. The project team will show initial progress but will be unable to finish strong. Most effort will go into defects.

17. Fool’s Gold • The Silver Bullet Syndrome • An elephant could be capture to pull block. However, after it is captured, it tramples 2 workers and runs off. The managers then think they should’ve spent time learning to handle the elephant.

18. Fool’s Gold • As project planning matures, the amount of effort spent on later stages should be manageable. • Despite the downfalls, “code-and-fix” is still heavily used today for two reasons: • It provides initial signs of progress • It requires no training

19. Fool’s Gold • Focusing on Quality • If an organization can remove ~95% of their defects before a release, they can minimize the amount of effort spent on correcting them later.

20. Fool’s Gold • Software isn’t “soft”. • Let’s suppose you are designing a system to print a set of 5 reports, and eventually 10 reports. • How the reports can be “soft”: • Is ten an upper limit on thee number of reports? • Will the future reports be similar to the initial five reports? • Will all of the reports always be printed? • Will they always be printed in the same order? • To what extent will the user be able to customize the reports? • Will users be allowed to define their own reports? • Will the reports be customizable and definable on the fly? • Will the repots be translate to other languages?

21. Fool’s Gold • How the reports can be “hard”: • Defining more than 10 reports. • Defining a new report that is different form the initial set of reports. • Printing a subset of the reports. • Printing the reports in a user-defined order. • Allowing the user to customize reports. • Translating the repots to another language that users the Latin alphabet. • Translating the reports to a another language that uses a non-Latin alphabet or reads right-to-left. • Bottom Line: Flexibility costs money. Limiting flexibility can save money in the short term, but can incur higher costs later on the development life-cycle.

22. Chapter 3:Cargo Cult Software Engineering “In the South seas there is a cargo cult of people. During the war they saw airplanes with lots of good materials, and they want the same thing to happen now. So they’ve arranged to make things like runaways, to put fires along the sides of runways to make a wooden hut for a man to sit in, with two wooden pieces on his head for headphones and bars of bamboo sticking out like antennas – he’s the controller – and they wait for the airplanes to land They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No airplanes land. So I call these things cargo cult science, because they follow all the apparent precepts and forms of scientific investigation, but they’re missing something essential, because the planes don’t land.” –Richard Feynman

23. Cargo Cult Software Engineering • Process-oriented vs. Commitment-oriented Development styles • Process-oriented • Relies on a carefully defined process, planning, scheduling, and directly application of Software Engineering best-practices. • This succeeds because the organization is constantly improving on their best practices. • Commitment-oriented • Also known as “hero-oriented” or “individual empowerment”, this style relies on hiring the best people and asking them for total commitment to a project. They work with completely autonomy something work 60-100 hours a week until a project is completed. • This style succeeds because it utilizes individual motivation.

24. Cargo Cult Software Engineering • Imposters: • Process-oriented imposter (Bureaucratic) • Observes successful companies using best-practices (such as NASA’s Software Engineering Laboratory), see that they have many meetings and documents and emulate the deliverables only. • Commitment-oriented imposter • Observe successful companies like Microsoft, and emphasize the long hours and large compensation packages, not the fact that the employees of Microsoft love to create software. • Cargo-cult engineering is simply using a development style “just because” or “because the company requires it”, rather than using the style advantageously.

25. Chapter 4:Software Engineering, Not Computer Science “A scientist builds in order to learn; an engineer learns in order to build.” –Fred Brooks

26. SE not CS • Like other engineering disciplines, Software Engineering can be tailored for several objectives: • Minimal defects • Maximum user satisfaction • Minimal response time • Good maintainability • Good extendibility • High robustness • High correctness. • Unlike other disciplines in which risk relies on the physical materials, Software Engineering carries risk in optimizing the project goals. • Short schedule • Predictable delivery date • Low cost • Small team size • Flexibility to make mid-project feature-set changes.

27. SE not CS • What is the best way to think of software development? Is it a science? Art? Craft? Something else? • ~40% of developers today have degrees in Computer Science. • Scientists learn what is true, how to test hypotheses, and how to extend their knowledge. • Engineers learn what is true, useful, and how to apply their knowledge to solve a problem. • Is Software Engineering just a buzzword? • Some object to Software development being classified as engineering because the commercial market doesn’t allow for the careful, time-consuming engineering process to be completed.

28. Chapter 5:Body of Knowledge “Truth will sooner come out of error than from confusion.” –Francis Bacon

29. Body of Knowledge • To be an expert in a field, a person needs to know around 50,000 pieces of information. • But with software engineering evolving, how can anyone know enough to be considered an expert? • Essence vs. Accident • No Silver Bullet – Essence and Accidents of Software Engineering – Fred Brooks, 1987. • Essence – properties that something must have. • Wheels on a car • Software engineering: Specification, Design, and Verification. • Accident – optional properties. • Air-Conditioning • Software Engineering: Coding and testing.

30. Body of Knowledge • Computer programs are complex. • At the center, the goal is to define underlying real-world concepts and debugging that understanding. • Software must be flexible and have the ability to change. • If a program is successful, more people will use it and it will be adapted to be used outside of its original scope. • Software is “invisible.” • It’s not possible to create a 2-d or 3-d geometric model of the system.

31. Body of Knowledge • In 1968, NATO held its first conference on Software Engineering. • McConnell estimates that in 1968, the average half-life of knowledge was about 10 years, with only about 20% of that knowledge at the Stable Core.

32. Body of Knowledge • By 2003, McConnell estimates that the average half-life of knowledge was about 30 years, with about 50% of that knowledge at the Stable Core.

33. Body of Knowledge • How can we categorize the Software Engineering “body of knowledge” (SWEBOK)?

34. Body of Knowledge • What sources contribute to the SWEBOK?

35. Body of Knowledge • What information is contained within SWEBOK? • SWEBOK Knowledge Areas • Software Requirements • The discovery, documentation, and analysis of the functions to be implemented in software. • Software Design • Definition of the basic structure of the system at the architectural and detailed levels, division into modules, definition of interfaces for modules, and choice of algorithms within modules. • Software Construction • Implementation of the software including detailed design, coding, debugging, unit testing, technical reviews, and performance optimization. This area overlaps somewhat with Software Design and Software Testing.

36. Body of Knowledge • SWEBOK Knowledge Areas • Software Testing • All activities associated with executing software to detect defects and evaluate features. This includes planning, test-case design as well as the tests themselves: development, unit, component, integration, system, regression, stress, and acceptance. • Software Maintenance • Revision and enhancement of existing software, related documentation, and tests. • Software Configuration Management • Identification, documentation, change control of all deliverables generated on a project (source code, content, requirements, etc…). • Software Quality • All activities associated with providing confidence that a software item conforms or will conform to technical requirements.

37. Body of Knowledge • SWEBOK Knowledge Areas • Software Engineering Management Plan • Planning, tracking and controlling of software projects, work, and organizations. • Software Engineering Tools and Methods • Tolls and methodologies support (CASE tools, reusable libraries, formal methods and practices, etc…). • Software Engineering Process • Activities related to improving development quality, timeliness, productivity, and other characteristics. • http://www.swebok.org/

38. Chapter 6:Novum Organum “A prudent question is one-half of wisdom.” –Francis Bacon

39. Novum Organum • In the 1620s, Francis Bacon published Instauratio Magna which attempted to redefine scientific inquiry. Within this work is an essay, Novum Organum which challenges his colleagues to focus on scientific methodologies rather than deductive reasoning when studying the world. His view of the scientific method had three steps: • Purge your mind of prejudices (superstition) • Collect observations and experiences systematically. • Stop, survey what you’ve seen, and draw an initial conclusion.

40. Novum Organum • What does it mean to have a Software Engineering “profession”? • According to the Code of Federal Regulations, a profession has: • A requirement for extensive learning and training • A code of ethics imposing standards higher than those normally tolerated in the marketplace. • A disciplinary system for professionals who breach the code • A primary emphasis on social responsibility over strictly individual gain, and a corresponding duty of its members to behave as members f a disciplined and honorable profession. • A prerequisite of a license priori to admission to practice.

41. Novum Organum • Is Software Engineering a profession? • The Software Engineering Institute (SEI) has identified 8 elements of a mature profession.

42. Novum Organum • Elements to a mature profession: • Initial Professional Education • Completing a university program in their field. • Accreditation • The university program is accredited by an oversight body that determines if the program provides adequate education. Accreditation Board for Engineering and Technology (ABET) oversees American engineering programs. • Skills Development • Education is not enough to develop full professional capabilities, some type of further experience is needed to perform the job individually with a satisfactory level of competence.

43. Novum Organum • Elements to a mature profession: • Certification • A professional is requirements to pass one or more exams that ensures they have a minimum level of knowledge. • Licensing • Similar to certification, this is a mandatory exam administered by a overrunning authority. • Professional Development • Ongoing professional education to maintain or improve a worker’s knowledge and skills.

44. Novum Organum • Elements to a mature profession: • Professional Societies • A Community of like-minded individuals who put their professional standards above their self-interest. • Code of Ethics • Ensures that a profession’s practitioners behave responsibly. • *Organizational Certification • Organizations must also be certified.

45. Novum Organum • Maturity Levels for the elements • Nonexistence • The element simply doesn’t exist. • Ad Hoc • The element exists, but only in isolated instances • Established • The element exists and is clearly identifiable. • Maturing • The element has existed for many years and is being maintained and improved.

46. Novum Organum • How does Software Engineering rank? • Initial Professional Education – Ad Hoc to Established • Accreditation – Established • Skills Development – Established • Licensing – Ad Hoc • Professional Development – Ad Hoc • Professional Societies – Established • Code of Ethics – Established • Organizational Certification - Established

47. The Software Tar Pit Summary • Wrestling with Dinosaurs • Has anything changed? • Fool’s Gold • Building the Egyptian Pyramids • Code-And-Fix, Last-Minute, Silver-Bullet • Software isn’t “soft.” • Cargo Cult Software Engineering • Process-Oriented vs. Commitment-oriented • Software Engineering Not Computer Science • Body of Knowledge • SWEBOK • Novum Organum • What is a profession?

48. Part TwoIndividual Professionalism

49. Chapter 7:Orphans Preferred “Wanted: Young, skinny, wirey fellows not over 18. Must be expert riders willing to risk death daily. Orphans preferred. Wages $25/week.” -Pony Express, 1860 “We realize the skills, intellect, and personality we seek are rare, and our compensation plan reflects that. In return, we expect TOTAL AND ABSOLUTE COMMITMENT to project success – overcoming all obstacles to create applications on time and within budget.” –Jobs Rated Almanac, 1995 for an SE posting.

50. Orphans Preferred • “The stereotypical programmer is a shy young man who works in a darkened room, intensely concentrating on magical incantations that make the computer do his bidding. He can concentrate for 12 to 16 hours at a time, often working through the night to make his artistic vision a reality. He subsists on pizza and Twinkies. When interrupted, the programming creature responds violently, hurling strings of cryptic acronyms at his interrupter – “TCP/IP, RPC, RCS, ACM, and IEEE!” he yells. The programmer breaks his intense concentration only to attend Star Trek conventions and watch Monty Python reruns. He is sometimes regarded as an indispensable genius, sometimes as an eccentric artist. Vital information is stored in his head and his head alone. He is secure in his job, knowing that, valuable as he is, precious few people compete for his job.”