A Comprehensive Approach to Quality (COACH) - Enhancing IETF Work Groups

1. A Comprehensive Approach to Quality (COACH) IETF 57 Vienna, Austria Monday, July 14, 2003 13:00 – 15:00 http://www.drizzle.com/~aboba/IETF57/COACH/

2. Agenda • Preliminaries (13:00 – 13:05) • Agenda Bashing • Bluesheets • Minutes • Introduction (13:05-13:15) • Existing RFC editorial guidelines - Scott Bradner • http://www.ietf.org/rfc/rfc2360.txt • Quality: Overview and Framework (13:15-13:45) • A Comprehensive Approach to Quality (15 minutes), Bernard Aboba • http://www.ietf.org/internet-drafts/draft-loughney-coach-00.txt • IETF Problem Resolution Processes (15 minutes) - Margaret Wasserman • http://www.ietf.org/internet-drafts/draft-ietf-problem-process-00.txt • Starting New Work (13:45-14:05) • The BOF Process: A Critique (10 minutes) - Leslie Daigle • IESG Overload and Quality of WGs (10 minutes) - John Klensin • http://www.ietf.org/internet-drafts/draft-klensin-overload-00.txt

3. Agenda • The WG Process (14:05-14:15) • Decision points/milestones in the WG process (10 minutes) - Margaret Wasserman • http://www.ietf.org/internet-drafts/draft-wasserman-wg-process-00.txt • The Review Process (14:15-14:40) • Careful Additional Review of Documents (CARD) (15 minutes) - Brian Carpenter • http://www.ietf.org/internet-drafts/draft-carpenter-solution-sirs-01.txt • The Review Process in Action: The DCCP WG (10 minutes) - Aaron Falk & Allison Mankin • Summary and Discussion (14:40 – 15:00)

4. Some Groundrules • A mindset. • “The truth is out there” - Mulder • A reading List. • People who have read at least three of the documents, please sit in the first three rows. • A format. • Strict time limit for each topic & presentation. • Q&A at the end. • Expectations of Behavior. • Please use the mike. • Please state your name before speaking. • Only one person at the mike at a time. • Please do not interrupt the person at the mike. • Please relinquish the mike when requested to do so by the chair. • Please show courtesy to your fellow IETF participants.

5. A Comprehensive Approach to Quality (COACH) John Loughney Bernard Aboba Draft-loughney-coach-00.txt

6. Basic Principles • IETF Working Groups are responsible for completing work… • And are accountable for the quality of that work. • If Working Groups are to improve the quality (and timeliness) of the work… • It is necessary for them to plan for, and carry out, that improvement. • If the IETF is to improve… • It is necessary to do post-mortems on the plans.

7. A Theory • Many reasons why Working Groups deliver poor quality documents. • Unlikely that the plan was to deliver poor quality documents all along. • More likely that the level of effort, skill and experience required is beyond the capability of working group participants. • We can’t all be experts in everything. • Poor quality not the result of bad intentions, but insufficient resources. • Naïve optimism required to start the work… • But not enough to finish it.

8. What is a WG Quality Plan? • A public document no more than 5 pages long • Made available on the IETF web site. • Developed in concert with the Working Group charter • Revised when the WG charter or schedule changes significantly. • Since each WG is different, no “one size fits all” quality plan. • Sharing of “best practices” is encouraged. • Signed off by the AD and the WG founders • Reviewed by the IETF community

9. Potential Components of a WG Quality Plan • WG charter – the foundation. • What the WG plans to do (documents), who will do it (chairs, editors, advisors), by when (milestones & schedule). • Challenge assessment – what’s in the way (reality check) • The technical issues. • The dependencies. • Areas of skill, knowledge or experience that are lacking. • Tools plan. • WG mailing list. • WG web site. • Issue tracking tools. • Revision control systems. • Document production and build environments. • Review plan. • Checkpoints. • Required reviewer skill set. • Review process. • Reviewer recruitment.

10. Challenge Assessment • The Charter: What the WG is trying to achieve, and the resources at its disposal. • The Challenge Assessment: “what does the WG need to do in order to have a high probability of completing the work on schedule and with high quality?” • Some questions • Are the goals achievable? • With infinite resources? With the resources available? • What are the key assumptions? • What are the risks? • Within the core area of expertise of the WG? • Outside the core area of expertise? • How can the risks be mitigated?

11. The Tools Plan • The technology and host for the Working Group mailing list. • Anti-spam plan. • Archive access. • The Working Group web site. • Permanence? BW limits? • Document production system. • Tools & templates • Build environment • Revision control system. • Issue tracking and reporting system.

12. Issue Tracking & Reporting • An important tool for: • Tracking issues: encourages participants to raise issues, knowing that they won’t be “forgotten” • Focusing WG discussion: encourages discussions that lead to document improvements. • Measuring progress: • Provides metrics for assessing current status: • Open/resolved • Accept/reject • Days to resolution • Enables estimates of the task remaining

13. The Review Plan • Challenge Assessment review. • Review of the WG Charter • Should include reviewers from outside the area of the Working Group • Work item review. • Review of documents before they become WG work items. • Does the architecture make sense? • Are there major issues lurking? • Midterm assessment. • Is the document on the right track? Has something important been missed? • Late review. • Is the document ready for WG/IETF last call? • More on this later…

14. An Observation • Most documents are competent in the Area in which the WG resides. • Transport Area WGs understand transport. • Security Area WGs understand security issues. • Problems, when they occur, are generally in areas outside of the core competence of the WG • Internet Area documents with security issues. • Security documents with transport issues. • Suggestion: Cross-Area review is an important aspect of the review plan.

15. The Post Mortem • No more than three pages long. • Quality assessment. • An evaluation of the quality of working group output, written by the Area Director. • Challenge Assessment. • The unforseen challenges that the Working Group encountered. • Tools assessment. • An evaluation of the tools that were used and how well they worked. • Reviews assessment. • An assessment of the review process. • Recommendations. • Recommendations to future working groups looking to avoid similar problems.

16. Required Materials(What a WG might produce?) • An archive of sample document build environments and templates. • A website covering issue tracking and reporting tools. • A document on use of tracking tools for document management, including metrics and reporting. • A document on the review process, including Challenge Assessment. • A website including sample quality plans and post-mortems.

17. Summary • Quality doesn’t just happen, it can be planned. • The WG charter provides what, who, when. • The WG quality plan provides the how. • The resources available need to match those required to finish the job. • It isn’t what we know, it’s what we think we know that isn’t so.

18. Feedback?