802.11 Comment Resolution – a Tutorial

802.11 Comment Resolution – a Tutorial Authors: Date: 2013-03-19 Adrian Stephens, Intel Corporation

This tutorial is intended to help 802.11 members to approve adequate comment resolutions to working group and sponsor letter ballots The audience is anybody in 802.11 involved in writing or voting on comment resolutions See also 11-11/1625 “Comment Resolution Guide” for more detail Red text is reserved for examples of poor practice Introduction Adrian Stephens, Intel Corporation

We haven’t always don’t an adequate job in the past, which has cost us time/hot-air/embarrassment/quality We have specific rules that we need to follow New people are coming in all the time, which dilutes “corporate memory” of what to do and what to avoid We should, as much as we can, avoid “learning by making mistakes” because the cost of making mistakes in this process can be significant Why do we need a tutorial? Adrian Stephens, Intel Corporation

To increase the level of support by ballotters for a draft To improve the quality of a draft standard To engage with voters and gain from their viewpoint/experience To complete a ballot with a draft of high quality that meets the approval of most (>75%, typically 95%) of its voters in a timely fashion. Note the tension – frequently cannot satisfy all the commenters and complete in a timely fashion The purpose of comment resolution Adrian Stephens, Intel Corporation

We do not gain sufficiently from the possible benefits of engaging with voters We produce a poor quality draft document We get inadequate involvement from WG members who might give up trying to the process We risk delay by not dealing with real issues in a responsive or timely fashion We risk delay in the EC through appeals to process We risk delay in the IEEE-SA standards board through appeals to the process What can happen if we do this badly? Adrian Stephens, Intel Corporation

A comment needs to identify where the issue is in the draft It needs to identify what the issue is It needs to identify a proposed change in sufficient detail that the CRC can readily identify changes that they would reasonably expect to satisfy the commenter. The wording from the IEEE-SA Standards Board Operations Manual (2010 p24) is: “This vote must be accompanied by one or more specific objections with proposed resolution in sufficient detail so that the specific wording of the changes that will cause the Do Not Approve voter to change his or her vote to Approve can readily be determined .” Needs to be in scope (next slide) Valid Comments Adrian Stephens, Intel Corporation

A comment is in scope if: • It is in the first ballot • It is in a recirculation, and relates to material in the balloted draft changed since the previous draft, or relates to material affected by that change • Is in a recirculation, and relates to text that is the target of an unsatisfied “must be satisfied” comment from a “no” voter. Valid comment scope Adrian Stephens, Intel Corporation

Examples of invalid comments - 1 Adrian Stephens, Intel Corporation

Examples of invalid comments – 4 Adrian Stephens, Intel Corporation

Address all comments – be responsive to the technical issue wherever this is possible • Answer the comment made, not the comment you wish they’d made. Answer all the points in the comment. • The proposed change should relate to the comment directly, not be an excuse to fix something else. • Minimize procedural resolutions, except near the “end” • Address the technical issues in preference to using a “scope” or “invalid comment” resolution. • Use neutral language – address the issue, not the commenter Recommendations for comment resolution Adrian Stephens, Intel Corporation

Use the correct resolution • Accepted means “we agree with the comment and the proposed change”. It cannot be used if the proposed change is not sufficiently detailed that the editor and TG members know what the intended change is, e.g. if the commenter offers alternative solutions. • Revised means “we agree with the comment, but we approved a change different from the commenter’s proposed change” • Rejected means “we disagree with the comment, and made no changes”. Don’t use “Rejected” if the comment is addressed in another comment. Instead, repeat part of the resolution of that comment as appropriate. Don’t use “Rejected - Fixed in Draft x.y” as a resolution because it doesn’t say what you changed to resolve the comment. Technical aspects of comment resolution - 1 Adrian Stephens, Intel Corporation

Avoid over-use of references to submissions • This forces the commenter to do work that you could have done. • Copy and paste textual resolutions from submissions. • Reserve references for material that cannot be represented in plain text, e.g. graphics and heavy use of insert/delete. • e.g. – bad usage: “Rejected – see 11-12/1234r0”. • Use URLS • Any references to external documents should be URLs, not 802.11 document numbers. This is particularly important in sponsor ballot, because Technical aspects of comment resolution - 2 Adrian Stephens, Intel Corporation

Avoid references to other comment resolutions • When we present unsatisfied comments to the EC (or RevCom read them), they see only a subset of comments. When RevCom look at a comment resolution, they do not necessarily see the same comment ID as the TG works with. • These both mean that references to comment IDs in resolutions are useless. • Instead of referencing a comment resolution – copy it. • For example: • Revised. At page 1234 line 56, change ‘nurgle’ to ‘flange’. • Note to editor (this is the same as comment resolution for CID 321). Technical aspects of comment resolution - 3 Adrian Stephens, Intel Corporation

Invalid Resolution:“Rejected. We will consider this comment in a later ballot/the next revision, but we ran out of time to discuss it now.” Avoid promises of future action. Any such promise is an immediate red flag to the EC / RevCom. A member of the TG who is in the sponsor ballot pool might offer to submit certain comments from the end of WG ballot into sponsor ballot; but if he does so, it is an individual act, not an official one. Technical aspects of comment resolution – 4 – promises of future action Adrian Stephens, Intel Corporation

Resolution should include: • “Accepted”, “Revised” or “Rejected” classification. • A response to each issue raised in the comment. • Clearly distinguish between respond to the commenter and instructions to the editor. • e.g.: • “Revised. • At p 1234 line 56 change “shall” to “should”. • In reply to the commenter, this allows legacy STAs to continue this behaviour, but allows new STAs to choose what to do. Technical aspects of comment resolution – 5 structure of resolution Adrian Stephens, Intel Corporation

Generally comments are resolved without regard to how the commenter classifies them, except: • Usually the editor gets to propose resolutions to most of the editorial comments • Editorial comments that are not part of a no vote might be resolved “This comment has been passed to the TG technical editor for consideration during preparation of a subsequent draft” • Near the end of the sponsor ballot, editorial comments might be resolved with “This comment will be passed to the IEEE-SA editor for consideration during publication editing.” • The last two are permissible exceptions to the “don’t promise future action” rule. Technical / General / Editorial comments Adrian Stephens, Intel Corporation

Examples of Unresponsive Comment Resolutions Adrian Stephens, Intel Corporation

Resolution: Reject. The TG considered the comment and did not agree with the proposed resolution. Analysis: Unresponsive. The response is generic and does not include the subject matter of the comment. Responsive / unresponsive - 1 Adrian Stephens, Intel Corporation

Resolution: The TG wasn’t sure whether to accept the comment or not and couldn’t agree on a change Analysis: Unresponsive. The response is wishy-washy and leaves the commenter with inadequate information on the TG’s position. Responsive / unresponsive - 2 Adrian Stephens, Intel Corporation

Resolution: Reject. Coexistence with mode X devices is not a requirement of the PAR. The TG considered the comment and did not agree with the proposed resolution on the basis that it added significant complexity with insufficient benefit. Analysis: Responsive. The response includes the subject matter of the comment, and provides concise reasons why the comment was rejected. It is not necessary for the resolution to provide extra detail on precisely how much complexity would be tolerable and how the TG members traded off complexity versus benefit. Responsive / unresponsive - 3 Adrian Stephens, Intel Corporation

Types of Procedural Rejection Adrian Stephens, Intel Corporation

Procedural Rejections – 1 (insufficient detail) Adrian Stephens, Intel Corporation

Procedural Rejections – 2 (asking a question) Adrian Stephens, Intel Corporation

Disagree. The comment fails to identify changes in sufficient detail so that the specific wording of the changes that will satisfy the commenter can be determined. Note, the CRC can (but is not required to) admit or accept the validity of any reported “big issue”. If it wants to leave the resolution as a straightforward reject, this comment can be used to later introduce changes to resolve the big issue. Note also that the CRC doesn’t need a valid comment to make any changes it wishes, and if it agrees that there is a big issue and some ballot cycles later determines how to resolve that issue, it is at liberty to make those changes. Procedural Rejections – 3 (big issue) Adrian Stephens, Intel Corporation

Disagree. The comment is out of scope: i.e., it is not on changed text, text affected by changed text or text that is the target of an existing valid unsatisfied comment. Procedural Rejections – 4 (out of scope) Adrian Stephens, Intel Corporation

Context: The TG cannot reach a consensus on any other resolution. • The CRC have extensively debated the comment and there are different views on how to respond to it. One or more other resolutions have been considered but no such resolution has received 75% approval. • It should be recorded in the minutes what other solutions have been considered, together with the results of any straw polls or motions on those alternative solutions so that the commenter can understand that this solution was used as a last resort. • Disagree. The CRC could not reach consensus on any change to the current text that would satisfy this comment. • Use with caution. The TG chair may well have to stand up in front of the EC or RevCom and justify use of this resolution. • You have been warned! Procedural Rejections – 5 (can’t agree on a change) Adrian Stephens, Intel Corporation

Approval cycle: • WG Ballot • EC approval to proceed to sponsor ballot • Sponsor Ballot • EC approval to proceed to RevCom • IEEE-SA RevCom and Standards Board approvals • Permissive vs restrictive • Solicit input from “no” voters • Recirculate unchanged • EC approval • RevCom / IEEE-SASB approval Typical Ballot Lifecycle Adrian Stephens, Intel Corporation

In the first ballot, all comments are in scope (although not all comments are necessary valid). Early in the process, the TG will attempt to satisfy comments regardless of whether they are valid As the draft nears the end of a sequence of recirculation ballots, the TG wants to proceed to the next stage in the process. At this point they are likely to become restrictive, rejecting comments on the basis of validity. Ballot Lifecycle - Permissive vs restrictive Adrian Stephens, Intel Corporation

Comment resolution provides an opportunity for 2-way communication between the commenter and the TG. There is no reason to limit this to communicating via the official tools. The TG are encouraged to engage directly with commenters to understand better what they are saying in a comment, and to seek feedback on its possible resolution. The TG chair should seek feedback from unsatisfied comments from previous rounds so that the actual unsatisfied comments are identified for presentation to the EC. Ballot Lifecycle - Solicit input from “no” voters Adrian Stephens, Intel Corporation

At the end of some recirculation ballot, unless the ballot received no comments, the group will determine that there are reasons to validly reject all the comments received. • In this case an unchanged draft is sent to the next (final) recirculation along with the reasons for these rejections • Comments might be received from this final recirculation, but they are either a reiteration of an earlier comment (i.e., not new) or they are out of scope. • Thus the process is guaranteed to terminate, should the TG so wish • i.e., no commenter can force the process to continue indefinitely Ballot Lifecycle - Recirculate unchanged Adrian Stephens, Intel Corporation

The approval of the EC is required to progress between WG ballot and Sponsor ballot, and to go to RevCom Approval is granted (either unconditional or conditional) at plenary sessions of 802 (March, July, Nov) The TG produces and approves a report on the status of the ballot and unsatisfied voters/comments to solicit this approval. In the case of conditional approval, an email on meeting the terms of the conditional approval is sent by the WG chair to the EC when the terms are met. Ballot Lifecycle - EC approval Adrian Stephens, Intel Corporation

RevCom reviews material it retrieves from the MyBallot system in order to determine whether the process for sponsor ballot has been properly followed. They might review unsatisfied comments, and have visibility of all comment resolutions. RevCom makes a recommendation to the standards board (IEEE-SASB), which is responsible for the actual approval. Ballot Lifecycle - RevCom / IEEE-SASB approval Adrian Stephens, Intel Corporation

During WG comment resolution, the TG resolves comments • Motions can be made only during 802.11 sessions, except • Once conditional approval has been obtained from the EC, a TG telecon can approve resolutions by motion • During Sponsor ballot comment resolution, the WG chair delegates responsibility to a comment resolution committee (CRC) to resolve comments • Any 802.11 voter can join this CRC, which is practically indistinguishable from a TG, except that it can vote during telecons. Difference between WG and Sponsor Ballot Adrian Stephens, Intel Corporation

Post timely spread sheets. All 802.11 members should have essentially equal access to the comments and their proposed resolutions. • Ensure all comments have a valid resolution • Ensure resolutions are actioned in the draft. This is mainly the responsibility of the editor. • Nothing discombobulates commenters more than the group approving a change than then not making that change. • Chase unresponsive “no” voters • Prepare reports to the EC • (Tools, support and training are available from WG officers) Requirements on TG officers Adrian Stephens, Intel Corporation

If this tutorial informed you, but has no effect on your future behaviour, your attendance was a waste of time. • What are you going to do to improve the quality of your comment resolutions? • Call to action: Be critical of your own and others comment resolutions. • Spending time getting this right now should save you time in the future. The end? Adrian Stephens, Intel Corporation

802.11 Comment Resolution – a Tutorial