Download
1 / 23
230 likes | 244 Views
Explore the evolution of Gen-ART's role in the IETF Last Call process, from review assignments to reviewer responsibilities and key improvements made over the years.
E N D
Mary Barnes (Gen-ART secretary) IEFT Last Call and Gen-ARTIETF 72WG chairs July 30, 2008
Overview • Background of Gen-ART • Summary IETF-LC Process • Gen-ART Role during IETF-LC: • Reviewer assignment process • Reviewer responsibilities • Handling review comments • Other Considerations • Summary
Gen-ART Background (1) • Original Gen-ART review team formed by Harald Alvestrand in 2003 (original secretary Avri Doria). • Assignments were made at time document was placed on telechat agenda • Objective was to aid the General Area Director to more effectively determine the readiness of documents • Original team was comprised of less than 10 active reviewers • Around 1380 documentreviews completed since 2004 - averaging 350/year
Gen-ART Background (2) • Reviews were often sent right up to the start of the telechat: • Completing reviews was done under a “best effort” model • Many reviews were sent only to gen-art mailing list • Review guidelines evolved based on feedback from reviewers • Oftentimes, recipients of reviews were not certain how they should be handled • In general – a positive response to reviews • IETF Last Calls added as a “nicety” or “wbn” in 2005.
Gen-ART Evolution (1) • Kicking off the review process for all documents during IETF LC was initiated by Brian Carpenter • Overall, this has improved quality in terms of the number of documents that are deemed ready at Telechat time, based on 2007 data: • 250 documents assigned at IETF LC time • 80% of the last call reviews completed (205 documents) • Around 25% of the docs are deemed “Ready” at LC time. • 75% of the docs were ready at Telechat time (144 documents) • For the docs not reviewed at LC time, around 20% were ready at the time of the telechat.
Gen-ART Evolution (2) • Gen-ART and other directorate/review team reviews (e.g., Secdir) are now expected - the reviews are received, considered and responded to along with other LC comments. • Many ADs now consider the reviews as input in forming their decisions.
IETF LC Process Overview • Gen-ART does not add steps to the process. • Input is part of the IETF Community Review in the form of “Comments, Suggestions” in a std manner. • Gen-ART reviewer is very often involved in resolution of “Concerns”, but reviews are non-blocking although Gen-ART ADs consider reviews in forming their positions.
Gen-ART: Reviewers • Current Team is comprised of around 13 individuals: • Past and present WG chairs • Past and present IAB members • Key technical experts from various WGs/areas • Past ADs/IETF chairs • 5 Reviewers only receive one document assignment per month • List of current/past reviewers: http://www.alvestrand.no/ietf/gen/art/reviewer-list.html List is dynamic as reviewers’ day job workloads vary. • Current size is due more to availability of reviewers than any objective to limit size of team. • Estimate that 32 reviewers is optimal.
Gen-ART: Reviewer Assignment • During the week IETF LC is announced a Gen-ART reviewer is assigned by the secretary. • Assignments are made on Thursday evenings CST. • No attempt is made to assign reviewers based on expertise: • This has been requested in the past and nacked. • Only reasons why the next reviewer (per round robin process) doesn’t get next doc is due to: • WG chair • Individual involvement as contributor, WG reviewer, etc. • Reviewer doesn’t have the bandwidth
Gen-ART: Reviewers’ reponsibilities • Reviews are due on or before the IETF LC deadline. • Reviews are always sent to Gen-ART mailing list and typically to author, WG chair and responsible AD. • May additionally be sent to IETF list. • Reviews are not typically sent to WG mailing lists. • However, it may be very, very useful for PROTOs/Authors to do so in some cases to ensure WG support for resolution of concerns. • Reviewer is responsible for re-reviews of any document updates, in particular when the doc comes up on the IESG telechat agenda.
Gen-ART Review Details • Reviews are preceded by the standard template which provides the recipients a link to the Gen-ART FAQ along with a simple statement as to the handling of the comments. • Review template contains a summary, which is some variation of the following: • This draft is ready for publication as a [type] RFC, where [type] is Informational, Experimental, etc. (In some cases, the review might recommend publication as a different [type] than requested by the author.) • This draft is basically ready for publication, but has nits that should be fixed before publication. • This draft is on the right track but has open issues, described in the review. • This draft has serious issues, described in the review, and needs to be rethought. • This draft has very fundamental issues, described in the review, and further work is not recommended. • Unfortunately, I don't have the expertise to review this draft.
Gen-ART Review Details • All substantive comments must be included in the public review. • Editorial nits may be identified along with other issues or they may be sent privately. • The review should apply generally agreed IETF criteria, such as: • [RFC1958] The Architectural Principles of the Internet • [RFC3426] General Architectural and Policy Considerations • [RFC3439] Some Internet Architectural Guidelines and Philosophy • [NITS] The "I-D Nits" document maintained by the IESG • [rfc2223bis] Instructions to RFC Authors • [BCP26, RFC5226] Guidelines for Writing an IANA Considerations Section in RFCs • [RFC3552] Guidelines for Writing RFC Text on Security Considerations
Handling of Gen-ART IETF-LC reviews • Handling of comments is always at the discretion of the AD/WG chairs/PROTO Shepherds. • In general, unless a document is deemed “Ready” (or “Ready with nits”), a response to the comments (typically by the author/editor) is sent to the original email recipients within a reasonable time period. • These email exchanges can be very important for resolving issues. • It’s often very helpful for author/reviewer to come to agreement on proposed changes for some of the more important concerns raised in the review.
Updating document in response to Gen-ART reviews • It is not recommended that a document be updated unless the revision is agreed with ADs/WG chairs/PROTO shepherd. • Document authors/editors must also consider other review teams that provide comments at IETF-LC time such as Sec-dir. • Very useful if a Gen-ART reviewer is able to review an intermediate version prior to the document being added to the telechat, however, this is entirely optional. • While reviews are non-binding, General Area Director does consider whether valid concerns have been adequately addressed.
Gen-ART reviews for other docs • Not all documents go through IETF-LC since it is optional for Informational documents and individual/AD sponsored documents. • Reviews at the time of telechat are done in a similar manner in terms of process. • However, some differences: • Shorter review period for reviewer. • Much less time for interaction between reviewer and AD/WG chairs/PROTO/author. • Reviewer involvement is more variable depending upon the AD’s consideration of the review and nature of review summary
Additional Gen-ART Info • Gen-ART process document: • http://ietf.org//draft-doria-gen-experience-02.txt • Gen-ART repository: • http://www.alvestrand.no/ietf/gen/reviews/ Note: plans to move this to IETF hosted server very soon. • Some files on a temp-dir at: www.softarmor.com/rai/temp-gen-art • Gen-ART mailing list: • NOT recommended that folks subscribe to this list. • List administrators (Mary Barnes and Brian Carpenter) add folks to the whitelist at time of first posting: • As of July 2008, around 313 folks on the whitelist (excluding ADs and Gen-ART members)
Moving Gen-ART Forward • Automating tools is a priority for the near term. • May leverage reviewer assignment tool used by secdir • Process may apply to other areas: • Modified forms of this process (in terms of “tools”) have been applied to other area review teams and WGs • Open Office spreadsheet is very handy: • Easy to sort information and gather statistics • Facilitates review information for General Area Director • Facilitates secretarial process in assigning, caching and summarizing reviews • Aids reviewers in determining outstanding reviews. • In general, reviewer assignment would likely be the primary common tool. • Other applications tend to require more manual intervention, however, the frequency of reviews is much lower.
Summary • While Gen-ART increases the quantity of review comments for documents during IETF Last Call, the overall improvement in quality makes the process worthwhile. • Document editors and WG chairs should always expect a Gen-ART review for their documents: • Info always available in the assignment spreadsheets. • Ping secretary if you don’t get one. • In the long run, tools should greatly facilitate the process and dissemination of information • Review team is very lean right now: • Could always use new reviewers. • WG chairs might consider and encourage conscientious WG reviewers/participants/leaders for the role. • Additional reviewers add minimal overhead to secretary’s job
Questions/Comments • Any general feedback or suggestions for improvements?
Additional Info on IETF LC Process • Good Overview/Reference from IETF-70: http://www.ietf.org/proceedings/07dec/slides/doclife-0.pdf • A few relevant slides included in the subsequent charts: • Shepherd’s Role: • < 50% of the docs seem to have PROTO shepherds versus AD’s . • PROTOs aren’t documented as part of IETF-LC process • IETF Process Overview • Handling of Review comments at LC time
