Best Practices for the Care and Feeding of a Program Committee

1. Best Practices for the Care and Feeding of a Program Committee, and Other Thoughts on Conference Organization Fred Douglis IBM T.J. Watson Research Center Best Practices

2. WOW … It’s WOWCS • First reaction: what a great idea! • Second reaction: will anyone actually submit? • Immediate fear: spend time writing something up, only to find out it’s been canceled • Final reaction: duty • I’ve chaired USENIX’98, USITS’99, and a few other conferences • Most went well but there were some issues • End goal: guidebook (wiki) for program/general chairs • https://wiki.usenix.org/bin/view/Main/Conference/CollectedWisdom Best Practices

3. Wiki • Goals • Living document • Allow discussion • Two types of info eventually • Guide for new program chairs or conference organizers • Review management software • Best practices • Ideas for new ways to do things • How well do rebuttals work? Ratings? Bidding? Best Practices

4. Problem 1: Bad Reviewers • Not everyone is a good PC member • Some never do anything • Some do really crummy reviews • Some are too argumentative, dominate the rest of the PC, or have other personality issues • How to find who is a known bad reviewer? • Past experience • Word of mouth… but people tend not to know to ask • How to expand word of mouth? • Reviewer database? Best Practices

5. Problem 2: Inexperienced Reviewers • Always want to bring some “new blood” into a PC • Search for repeat authors who haven’t served (need a script to scour Google Scholar for this) • Caveat: not everyone knows what to expect and may not be cut out for it. Don’t take too many people you don’t know if you can help it • Tips for ensuring the best PC • Set expectations early (# reviews, timing, … no surprises!) • Have multiple deadlines: force people to miss early rather than all at once just before the decisions • Help with calibration (average scores of reviewers compared to the scores of their peers on same papers) Best Practices

6. Problem 3: PC Composition • Avoid inbreeding, or the appearance of it • Overlap from year to year • Institutional overlap • There are enough people who will have published at an established conference that a chair should draw on them rather than outside the conference community • Best fit • Reward participation • Avoid problems with calibration Best Practices

7. PC Bells and Whistles • Options for running a PC • Rebuttals (Seem to be a nice idea, but add lead time) • Reviewer ratings (hard to calibrate; don’t want to insult) Best Practices

8. Running a Conference • Sponsorship • Some institutions make it a pleasure to run a conference, and some make it much harder • Doing it alone has risks (e.g., liability): who “owns” the conference risks? • Setting expectations • It’s hard to know just how many submissions a new conference will get: better to be swamped (and add PC members) than miss all your targets • Publicity is amazingly critical (who knew?) • Scheduling • Watch for conflicts and hope the others watch too • Consider the impact of rejected papers from one conference being routed to another. (Feed reviews?) Best Practices

9. Backup Best Practices

10. Problem 4: (Self)plagiarism • Simultaneous submission seems to be a big problem • Least publishable units • True plagiarism • We need a better mechanism for detecting • Serendipity doesn’t cut it Best Practices

11. Concrete Proposals/Challenges • Plagiarism • Neutral agent to collect papers and analyze for overlap • Get multiple organizations to buy in • Reviewer database • Neutral agent to collect feedback, and if someone gets multiple negative reports, somehow blacklist • See Internet Computing columns Sept-Oct/Nov-Dec 2007 Best Practices