We toiled, we submitted, … we conq got rejected

1. We toiled, we submitted, …we conq got rejected Discussion on close rejects Saptarshi Ghosh CNeRG Retreat

2. Typical review process (SIGIR) • 3 reviewers review paper • Primary Area Chair discusses with reviewers, writes meta-review • Secondary Area Chair double-checks reviews, and may provide additional review • Area Chairs  PC chairs: Accept / Reject / Accept If Room • PC chairs rank papers by average score • Clear accepts, clear rejects identified • “Accept if room” papers discussed further

3. Methodology and Disclaimer • What I should have done, but did not • Did not read the papers • Did not find out about state-of-the-art • So, I believe what reviewers said • What I did • Read the reviews carefully, formed my own views • Discussed with 1st authors why they think paper got rejected • Views are not personal attacks on anyone

4. Rejects

5. Rejects turned into Accepts

6. SIGIR 2013 (5, 3, 3, 2) • Submission • Nested Query Segmentation for Information Retrieval • Rishiraj, Anusha Suresh, NG, MonojitChoudhury • Reasons for rejection • Dataset used not well-known, 2 reviewers advise TREC • Improvement in proposed method is very low • No comparison with method in [Metzler, Croft], “the most commonly used method to segment queries” • How important / necessary is nested query segmentation?

7. SIGIR 2013 (5, 3, 3, 2) • Scores in range 1 – 5, accept threshold: 3 • Scores from 3 reviewers, one meta-reviewer (last) • Relevance to SIGIR: 5 – 4 – 4 – 5 • Originality of Work: 4 – 4 – 4 – 4 • Technical Soundness: 4 – 4 – 2 – 2 • Quality of Presentation: 4 – 4 – 4 – 4 • Impact of Ideas or Results: 4 – 2 – 3 – 3 • Adequacy of Citations: 4 – 4 – 4 – 3 • Reproducibility of Methods: 3 – 4 – 3 – 3 • Overall Recommendation (1-6): 5-3-3-2 (meta-review)

8. WWW 2014 (-4, -2, -2) • Submission • Stay where you belong: on the permanence of vertices in network communities • Tanmoy, SriramSrinivasan, NG, AM, SanjuktaBhowmick • Reasons for rejection • Presentation: other community detection methods heavily criticized • Incomplete literature survey • Evaluation: compared local measure with other local measures, not to global measures like modularity

9. WWW 2014 (-4, -2, -2) • Reasons for rejection • Questions over basic approach of how metric is defined • Contribution not enough • Permeanence maximization yields a poor performance on the LFR benchmarks • Poor performance for mu=0.6 questions the whole usefulness of the measure. “Why would one need it if there are already better techniques?”

10. WSDM 2014 (-1, 2, -2) • Submission • Searching for Topical Content in Microblogs: On the Wisdom of Experts vs. Crowds • Bilal, Parantapa, NG, Saptarshi, Krishna Gummadi • Reasons for rejection • Motivation / story-line was not clear – reviewers did not realize solution to a new type of search was proposed • Evaluation – more quantitative results required • Writing / presentation was not good

11. INFOCOM 2014 (3, 3, 3, 1) • Submission • Segmented message broadcast in delay tolerant networks: An analytical and numerical study • Biswajit Paria, Rajib, NG, AM, Tyll Krueger • Reasons for rejection • Positioning of the work as a DTN paper was not clear • Justification for some technical design choices not given • Presentation not good – lot of missing information

12. PRE (rejected after editorial review) • Submission • Attack tolerance of correlated time-varying social networks with well-defined communities • Souvik, NG, AM • Reasons for rejection • “will consider only papers with significant and new results” • “your manuscript is a variant of existing work in the literature, displays predictable results, and lacks novelty”

13. Failure  Success: PRE • Submission • Coverage maximization under resource constraints using nonuniform proliferating random walk • Sudipta Saha, NG • Editorial review: not suitable for publication in PRE • Review by two Editorial Board Members • Statistics of random walks is a reasonable topic for PRE • Results are a rather small technical incrementaladvance with respect to the previous methods • Basic idea suitable, but awkward presentation of theoretical arguments, limited numerical experiments

14. Failure  Success: PRE • For first review • Addition of more results on some different types of graphs • Reorientation of the content • For second review • Added a small theory to explain the whole phenomena • Possibly the physics community is not as excited bydevelopment of an algorithm, as they are by a new theory or model which explains some phenomenon

15. Failure  Success: WWW • Submission at IMC, WSDM • Who let the spammers in? Analyzing the Vulnerability of the Twitter Social Network to Spammer Infiltration • Saptarshi, students at MPI, NG, Fabricio, Krishna • Rejected at IMC (3, 3, 2, 2, 2) • 3: Good paper: can accept, but will not champion it • 2: Weak paper: should reject, but not strongly against it • Rejected at WSDM: -1, 0, -2 • -1: weak reject, 0: borderline, -2: reject

16. Failure  Success: WWW • IMC and WSDM: most reviewer issues were on • Not enough done to identify spammers • Not much distinction between spammers and marketers • Study explains only a small fraction of spammers’ links • Observations are mostly obvious • Accept at WWW (12%): 2, 2, 0, 2 (meta) • Understanding and Combating Link Farming in the Twitter Social Network • Focused more on marketers than on spammers • Clearly differentiated between the two

17. Summary of reasons for rejection

18. Questions: Are we … • choosing the right journals / conferences in terms of scope? • addressing sufficiently important problems? • aiming too high for some projects without realistically estimating the novelty / contribution? • contributing sufficiently for the chosen problems? • doing sufficient literature survey? • comparing with state-of-the-art? • using acceptable evaluation methodologies / metrics? • thinking of alternative / counter arguments? • writing the paper well? • giving sufficient time to a project?

19. Possible solutions • Do a comprehensive literature survey ‘early’ • Establish that the problem is really important • Discuss works in progress with others • To know alternative points of view / positioning • Better to be grilled by peers than by reviewers • Use reading group

20. Thank You