Project2 Post Analysis —General Things

1. Project2 Post Analysis—General Things • Reviewing is about voicing your opinion about the paper! Reviews that do not express any opinion—except for their summarization value—are useless. • However, if you make evaluations you should do your best to justify your evaluation; here are some bad examples: • “the paper fails to review important related work.” instead of specifically listing what important papers were not referenced! • “the paper experimental evaluation is not convincing” without saying why. • “the paper fails to evaluate alternative approaches” without saying which approaches should be evaluated. • You should not be afraid to mention difficulties you had in reading understanding the paper! • A lot of reviews were incomplete.

2. Project2 Post Analysis—Paper Summaries • The paper summary should briefly introduce the research area of the paper and then make a clear statement about the paper’s contribution and why the author believes this paper improves the state of the art. • Paper summaries should be stand alone and readers should be able to understand those without reading the paper itself. • 50% of the individual paper summaries are very short, not specific enough, and about 40% are not easy to comprehend. • Paper summary and paper evaluation should be clearly separated. • Paper summaries should be specific and should not lose themselves in generalities. Do not use sentences like that “The paper introduces a novel data mining technique and applies it to the challenging application”. • Do not introduce terms without explaining those e.g. • “…introduces a cocktail approach…” • “…uses a hybrid approach…”—explain what approaches are combined

3. Project2 Post Analysis—Paper Evaluation • Most reviews did a poor job with assessing the novelty of the paper, as they were lacking a clear statement about which extend a paper improves the state of the art, if it creates new bridge between research fields and extends the research field’s knowledge base. • Reviews should be clearly structured; some students did not use subsections and as their writing is not always the best, it was often not clear what a particular paragraph is currently assessing; e.g. is this about about novelty or technical. • Voice your own opinion, but do not discriminate: • “…One author is a professor in the MIT Math Department. So it’s unnecessary to challenge all the formulas.” • “…The references cited in the paper are a bit old” • Comment mainly on the specific paper and not the research field in general! • Education impact: many reviews fail to make a clear statement about what “you have learnt from reading the paper” and what other groups could learn from it.

4. Project2 Post Analysis—Other Issues • Websearch and in general: Have a reference list at the end of your review; also if you found other relevant papers, explicitly mention if they are not referenced in the paper. • Most reviews did a poor job in summarizing websearch results and other did not do much websearch • Some students are confused what “broader impact” is all about. • Avoid writing trivialities: • “The author has acknowledged what he has accomplished.” • “The author summarized the paper’s accomplishment in the paper’s abstract” • Do not paste large portions of the paper into your review; summarize it instead. • Do not trust anything the authors claim to be the case! • Look for inconsistencies which point to the fact that there is something fundamentally wrong with the paper.

5. Project2 Post Analysis—Other Issues • Some reviews do not follow the suggested template! • Some individual reviews are very hard to understand; the writing of the group reviews was somewhat better • Specific Comments, centering on • Any inconsistencies • Parts you did not understand • Parts were you disagree with the authors’ conclusion or assessment • Errors and things that are missing • Asking authors specific questions • Making suggestions on how to improve the paper • are an important part of a good review. • Moreover, assess the paper from the a more emotional point of view; e.g. does the paper does a good job to get people excited about the research field and the paper itself. • Finally, consider the paper’s entertainment valueand if it is high, add a positive paragraph to your review.