1 / 12

Reviews

This review form from the ICML 2003 conference provides a framework for evaluating the scientific merit, validity, and relevance of research papers. It guides reviewers in providing constructive feedback to the authors and assessing the overall quality of the paper.

atracey
Download Presentation

Reviews

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ICML’2003 Minitutorial on Research, ‘Riting, and Reviews Reviews Rob Holte University of Alberta holte@cs.ualberta.ca

  2. Purpose • Evaluate the paper’s scientific merit • Check the validity of the paper’s claims and evidence • Judge the paper’s relevance and significance • Provide constructive feedback to the author

  3. Example: IJCAI’99 review form 1. How RELEVANT is this paper to researchers in the field? 2. How SIGNIFICANT is this paper? 3. How ORIGINAL is this paper? 4. Is this paper technically SOUND? 5. How well is this paper PRESENTED? Further comments, advice or explanations (Please be specific and constructive, especially with respect to any negative judgements above. Point to the section(s) where an error occurs, cite omitted references, etc.)

  4. Excerpts from ICML’03 form GOALS/MOTIVATION. Does the submission state the goals of the research, including the criteria by which readers should evaluate the results? Is the learning problem well-motivated? CLAIMS/EVIDENCE. Do the authors make explicit claims or draw clear conclusions, and do they present reasonable evidence to support their position? If the claims are theoretical, are the proofs correct? If the claims are empirical, are the experiments appropriate, and do the results reveal the underlying reasons or causes for phenomena? How might the authors strengthen their claims, evidence, or reasoning?

  5. Example: COLT’97 instructions FOUNDATIONAL/CONCEPTUAL CONTRIBUTION: Note things like a new model, new notion, new definition, new approach. Note the significance and reasons for this novelty (and note the absence of such a novelty). TECHNICAL DEVELOPMENT: - Introduces a new technique - Novel use of known technique - Talented use of known technique - Traditional use of known technique - Trivial technically

  6. COLT’97 continued RELATION TO OPEN PROBLEMS: Does the paper solve completely/partially/ or does it address an open question? How important is this question? (Central/ important/ interesting/ legitimate/ stupid). How much effort has been invested before in solving it and by whom?

  7. COLT’97 continued SOCIAL INTEREST IN PAPER: Is it potentially interesting to the whole COLT community, to a major subarea, to everyone in a restricted area, interesting only to the authors. HOW WILL IT CONTRIBUTE: Fertilization, satisfy curiosity, who knows? PAPER TYPE: Is it a - First step (opens a new area) - Last step (closes an important area) - Giant step (makes essential progress) - None of the above.

  8. Good Reviews • Polite • Fair • Concise • Clear • Constructive • Specific • Well-documented • Represent the scientific community

  9. Anonymity • Reviewers are supposed to be anonymous • BUT, sometimes their identity becomes known or is at least guessed by the authors • Write your reviews so that you would not be embarrassed if your identity was revealed

  10. “In my opinion…” • Reviews are necessarily subjective • A paper must convince its readers that its claims are valid and significant • A reviewer is more careful, thorough and patient than an average reader

  11. Reviews that “miss the point” • Not uncommon, very frustrating • Don’t blame the reviewer. Fix your paper.

  12. Conference reviews • short time frame • each reviewer has several papers to review • Expect reviews to be terse, less thorough, less satisfying

More Related