1 / 71

Class 4 09.12.2004

Proposal Writing & The Refereeing Process. Class 4 09.12.2004. The Refereeing Process. Introduction Peer review process Journals Conferences Research programmes The tasks of a referee Reviewing a research paper Preparing the referee report & recommendations

bryanne-kennedy
Download Presentation

Class 4 09.12.2004

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Proposal Writing & The Refereeing Process Class 4 09.12.2004 Class 4, 09/12/04

  2. The Refereeing Process Class 4, 09/12/04

  3. Introduction Peer review process Journals Conferences Research programmes The tasks of a referee Reviewing a research paper Preparing the referee report & recommendations Evaluating a research proposal Acting as an editor or program chairperson How to become a referee? Final words Contents Class 4, 09/12/04

  4. There is no fixed mechanism for refereeing There are simple rules that help transforming a review in a constructive document In time you will develop your own style of refereeing Disclaimer Class 4, 09/12/04

  5. A scientific paper is expected to provide a sufficient contribution to the knowledge base of its field Number of scientific papers and articles (2000): > 600 000 (ISI) About 50% in the fields of science and technology The number of papers and articles submitted for publication is much larger refereeing process selects the ones to be published Examples of acceptance rates after refereeing: Journals: ~10-20% (large variance) Conferences: ~10-50% Workshops: ~30%-90% Refereeing is also used in selecting research projects to be funded Introduction Class 4, 09/12/04

  6. What is a sufficient contribution? new result, theoretical or experimental new insight novel synthesis of ideas useful survey useful tutorial What is not a sufficient contribution badly written erroneous data MPI = Minimum Publishable Increment depends on the forum Introduction Class 4, 09/12/04

  7. Peer reviews are carried out by anonymous referees who evaluate the sufficiency of contribution novelty, significance, correctness, readability Refereeing is public service to the scientific community professional obligation, carried out on volunteer basis requires high expertice helps in improving one’s own expertice ensures the integrity of science Peer review process Class 4, 09/12/04

  8. Peer review process of a journal submission editor author publish accept reject revise selection of associate editor reviews recommendations associate editors referees selection of referees checking of revised papers Class 4, 09/12/04

  9. Peer review process of a conference submission program committee program chair author accept/ reject/ accept with revisions selection of the referees checking of revisions accept/reject/minor revision recommendations referees extra referees Class 4, 09/12/04

  10. Peer review process of a workshop submit program committee program chair author accept/ reject refereeing checking of revisions extra referees Class 4, 09/12/04

  11. Peer review process of a research programme submission steering committee proposer accept with partial funding/ reject referees Notice: not representative of all research programmes Class 4, 09/12/04

  12. The reviewer grades a paper based on its novelty, significance,correctness, and readability In case of substantial conflicts of interest or if the paper is out of the field of the reviewer, the editor must be informed promptly Both positive and negative findings are summarized in a referee report Confidential part only for the editor/program committee: Information that could reveal the identity of the reviewer or in minor conflicts of interest non-confidential part for the author/program committee Learn from the other reviews, if they are sent to you after the process The tasks of a referee Class 4, 09/12/04

  13. Several reasons Enhance reputation (with editor/prog. committee) Expedites processing of your own papers Get on editorial board or program committee Good practice Increase your own critical appraisal ability Your papers become better Sometimes it gets preferential treatment for your papers … but refereeing means more work! Why do it? Class 4, 09/12/04

  14. Most reviews have strict deadlines By agreeing to review you take the responsibility of doing a thorough job If you cannot commit to this, notify the editor asap Editors understand you may not have the time, but are unforgiving if you commit and do a poor job Good editors keep a list … Consideration Class 4, 09/12/04

  15. Humbleness and an open mind needed; 100% self-confidence can be harmful Early assumptions on the correctness of the paper or the sufficiency of its references should be avoided an elegantly written paper may have zero actual contribution a paper with broken English may contain a major new idea The papers recommended for acceptance should have novelty and be correct If the reviewer can’t check a fact or is unsure, this should be stated in the review report But don’t waste your time on analysing in detail a paper that is never publishable a single crucial error is enough The right attitude: I can learn something! Class 4, 09/12/04

  16. The paper to be reviewed is typically accompanied with a review form fill the five point scale questions last it is most important to write an itemized review report Relevance [ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent Originality [ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent Background knowledge of the subject and references [ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent Technical content [ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent Presentation [ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent Reviewing a research paper Class 4, 09/12/04

  17. The analysis of a paper can be done by generating explanations to the following eight points (Smith 1990) What is the purpose of the paper Is the problem clearly stated and have the key issues been pointed out? Is it clear what has been accomplished? Is the paper appropriate for the intended forum? If it is not, what could be a better choice? Is the goal significant = has the work been worth doing? Are the results just trivial variations or extensions of previous results? Are there any new ideas, or novelties in research methodology? Citation analysis using electronic libraries are a big help! Reviewing a research paper: analysis Class 4, 09/12/04

  18. Is the method of approach clear and valid? Is there something fundamentally flawed in the approach? Are the assumptions realistic and does that matter? Is the method new? Can it be generalized to other problems? Is the actual execution of the research correct? Are the mathematics and statistics correct? Check! Have the simulations been described in sufficient detail for replication? What about the boundary conditions? Do the results make sense? This part may require considerable effort from the reviewer... Reviewing a research paper: analysis (cont’d) Class 4, 09/12/04

  19. Are the conclusions correct? What are the applications or implications of the results and are the results analysed to an adequate depth? Is the presentation satisfactory? Is the paper readable? Is it structured according to the convenstions of scientific publications? What did you as the reviewer learn? If you didn’t learn anything, then the paper is not publishable(provided that you understood the paper) Reviewing a research paper: analysis (cont’d) Class 4, 09/12/04

  20. It is researcher’s professional obligation to cite prior work the manuscript being reviewed includes claims of novelties; regularly citing prior research the reviewer needs to check the validity of the claims most efficient to carry out the analysis using electronic libraries At minimum: Check what is found using the key words of the article Study the references you don’t know beforehand Check which recent papers cite the same references Check the references of those recent papers Reviewing a research paper: analyzing the references Class 4, 09/12/04

  21. Review structure • The actual refereeing form • General comments on the paper • Specific comments on the paper • Confidential note to editor • General idea: be professional and non-hostile: write the review in a style that you would like to receive for your paper Class 4, 09/12/04

  22. The refereeing form • Forms might look quite different but basically ask the same things • Poorly designed ones just have yes/no answers, good ones prompt the referee to elaborate • Make sure you read and understand it well Class 4, 09/12/04

  23. No fixed rules exist, the following ones are according to (Smith 1990) Most important: make your opinions clear; avoid ”perhaps” and ”maybe”; evaluate the paper, not the author; itemize the contributions State the recommendation and its justification; the five point scale part of the evaluation form is not enough Show with a few summarizing sentences that you have understood the paper. The editor may use this part and compare your summary to those of the other reviewers Evaluate the significance and validity of the research goal Evaluate the quality of methodology, techniques, accuracy and presentation; recommendations for revisions can be written here Make a clear recommendation for or against publication with justifications Writing the referee report Class 4, 09/12/04

  24. Compiling the recommendations • Classification of papers (Smith 1990) • Very significant; includes major results (<1% of all papers) • Interesting work, a good contribution (<10%) • Minor positive contribution (10-30%) • Elegant and technically correct, but useless • Neither elegant nor useful, but not wrong • Wrong and misleading • Unreadable, impossible to evaluate • The acceptance level of the journals and conferences vary; 1,2, and perhaps 3(-4) Class 4, 09/12/04

  25. Outcome • Usually: • Accept the paper as it is • Paper requires minor changes • Paper requires major changes (with or without a new refereeing process) • Reject publication of the paper • You can only suggest, the choice is not yours • Decision is based on at least 3 reviews Class 4, 09/12/04

  26. Research proposals • A research proposal is a request for funding submitted to, • MCyT, MECD, GENCAT • European Commission • NIH, NASA, NSF, ESF • other funding organization such as a foundation • The key difference to reviewing research papers is that • the reviewers also evaluate the proposers • Not all organization use peer review as a means for selecting proposals for funding Class 4, 09/12/04

  27. Evaluating research proposals • The evaluation criteria vary between funding organizations • Key criteria: • Is the research topic significant? • Are the goals realistic? • Has the proposer sufficient expertice and facilities to reach the goals? • Is the requested funding reasonable? Class 4, 09/12/04

  28. Objectivity Judge paper on its own merits Remove prejudice If you are not able to review it, return it Fairness Author may have different point of view / methodology / arguments Judge from their school of thought not yours Speed Be fast, but do not rush. Author deserves a fair hearing Ethics of refereeing Class 4, 09/12/04

  29. Professional treatment Act in the best interest of the author and conference/journal Specific rather than vague criticism Confidentiality Cannot circulate paper Cannot use without permission Conflict of interest Discuss with editor Ethics of refereeing Class 4, 09/12/04

  30. Honesty About your expertise and confidence in appraisal Courtesy Constructive criticism Non-inflammatory language Suggest improvements Ethics of refereeing Class 4, 09/12/04

  31. Acting as an editor or program chairperson • The editor • maintains correspondence with authors and referees • finds new referees if the ones assigned fail to act in given time • decides on acceptance, rejection or a revision round based on 2-4 review statements. • should distribute all review statements to the referees • receives occasional negative feedback • Review is not a vote! The editor is likely to line himself according to • the best justified recommendations • Conference program committees often rely on the numerical • evaluations, occasionally resorting to vote Class 4, 09/12/04

  32. How to become a referee • Writing a publication that is cited is the most certain way to become a referee • Coordination or technical coordination of an EU RTD project is a direct road to proposal evaluations • Refereeing is very rewarding, helps to keep up-to-date and aware of developments in fields adjacent to ones own specialty Class 4, 09/12/04

  33. Final words • Good referee reports are valuable and free of charge • help in improving the paper • help in improving as a researcher • help in improving as a referee • Refereeing is a learning experience • Scientific progress rests heavily on peer reviews Class 4, 09/12/04

  34. PROJECT THESIS EDITOR OFFICE Using the articles prepared based on ‘The Six Napoleons’, we will set up an editorial office Each student will act as an associate editor of one article and will review three articles (see handout) Each student will peer review his/her three allocated articles and will return the referee report (see handout) to the assigned associate editor - deadline 17th December The associate editor will compile the final report and will return the final report with the individual referee reports to the Editorial Office - deadline 24th December Tutorial group discussion in January! Class 4, 09/12/04

  35. Proposal Writing Class 4, 09/12/04

  36. In order to carry out research, in general financing is required. There are several national and international sources of funding and the process for obtaining funding is realised through proposal submission and review. The aim of this section is to INFORM you of the proposal process, proposal formats and existing funding bodies. The homework of this class will be to draft a proposal of your DEA/PhD to assist you in your resaerch planning, but NOT with a view to preparing a formal proposal in the style of those submitted for financing! Proposal Writing Class 4, 09/12/04

  37. Why research ? Why should this be in a competitive context ? Why a research proposal ? Getting started What makes a good proposal ? Writing your proposal How to structure your proposal ? The review process Allocation of funding What next ? Getting help with your proposal ? Quick TIPS for writing a good proposal Content Class 4, 09/12/04

  38. Why is the development of research within universities a must ? To maintain the quality of teaching programs. Provide the basis for undergraduate and graduate thesis research projects. Universities should be more than degree delivering institutions. Universities should be the basket for new knowledge and developments. Why research ? Class 4, 09/12/04

  39. Do universities have the financial capacity to develop and support research activities ? Where can the money be found to develop and support research ? How can the society gets the highest return on investment ? Why should this be in a competitive context ? Class 4, 09/12/04

  40. Convince others the project you have designed is important, worth the effort. Convince others that you have the ability to carry out the research design and report the findings. Generate funds to sustain the research units operation. Why a research proposal ? Class 4, 09/12/04

  41. Know your subject. The reviewers will look for an up-to-date knowledge of the research area. Know your funder. Be aware of the priorities and interests of the funder you approach, and know that funders are unlikely to support the same idea twice. Getting started Class 4, 09/12/04

  42. Consult colleagues. Don’t be afraid to discuss your proposal with colleagues, or even with the grants officer at the funding body. Early discussions can ensure that your proposal is targeted appropriately. Getting started Class 4, 09/12/04

  43. A well-prepared application should require minimal effort on the part of the reviewer. Proposals must demonstrate high scientific quality. The requested funds must be in proportion to the proposed project (cost-effectiveness). What makes a good proposal ? Class 4, 09/12/04

  44. Allow plenty of time to prepare your proposal. A good starting point is to write a one-page summary of the whole project. This may take a while to get right, but once completed it will serve as an invaluable tool for writing your full proposal. Use your proposal to show the need and then fill the gap. Writing your proposal Class 4, 09/12/04

  45. Present your proposal in terms of the aims and objectives of the funder and not just your own – make it clear how you will be helping them to fund their priorities. Consider the questions the funder will be asking: Why fund you ? Why fund this ? Why now ? ... and make sure that the proposal answers them! Writing your proposal Class 4, 09/12/04

  46. Be aware that you will have limited to none opportunities to answer queries arising from a reading of your proposal. Consult the funders website and read clearly the call for research proposals as well as the criteria against which your proposal will be judged. Writing your proposal Class 4, 09/12/04

  47. Although it is the content that matters, good presentation is often crucial to making your proposal accessible to reviewers and keeping their interest. Use diagrams and tables to add clarity; Bullet points and sections can break up text; Keep to page, word and font size restrictions; and Activate the spell checker while writing. Writing your proposal Class 4, 09/12/04

  48. Check guidelines carefully – failing to meet the funder’s format and specifications is one of the most common reasons for applications being returned. A common proposal structure normally consists of: title, abstract, background, aims and objectives, methodology, work program, resources, outcomes (outputs & dissemination), project management, reviewers. How to structure your proposal ? Class 4, 09/12/04

  49. Title: This is the first impression the reader gets. The title should be short and clear, and the reviewer should be able to understand from the title the intentions of the research. A catchy title posing a question or including an apparant contradiction or acronym may be more easily remembered by a reviewer. How to structure your proposal ? Class 4, 09/12/04

  50. Abstract: Should be a concise summary of the WHOLE project. Use the abstract to identify the need for this research, state what you intend to do, and how you intend to do it. Do not include unnecessary detail; make each phrase count. And remember it is the first impression a reviewer gets of an applicant’s worth! How to structure your proposal ? Class 4, 09/12/04

More Related