How to get the best paper award

1. How to get the best paper award Hans-Dieter Burkhard Humboldt University Berlin

2. … I don’t know (but it happens)

3. What is a good paper? Would you read it? Why: • Interesting topics • Surprising results • Nice story What is the message? What to learn from the paper?

4. What is a good paper? Examples: • New efficient quality measure in SE. • New facts about neural control in animals. • A first correctness proof. …

5. What is a good paper? Who decides: • Reviewers of journals/conferences • Readers (citations of the paper)

6. What is a good paper? Typical criteria of reviewers: Contents • Contribution • Originality/novelty • Significance to theory and practice • Relevance to call of papers/to journal

7. What is a good paper? Typical criteria of reviewers: Technical aspects • Technically sound • Readability and organization • Appropriate length of the paper • Content style and clarity of writing • Quality of English

8. Highly ranked papers • Articles in main journals • (what are the most important journals in your research field?) • Contributions in main international conferences • (what are the most important conferences in your research field?) • What is the acceptance rate

9. Acceptance rate From a mail by the program chairs: We have almost completed the decision phase for the 531 papers that we received. We have followed the recommendations in 96% of the cases. There have been 21 papers out of 531 decisions where we changed between reject – poster or poster - accept compared to recommendations we received. In each case, it was done only after reading all reviews, the discussion, and the paper itself … • Accepted: 119 (22%) • Posters: 133 (25%) • Rejected: 277 (52%) • Transferred to Industrial Track: 2

10. Highly ranked papers need time • Leading researchers can produce 1-2 high quality papers per year • Result of work in their group • Result of discussions with others

11. Other papers • Other (peer reviewed!) journals • Other (peer reviewed!) international conferences • National journals • National conferences • Workshops There are so many conferences/journals Quality Check: Which kind of proceedings (e.g. by Springer)

12. Why workshop papers? Workshop papers are not highly ranked. But: • Don’t start publication at the highest level • Early presentation of your work • Discussion: critics and information by others (hopefully by reviewers)

13. What about books and book articles? Large differences in ranking: Fundamental book (cited by everyone)… …yet another textbook (among 100 others) (What are the standard books in your field?)

14. Organization of conference reviews Program Chairs invite members of the Program Committee (PC)

15. Dear Anton, The 5th International Conference on Multi-Agent Systems will be held in Leipzig (Germany), the 25-27 September 2007. … CEEMAS conferences (http://www.ceemas.org/) take place every second year in the Central and Eastern European region. The programme committee of the conference series consists of well known researchers from the region and renowned international colleagues. The conference proceedings are planned to be published by Springer Verlag in the LNAI series as before. In view of your expertise on the field, the Chairs would like to invite you to take part in the Program Committee of the conference. In addition to the task of reviewing a few papers in April/May 2007, your duties as a PC member will be to actively promote the workshop and to solicit paper submissions from colleagues. Please let us know if you are able, as we hope, to accept this invitation. Best regards … Organization of conference reviews Chairs invite Program Committee (PC)

16. Thank for the invitation; unfortunately I am totally overloaded right now and so much decline. I will be honored to join the PC as it was the case for the previous ones. Please count on me. Thank you for the invite. I will be happy to help out on the PC. Sure! What a pleasure to join you, my dear friends.

17. Organization: ConfMaster System • Web based system (same as for authors) • Authors upload their papers • Reviewers can download assigned papers • Each paper is usually assigned to 3 PC members • PC members usually get 4-10 papers to review • PC members ask their colleagues, assistants … (look for the list of reviewers in the proceedings …)

18. How papers are assigned to reviewers • By Chairs only. • By competencies: • PC members indicate their interests/expertise by given keywords • By bidding for papers: • PC members indicate their interests/expertise based on paper abstracts

19. Format of reviews • Fixed Attributes • Ranking, e.g. between 1 … 5 (best) Example: Evaluation of work and contribution : 4 Significance to theory and practice : 4 Originality novelty : 3 Relevance to the call of papers : 3 Readability and organization : 4 Overall recommendation : 4 Reviewer familiarity : 4 The important one

20. Format of reviews • Additional text fields -- Comments to the author(s): I was trying to find why it is so important to build coalitions and in what dynamic environments you are trying to find them. Is it for a plant? What are the application fields of your coalition formation? What is an example for a noisy environment? -- Comments to the PC: -- Summary:

21. Remarks of the reviewers General remarks Generally, I don’t understand what the intention of the authors is. Why do they call it data mining if, as I understand, it ends up as a specific way of searching? Also, the paper does not mention any related work, most notably from the case-based reasoning area where applications of that kind have been developed more than 10 years ago. Contributions of the paper … (here you can read what the reviewer has understood)… What are the ways in which the paper should be improved? Firstly, name the problem precisely. Do you address search or data mining? If the latter, what is the result of the data mining process?Investigate related work and then describe what really is unique and new.

22. Remarks of the reviewers The idea of using relative information between objects/landmarks is not new. At least, it has been used in SLAM (Simultaneous Localization and Mapping) for years. See the line of research on Relative Mapping (e.g. [Smith et al., 1990], … ). … In the paper, it does not describe if the observations are communicated to ALL robots (in the team). If it is the case, would it be better to have a central control where the observation are collected/computed and results send to all robots ? Detailed comments are as following: "Line point" is mentioned many times in the paper. I do not understand. It is true that the robot observes line segments ? I do not understand the figures 4a. Please explain in more details. The figure 5b is too small to see. Please make it larger and explain in a better way. Please clarify the last paragraph of page 3, column 1. …

23. Final decision • By Chairs only (mainly based on overall evaluation) • By discussion between reviewers (try to find consensus) Overall evaluation 1 – strong reject 2 – weak reject 3 – balanced 4 – weak accept 5 – strong accept

24. Tight schedule (!) for conferences • Bidding Period: January 22 to February 1PC Members/Reviewers notified of their allocation: 5 FebruaryDue dates for referee reports: March 2Consensus Discussions: March 2 to 12Author notification: March 23 (leaves some time for preparing final version)

25. Best paper award • Candidates: • Best ranked papers • Proposals of PC members • Decision (vote) by all PC members Inofficial rule: The next year award is often not given to same topic, country, university,… Award is subject to “scientific/political” goals …

26. What you can learn by reviews • What do others think about your work • References to other interesting work • Open questions in your paper • Presentation problems • Scientific problems

27. Writing papers You should already have some results … • The good case: Problems become more clear while writing • The bad case: You still don’t understand what you are writing NEEDS TIME!!! • Write it two times (with a delay of 4 weeks or more)

28. Ø quality # papers Time What are interesting topics? • Exploration of new fields (may arise by practical questions) • Main stream? Private research field? (who reads your paper)

29. What are interesting topics? • Improvements of other work: • New insights • Better estimations, better experiments, … • Fewer preconditions … • Corrections … • Interdisciplinary: Go to other communities Why should someone read my paper?

30. What a paper should contain What people (reviewers) read first: • Abstract, introduction, conclusion, references • Figures (readability!), tables They want to get an idea: What are the main contributions? For their later reconsideration (for fixing ranking): What were the main contributions?

31. What a paper should contain • Good structure • outline/explanation in the introduction • Clear (formal) Definitions and Results (!) • If appropriate: Evaluation by experiments • with discussion! • Related work – what is new in your paper • Open problems • Don’t forget to give the credits to other work

32. Credits, Acknowledgements, Citations • Strict honesty with regard to contributions of partners, competitors and predecessors

33. How to find relevant papers • Libraries • References in other papers • Citation indices: Links to citations of a paper • Citeseer • Google Scholar

38. Use of citation indices • Investigation of more recent work (citing articles are newer than the cited article) • Relevance of work (count of citations) • Additionally: Who reads your papers?

39. Technical aspects • Given formats: • paper size • pages • citation styles • Works best with LaTeX • Use the correct terminology … • Include a nice story

40. I. INTRODUCTION The classic example of negative information was described in the Sherlock Holmes case “Silver Blaze.” In this case, a house has been broken into. Under such circumstances, one would expect the watch-dog to bark. The curious incident of the non-barking of the dog in the nighttime provides Holmes with the information that the dog must know the burglar, allowing him to solve the case. Applied to mobile robot localization, this means that conclusions can be drawn from expected but actually missing sensor measurements [5].

41. Open problems … … will be left for future research … What can it mean to the reader: • You didn’t like to work more on it? Reviewer answers (hopefully not): • Present the paper if you are finished. But of course: Be honest with respect to unsolved questions and further plans. • Why is it unsolved?

42. Co-authorship • Cooperative work can help a lot. • Authorship: • all cooperating people • order by (inverse) alphabet or better by contribution (clarify role/contribution of co-authors) • Co-Authorship by recognized people can help to be accepted • Inclusion of the professor (if she/he agrees)?

43. How to be visible by others • Be present on the Web • provide papers for down load • Use appropriate key words • Discuss with others on the conferences • Join scientific networks!

44. Write more papers Why? • want to have larger publication list • institution pays travel expenses • Consistently question your own findings • Co-authoring • Reuse of papers … ? cut-and-paste-papers …?

45. If the paper was rejected Consider it as in sports: • a step to future success. • referees are also humans. • learn from comments (hopefully they explain the reasons).

46. If the paper was accepted • Final version: Follow the hints of reviewers. • Journals: further review after revision. • Conferences: no time for further review. • Exception: after conference proceedings • Technical aspects: • No problem with LaTeX styles.

47. Presentation at conference • Keep the time Prepare parts (proofs, remarks, explanations, slides, …) near the end that you can easily • add for remaining time • skip for missing time • Preparation: Test the talk with clock and audience (with audience it will take more time than expected)

48. Presentation at conference Technical aspects • Common Fonts, not less than 24 points • Color consider physiological aspects and: projector may change color • Sufficient size of figures (readable inscriptions!) • Parts of programs (implementation details) are usually not interesting and not readable Canyou read ? red on blue ? yellow on white?

49. Presentation at conference Get the interest of the audience by … (a lot of papers are presented!) if possible: recall other talks from the session Discussion: • Source of new questions for future work • If you are lucky: You will be invited for common work

50. Poster-Presentation at conferences • Highly competitive • Try to get attention: • (provocative) thesis • interesting picture • Stay at your poster for explanation/discussion • Keep ready: business card and printed papers (if available)