Handling the Literature

1. Handling the Literature Prof Carole Goble carole.goble@manchester.ac.uk COMP80122 28 January 2015

2. You will read a lot of papers Most will be irrelevant Many will be poor Some will be important A few will change your life You will find lots of papers or none Find, Manage, Read, Synthesize ….

3. Defendresults are plausible or correct and method convincing and repeatable. Review & LearnVerify the results empirically. Trust. Understand. Convince, comfort, credibility. ReuseUse the explained and trusted results (data, method) for new / my science on demand. Compare. Extend. Purpose: Announce and Convince Is it “true”? Can I repeat it? Am I convinced? Is it plausible? Can I reproduce it? Can I use it? Is it a useful contribution?

4. Scholarly Communication Forms • Making an impact • Demo, Magazine articles: reviewed • Blogs, twitter, forums: unreviewed • Technical reports • Proposing an idea or view • Position statement, Commentary, Perspectives, Magazine Department, Doctoral Consortiums • Highly cited, editorialised, low rigour, established figures • Presenting a preliminary research finding, on-going work, ideas, small extensions to existing work, • Short paper, workshop paper, poster • Medium rigour, peer review

5. Scholarly Communication Forms • Presenting a research finding • National Conferences: • new ideas/applications/tools, medium extensions, more serious reviewing • International Conferences: • mature work, serious reviewing, but time-constrained, check track • Journal article: • lots of mature work (e.g., 2 conference papers into 1 journal paper), serious reviewing, not time-constrained • High rigour, peer-review

6. Additional material • Conference papers means a presentation • Slides, videos • Web pages • Blogs • Technical reports • Other?

7. More forms • Position paper • Systems paper • Theory paper • Vision paper • End-to-End paper • Surveys papers • Summary papers • … • Deep and narrow • Broad and shallow

8. Salami paper writing Challenge: rebuilding a body of work

9. Finding and choosing papers • Key players and key papers everyone cites

10. Finding and choosing papers • Special repositories / libraries

11. Lab Note books • Use one. A book. Or electronic. • A wiki? A Blog? http://www.atriumresearch.com/html/eln.htm

12. Finding and choosing papers • Citation management

13. Managing your references • Use a reference management system http://en.wikipedia.org/wiki/Comparison_of_reference_management_software

14. Places to find tools https://www.force11.org/catalog

15. Utopia Documents: http://getutopia.com

16. Kristian Garzia PhD student speaks about tools

17. I can’t find any papers! • Adjacent fields? • Different terminology? • Related topics?

18. Synthesize: beyond Shopping Lists • Annotated Bibliography • Literature review framework • Categories • Clusters • Timelines • Comparisons on aspects • Cross cutting the papers

19. Mind-mapping tools can help • http://cmap.ihmc.us/ • http://www.mindjet.com/uk/mindmanager/ • http://freemind.sourceforge.net/wiki/index.php/Main_Page • http://www.biggerplate.com/ http://www.biggerplate.com/mindmaps/NwUuYpZx/critical-literature-review-template

20. Smart Reading • Understand the context of the paper. • Beginnings and endings. • Survey the structure. • Use figures and tables. Generate them if absent. • Decide when to read every word. • Summarise the paper. • Read out loud. • Explain the paper to your cat. • Multiple reading passes. Over time. • Set aside time to think about it and digest it. • Get help. Talk to people. Set up a reading group.

21. Where does it fit with my work • Is it relevant? If not why not? • How does it fit with your framework? • Yes – you will need a framework! • Can you relate the terminology and notation to yours? • Keeping a record of the contribution.

22. When will the paper become relevant? • Over time • An ongoing framework • Revisiting

23. Hints for Reviewing Papers • The answer to each question tells you something about the technical content of the paper • The ease of extracting the answer to each question tells you something about the quality of the writing. Questions • Is this a vision/position/direction paper, or a measurement/implementation paper? • If you know the area well, can you mentally slot this paper somewhere in the taxonomy? ("Differs from X as follows; has the following in common with Y;" etc.)  If the paper is radically brilliant, new, or iconoclastic work, this question may not apply. • Can you summarize the single most important contribution in one or two sentences? Issues • Will this advance the state of the art? • Did you learn anything new? • Does it provide evidence which supports/contradicts hypotheses? • Experimental validation? • How readable is the paper? • Is the paper relevant to a broader community? Goals of Review • Guide the program committee in selection process • Help authors (to revise paper for acceptance, to understand rejection, to improve further research and future projects) John Ousterhout's Hints for Reviewing Papers

24. Make yourself a template • Author housekeeping stuff • Paper genre • Problem statement/motivation • Key ideas • Technical contribution • Technical flaws • Evaluation • Presentation • Comparison • To authors’ other work • To third party’s work • To your work • How would I extend this paper? • What questions does it raise? • Future work of author. • What else? • Author log

25. Make yourself a template • Author housekeeping stuff • Paper genre • Problem statement/motivation • Key ideas • Technical contribution • Technical flaws • Evaluation • Presentation • Comparison • To authors’ other work • To third party’s work • To your work • How would I extend this paper? • What questions does it raise? • Future work of author. • What else? • Author log Well-established class of problems, e.g., FO theorem proving, image retrieval etc. Novel class of problems (is it really new?) Single problem or many problems Implicit or explicit new way of thinking about a problem?

26. Make yourself a template • Author housekeeping stuff • Paper genre • Problem statement/motivation • Key ideas • Technical contribution • Technical flaws • Evaluation • Presentation • Comparison • To authors’ other work • To third party’s work • To your work • How would I extend this paper? • What questions does it raise? • Future work of author. • What else? • Author log Implicit or explicit new way of doing things? New, i.e., developed by authors? existing? Developed by authors or others? New combination of existing techniques? Good or better/worse than X and why

27. Make yourself a template • Author housekeeping stuff • Paper genre • Problem statement/motivation • Key ideas • Technical contribution • Technical flaws • Evaluation • Presentation • Comparison • To authors’ other work • To third party’s work • To your work • How would I extend this paper? • What questions does it raise? • Future work of author. • What else? • Author log What is the author's thesis? What are they trying to convince you of? Summarize the author's argument. How does the author go about trying to convince you of the thesis?

28. Make yourself a template • Author housekeeping stuff • Paper genre • Problem statement/motivation • Key ideas • Technical contribution • Technical flaws • Evaluation • Presentation • Comparison • To authors’ other work • To third party’s work • To your work • How would I extend this paper? • What questions does it raise? • Future work of author. • What else? • Author log Does the author describe other work in the field? If so, how does the research described in the paper differ from the other work?

29. Make yourself a template • Author housekeeping stuff • Paper genre • Problem statement/motivation • Key ideas • Technical contribution • Technical flaws • Evaluation • Presentation • Comparison • To authors’ other work • To third party’s work • To your work • How would I extend this paper? • What questions does it raise? • Future work of author. • What else? • Author log Empirical (run tests): test suite and testing must match problem targeted Theoretical: correct and understandable/convincing and relevant Does the paper succeed? Are you convinced of the thesis by the time that you have finished reading the paper?

30. Make yourself a template • Author housekeeping stuff • Paper genre • Problem statement/motivation • Key ideas • Technical contribution • Technical flaws • Evaluation • Presentation • Comparison • To authors’ other work • To third party’s work • To your work • How would I extend this paper? • What questions does it raise? • Future work of author. • What else? • Author log Does the author indicate how the work should be followed up on? Does the paper generate new ideas?

31. What are your tips? Reading groups Printing and physical markup Not printing and electronic markup Read everything 3 times