INFO4990 Research Methods

1. INFO4990 Research Methods Bing Bing Zhou http://www.cs.usyd.edu.au/~info4990/ Lecture based in part on materials by Alan Fekete, Mary Lou Maher, Joseph Davis, Irena Koprinska and others Literature Review and Literature Search INFO4990 Research Methods, s2 2008

2. Outline • Literature review • why do you need to review the literature • goals of literature reviews • How to search • Finding relevant papers • Search practicalities • How to read the literature INFO4990 Research Methods, s2 2008

3. “If I have seen further, it is only because I have stood on the shoulders of giants”. Attributed to Sir Isaac Newton INFO4990 Research Methods, s2 2008

4. Why do you Need to Review the Literature? • To help you find a research question • What is the current state of the art? • Where are the gaps or the next steps? • To help you establish the importance of the research question • Why is this problem important and worth researching? • To establish your work as part of the knowledge base of your research community, e.g. • “[Cite] defined the following concept, about which we prove …” • “[Cite1, cite2, cite3] have all worked on systems like this” • “our performance is better than that of [Cite]” • To solve your problem • E.g. combining ideas from several papers INFO4990 Research Methods, s2 2008

5. Goals of the Literature Review • Demonstrate that you know the field • Your research question follows from the literature review • Justifies your research, provides the rationale for the research • how does your work connect to previous work • how does your work differ from previous work • Allows you to establish the conceptual framework and methodological focus INFO4990 Research Methods, s2 2008

6. Literature Review • Not just a summary of related papers but a critical examination of these papers • What are their strengths and limitations? • How exactly do they relate to your question? • Should be coherent and structured • Learning from examples • Ask your supervisor to suggest good previous thesis (Honours/Masters/ PhD) and research papers (journal/conference) • Many available on-line • Discuss the literature review section and the overall thesis/paper structure together! INFO4990 Research Methods, s2 2008

7. How to Find Relevant Papers • Papers given by your supervisor • In many cases your research question will be defined as modification/extension of existing work • Papers you found by searching and following up • Based on terms of interest • Based on links from other work you find • Reference chains • Papers listed as references in an important paper are also worth examining • Community links – important people, journals, conferences INFO4990 Research Methods, s2 2008

8. Communities of Interest • Your work should be targeted at a community • It must be explained in relation to work in that community • Find who are the important people in the community and where their work is published • Find their websites using Google • What else they have written? • Who are their students? • Do they continue to work on the same topic? • What are their current projects INFO4990 Research Methods, s2 2008

9. Searching for Papers - Some Practicalities • Given a reference to a paper, you need to get the actual content • Look online • most of the literature is available as pdf even very old papers have been scanned • ACM Digital Library – full access via the library’s home page • IEEE Digital Library – full access via the library’s home page • Publisher’s site, e.g. Springer, Elsevier, etc. – hundreds of journals accessible via the library’s home page • Google Scholar • Citeseer • Just using Google • with the author’s name and a few rare words from the title INFO4990 Research Methods, s2 2008

10. Effective Search • We are all used to Google-style search • List words of interest • Make sensible choices for your search • prefer words that are not very common in general English • not much point searching for “design” or “information” or “relation” but “congestion” and “heterogeneity” will work • often add a term for the field as a whole, e.g. “database” or “network” • Start with a few targeted words and add more if too much junk is returned INFO4990 Research Methods, s2 2008

11. Effective Search (2) • Most search tools offer a variety of features, especially important when words of interest are common • http://www.google.com.au/intl/en/help/basics.html • http://www.google.com.au/intl/en/help/refinesearch.html • AND (implicit in Google) • OR • Exact phrase match (Goggle: “words in exact order”) • Allow synonyms (Google: ~tern) • Exclude terms (Google: -term) - useful for terms with more than one meaning. • Include common terms (Google: +term) – common terms are excluded by default, e.g. “how’, where’ INFO4990 Research Methods, s2 2008

12. Effective Search (3) • Topic-Specific Searches in Google • Google Book Search - Search the full text of books • Google Code Search - Search public source code • Google Scholar - Search scholarly papersGoogle News archive search - Search historical news INFO4990 Research Methods, s2 2008

13. How to Reference Web Documents • Some publications exist only on the web • No formal journal or conference publication • May be TR • Reference them by giving the URL used and the date on which it was accessed • web information changes • Also, be careful when getting a version from the author’s web page as it may be posted in preliminary not final version INFO4990 Research Methods, s2 2008

14. Guide to Research Literature • Types of publications • conference and workshop papers • journal papers • technical reports • monographs INFO4990 Research Methods, s2 2008

15. Conference Papers • Call for papers - ~1 year before meeting • Paper submission - ~4-8 months before meeting • Page limit e.g. 8 pages • Details often omitted (proofs, design technicalities) • Program Committee reviews the papers • Criteria: relevance, significance, originality, soundness, readability • Final version for proceedings due ~3 months before meeting • revise by author in light of reviews • but not checked again • Annual or bi-annual conferences INFO4990 Research Methods, s2 2008

16. Selection Process • Typically 3 reviewers • Acceptance rate – varies • Some 10-15%, others 50% • Some review “blind” (author details not shown to reviewers), others do not • Ask your supervisor for guidance about which are the reliable and important conferences in your field! INFO4990 Research Methods, s2 2008

17. Workshop Papers • A workshop is typically a smaller meeting than a conference • Sometimes workshop papers are just like conference papers • Other workshops are more preliminary • can publish a position paper (draft of an idea without evidence, or proposal for future work) • less rigorously reviewed, the goal is mainly to allow the community to meet INFO4990 Research Methods, s2 2008

18. Journal Article • Typically longer than a conference paper • Often based on a conference paper with additions, corrections and improvements • Refereed by • at least 3 reviewers, experts in the field • they spend months on the paper checking details, etc. • Decisions: accepted, accepted with minor revisions, major revisions and resubmission, rejected • Revisions, refereed again • Accepted, published after several months (journal issues have limited capacity) • Time from submission to publication varies, typically 1-1.5 years but may be 3-4 years INFO4990 Research Methods, s2 2008

19. Standard of Journals • Many journals in each area with different standards • Typically IEEE Transactions and ACM Communications are some of the top-ranked journals • Not all IEEE Trans. and ACM Comm. are top journals • Not all papers published in top journals are good papers • Ask you supervisor which journals are the top-ranked and most important in your area! INFO4990 Research Methods, s2 2008

20. Technical Report • Issued by the author’s department, with a number and date • May be based on a conference paper • Longer, includes all the boring details that are omitted from the conference paper due to space limitations • Used to establish priority • E.g. produce TR before submitting to conference or journal + conference and journal papers may get rejected INFO4990 Research Methods, s2 2008

21. PhD or MSc Thesis • Very extensive account • Show much of the research process • Extensive survey of the literature • Very complete evaluation of the work • The goal is to establish that the author is ready to become independent researcher • i.e. PhD and MSc provide research training • Typically checked by 2 or 3 reviewers INFO4990 Research Methods, s2 2008

22. Monograph • A collection of selected papers from a conference or workshop • A bit more checking than for the conference/workshop • An author can offer a coherent and unified account of a whole research topic • often combines their own results with other people’s • Revisits several papers using unified notation, better exposition, better literature review, etc. • Publisher may get reviewers but their focus is “will it sell” not “is it correct” INFO4990 Research Methods, s2 2008

23. Warnings • There's a whole bunch of conferences/journals in your community. • You could spend all your time reading them. Fortunately, only a few are worth looking at. • Quality of conferences and journals varies, and this is reflected in the checking of the papers • Read papers with a critical eye! • Some communities are very clique-dominated • Unpopular opinions are not welcome • Clique leaders can publish anything, even half-baked ideas without evidence INFO4990 Research Methods, s2 2008

24. Fake Conferences and Random Papers • http://pdos.csail.mit.edu/scigen/ • A random paper accepted to a journal? INFO4990 Research Methods, s2 2008

25. The Research Community • A community has conferences and journals of high prestige which they read and publish in • They meet often, and each knows (more or less) what others are doing • You must place your work in the context of a community INFO4990 Research Methods, s2 2008

26. Quality Metrics • How important is an article? How influential is an author? • Based on citation analysis - number of times a paper or author is cited • How to calculate citations – Google Scholar + other software • Assumption: important authors and articles are cited more often than the others • Increasingly used by governments, funding bodies, promotion committees to evaluate the quality of author’s work • Some drawbacks • Citing errors – authors with the same names are not separated • Cliques (friends, colleagues) cite each other in turn to build their citation index • Negative citations are included (citations to incorrect results) INFO4990 Research Methods, s2 2008

27. ISI Citation Database • Very popular, established in 1960, contains >40million records, contains • Arts and Humanities Citation Index (A&HCI) • Science Citation Index (SCI) • Social Sciences Citation Index (SSCI) • However • it doesn’t index a large number of journals • ignores open-access journals • doesn’t index conferences “Read the Rise and Rise of Citation Analysis” by L. Meho! INFO4990 Research Methods, s2 2008

28. Journal’s Impact Factor • Journal impact factors • Used to determine the importance of a journal • E.g. journal impact factor for 2007 = # citations in 2007 to articles published in the journal in 2005-6 = ------------------------------------------------------------------------------- # articles published in the journal in 2005-6 INFO4990 Research Methods, s2 2008

29. CORE’s ratings • Computing Research and Education Association of Australasia (CORE) • Australia and New Zealand • Ranking of journals and conferences in CS – not finalised http://www.core.edu.au/ INFO4990 Research Methods, s2 2008

30. How to Read the Literature • Reading papers is a skill that takes practice. • You can't afford to read in full all the papers that come to you. • Normally there are three phases to reading one. • The first is to see if there's anything of interest in it at all. Abstracts, the table of contents, conclusion section, and introduction are good places to look. • Once you've figured out what in general the paper is about and what the claimed contribution is, you can decide whether or not to go on to the second phase, which is to find the part of the paper that has the good stuff. • Finally, you may go back and read the whole paper through if it seems worthwhile. INFO4990 Research Methods, s2 2008

31. How to Read the Literature(2) • With a critical eye • Take notes • What is the contribution? • What is the evidence? What is the research method used? • How does this contribution relate to the previous work? • What are the important references cited in the paper? (follow them up) • Do you have any doubts, questions, ideas? • Keep reading the important journals and conference papers in your field • IT and Eng research ages quickly • Maintain annotated bibliography from the start INFO4990 Research Methods, s2 2008

32. How to Read the Literature(3) • Don’t rely on second hand summaries! Go to the original source if you can! • Get attributions right in your own writing: don’t just accept citations from other work, even with full reference! • When reading a well written paper • What makes this paper easy to read? • What level of detail is provided? • What examples are used to demonstrate important concepts? • What questions are left unanswered (future work)? INFO4990 Research Methods, s2 2008

33. Organising the Literature • Isolate issues and highlight the findings and contributions that are central to your research • Group together papers that deal with a common or related theme or issue • Organise the materials with diagrams, tables, concept maps • Try out different structures for organising; they should be most relevant to the goals of your research • Chronological order is not particularly useful but citation chains are useful • Warning • papers often don’t use common terminology, or focus on common issues, or explain relationships fairly, clarifying these aspects is a key contribution you can make INFO4990 Research Methods, s2 2008