Research Methods: Introduction

1. Research Methods:Introduction James Gain jgain@cs.uct.ac.za

2. What is Computer Science? • Origins • mathematics, • engineering, and • commercial practice. • Evolved into • theoretical, • experimental and • design (or user) orientated aspects. • balance and synthesize these aspects

3. Research Tools • Theory • Abstraction (= experimentation) • Design Experimentation Theory (Maths) Design CS

4. What is Research? • New Stuff? • What sort of stuff? • Only for academics? • What is good research? • Who makes a good researcher?

5. Introduction to CS Research • A research lifecycle • Flavours of research • Ethics in research • Pitfalls • Getting started • Research proposals • Literature reviews • Presentations • MSc & PhD work

6. The Research Lifecycle Problem Identified • Research Activities: • Literature Search (survey previous work) • Do the Work (elaborate ideas and get results) • Write Up (plan and write a draft) • 3! = 6 orderings Research Activities Solution Adopted

7. Is Research a Linear Progression? • Progress? • Circular? • Evolutionary?

8. Orders: The Classics • LIT  WORK  WRITE • Don’t know when to stop the literature search • Can inhibit innovation • WORK  LIT  WRITE • Might get a nasty shock (someone else beat you to it) • CONCLUSION: safe options

9. Orders: quick and dirty • WORK  WRITE  [LIT] • Gamble that the referees (examiners) know less than you • WRITE  WORK  [LIT] • Suits speculative work • Sometimes used to drag the referees in as unwitting collaborators • CONCLUSION: living on the edge

10. Orders: paperchasers • WRITE  LIT  WORK • Writing serves as a plan of work focused to produce a single publication • LIT  WRITE  WORK • Good for an idea out of your normal line of research • CONCLUSION: unconventional

11. Flavours of Research • Theoretical • Develop new theories • Engineering • Develop better mechanisms to improve current practice • Experimental • Evaluate a theory/mechanism – usually via implementation and testing • Can include human factors • Some research projects cover the spectrum

12. Research Outputs • Theoretical • Theorems • Models • Analyses of existing research • Engineering • System architectures & prototypes • Code libraries • Knowledge bases • Ontologies • Hardware specifications

13. Experimental Research Outputs • Human factors • Surveys (questionnaires, interviews) • Experimental results (lab tests, field studies, case studies) • Measurements and opinions (quantitative and qualitative results) • System • Benchmarks • Test programs & measurements • Comparative analyses

14. Ethics in Research • A research lifecycle • Flavours of research • Ethics in research • Pitfalls • Getting started • Research proposals • Literature reviews • Presentations • MSc & PhD work Resource: “On being a Scientist: responsible conduct in research” www.nap.edu/readingroom/books/obas

15. Avoid conflicts of interest • Science relies on professional judgement which can be compromised by: • Financial conflicts (e.g., undisclosed shares in a company with interest in the outcome of research) • Social and personal beliefs (e.g., Einstein’s “God does not play dice”) • Pressures of competition (e.g., reviewing a paper with similar work) • Solutions: • Disclosure • Self knowledge • Peer review

16. Give credit where its due • Authorship (and order of authors): • Conventions can vary considerably • Best to decide upfront • Avoid “honorary” authors – must make a direct and substantial contribution • Establishes accountability as well as credit • Acknowledgements: • The place to give credit for less substantial assistance • Citations: • Part of the reward system – connected to funding and reputation

17. Case Study: Pulsars • Credit is a sensitive issue when researchers are of different seniority: • In 1967 Jocelyn Bell, a 24-year-old graduate student, discovered pulsars • Supervised by Anthony Hewish she was in charge of operating and analyzing data from a 4.5 acre radiotelescope • One day Bell noticed "a bit of scruff" on the data chart • Together Bell and Hewish analyzed the signal and found several similar examples elsewhere in the sky • With 3 others they published a paper announcing the discovery

18. Judgement on Pulsars? • Hewish got a Nobel Prize, Bell did not • Against: • Bell’s recognition of the signal was the crucial act of discovery • For: • Bell didn’t deserve a Nobel Prize for doing what is expected of a graduate student in a project conceived and set up by others

19. Shut Down the Paper Mill • The publish or perish paper mill: • Research careers seem to depend on quantity of publications not quality • Consequences: • Haste and negligence • But progress relies on a trust in previous results • MPUs (minimum publishable units) • But this dilutes contribution and forces wading through masses of literature

20. Case Study: nanotechnology • Jan Schön: • Worked in condensed matter physics and nanotechnology • Claimed he could replace silicone-based transistors with organic dye molecules • In 2001, averaged 1 paper every 8 days • On track for a Nobel Prize • Found Out: • Results seemed suspiciously precise • A researcher spotted identical graphs in two separate papers • Whole constructed data sets reused in different experiments

21. Judgement on Nanotechnology • Outcome: • Schön was fired from his position at Bell Labs after an internal investigation • Many of his papers were rescinded • He was banned from applying for funding in Germany • Other Consequences: • For his co-authors? • For reviewers of his papers?

22. Photo Manipulation • Recently many journals (Cell Biology, Science, Nature) have begun testing for photo manipulation • The following manipulations are not allowed: • Splicing together different images to represent a single experiment • Changing brightness and contrast of only a part of the image • Any change that conceals information, even when it is considered to be aspecific • Showing only a very small part of the photograph so that additional information is not visible

23. Misconduct • Fabrication (making up results) • Falsification (modifying results) • Plagiarism (copying without credit) • Suppresion (not reporting negative results) • Other deviations from accepted research practice: • Covering up misconduct, misuse of research funds, etc. • Consequences: • Harm to individuals, squandering of public funds, attracts criticism of Science • But how can scientists expect to get away with it?

24. Case Study: Dealing with misconduct • Francine is finishing her Ph.D. and Sylvia is a fellow grad student. Both have the same supervisor. • Francine realizes there are problems with Sylvia’s work, she: • Is rarely in the lab • Never shows anyone her code • Has performance results that seem too “clean” to be real • Also: • Francine needs a reference from her supervisor and Sylvia is one of her favourites • Both Francine and her supervisor are using Sylvia’s results for their own research

25. Judgement on Misconduct • Should Francine first try to talk with Sylvia, with her supervisor, or with someone else entirely? • Does she know enough to be able to raise concerns? • Where else can Francine go for information that could help her decide what to do?

26. Final Do’s and Don’t’s • Do: • Get ethics clearance if your research is potentially hazardous to human subjects • respect: • IP rights and confidentiality • Patents • The ACM code of ethics • Don’t: • Publish the same thing in more than one place • Inform the media of results before peer-reviewed publication

27. Pitfalls • A research lifecycle • Flavours of research • Ethics in research • Pitfalls • Getting started • Research proposals • Literature reviews • Presentations • MSc & PhD work Source: “The Researcher’s Bible” Homepages.inf.ed.ac.uk/bundy/how-tos/resbible.html

28. Solving the World • Easy to pick research goals that are too ambitious • Especially in Artificial Intelligence • Instead: • Allow the main burden of scoping to fall on your supervisor • Find out where the state of the art lies • Look to the future work section of papers • Can also redo bad work, properly

29. Manna from Heaven • Don’t expect inspiration to strike, staring at a blank piece of paper • “Science is 99% perspiration and 1% inspiration” • Instead: • Read the literature with a question in mind • Talk to people - your project partner and supervisor - and explain your ideas • Tackle a simplified version of your problem • Write down your ideas in a working form

30. Boondoggling • The appearance of work without actual productivity • Surprisingly seductive • Coding for its own sake • Writing Yet Another Programming Language (YAPL) • Gathering unnecessary experimental data • Instead: • Make sure your programming and experimentation contributes directly to the research

31. Ivory Tower • Focus on your topic is good but don’t shut out the rest of the world completely • Because it prevents cross-pollenisation of ideas • Instead: • Keep in touch with the state of the art in related fields - attend colloquia and talk to other students about their research • Set aside a part of the week for reading abstracts and skimming papers

32. Misunderstood Genius • Easy to believe that no one understands your ideas because you are a genius • More likely: • Love of jargon. CS is full of jargon. Try to rephrase your ideas using ordinary English • If I can do it, it's trivial. Once you have seen the solution to a problem it appears simple • Love of complexity. It’s not a virtue to make an unnecessarily complicated program - it’s just a nuisance to other people. Occam and Einstein were right!

33. Starting Research • A research lifecycle • Flavours of research • Ethics in research • Pitfalls • Getting started • Research proposals • Literature reviews • Presentations • MSc & PhD work

34. Finding a Research Question • What problems to tackle?: • What matters to you • Anything messy or difficult • New technologies • New users • A paper you enjoyed • A paper you disliked • Example question: “How can the new technology <T> be adapted to run on Cell Phones/PDA’s?”

35. Research Fit • What research are you suited to? • what interests you? • what expertise exists around you? • What are your skills? • Don’t neglect methodology: • Look at similar research to decide what kind of methodology is best for your research question • Make sure you are willing to apply that methodology

36. It’s not easy so why do it ? • Satisfaction & thrill of being the first to ever create/understand something • Famous Eureka moment • Privilege and recognition of adding to human knowledge • Meet/work with passionate, deep thinkers • Freedom – what to do, how, when, with whom

37. Refining the Research Question • Choose an initial objective • Read the literature • Refine the objective: • Narrow it • Write it as a question • Describe it in a single sentence • Decide on the measure of success • Do a quick first prototype/experiment

38. Research Proposal: Preparation • Ask yourself: • Am I familiar with related research in this area • Do I have a good understanding of the steps that will be involved in achieving these goals • Do I have the ability to successfully conduct each of these steps • Am I sufficiently motivated and enthusiastic about all the steps in this project • Am I convinced that the results of this research will be useful to others

39. Research Proposal: Structure • Honours proposal structure: • Project Description • Related Work • Outcomes (system, questions tackled, expected impact, key success factors) • Work Detail (timeline, resources required, deliverables, milestones, work allocation) • References • More general proposals might include: • Budget, CV, Dissemination plan

40. Research Proposal: Evaluation • Have you answered these questions: • What you are planning to do ? • Why ? • What the difficulties are? • Is it feasible for you ? • Do you have a plan of how to do it ? • Have you done your homework ?

41. Literature Reviews • A research lifecycle • Flavours of research • Ethics in research • Pitfalls • Getting started • Research proposals • Literature reviews • Presentations • MSc & PhD work

42. What is a Literature Review? • As a process: • Reading, taking notes, organising, documenting • Start with general, broad, textbook works • Move toward specialised, recent papers • As a document: • Not just a string of article summaries • Rather a coherent discussion of previous related work • Includes intro, conclusion, references • Don’t confuse the document and process

43. The Literature Review Process Understand the Field Broad • Lecture Notes • Text Books Find a Survey • Text Books • Survey Papers • Theses Find Focused Research • Research Papers Back Chain to References Forward Chain to Citations Narrow

44. Resources • Experts: lecturers, supervisors, librarians • ACM Digital Library • http://portal.acm.org/dl.cfm • Includes most ACM pubs (but not IEEE) • UCT has a subscription • Google Scholar • http://scholar.google.com/ • Good all-round resource • CiteSeer • http://citeseer.ist.psu.edu/ • Digital library and search engine • Heavily linked meta-data allows chaining through citations

45. Critical Reading Required • Being able to read rapidly and critically is a vital skill • First Skim: • Abstract, section headings and figures • Then Dive into Detail: • May require reading references for a full understanding • Take Notes: • Complete citation • Main research question & conclusions • Research methodology • Key ideas or results relevant to your research • Future work • Gaps/problems

46. Presentations • A research lifecycle • Flavours of research • Ethics in research • Getting started • Research proposals • Literature reviews • Presentations • MSc & PhD work

47. Communication of Ideas • Feedback is important: • To learn of new developments • To share responsibility • To get support and advice • To develop communication skills • Teamwork is important: • As a forum for feedback • To tackle larger problems • To learn interpersonal skills • Modern Science is advanced by sharing ideas and working in teams

48. Presentations • Present your main idea & its significance • Omit complicated & old ideas • Structure: • Intro, context, body, some detail, conclude • Know your audiences’ background • Make sure that both non- & experts benefit • Don’t gloss over problems with your ideas • Anticipate questions

49. Slide Layout • Allow about 2-3 minutes per slide • Avoid too much text • Just cues • About 7-15 bullet points at most • Include graphs/charts/pictures • Avoid code/maths

50. Nervous ? • Prepare well, run through with your supervisor • Time yourself • Contract stomach muscles & breathe out hard • Speaking Skills: • A pause is better than an interjection • Speak more slowly than you think necessary • Repeat questions if they are inaudible