Research Method - Review

1. Research Method - Review Wei Le

2. What is research • Knowledge discovery: new, never done before - can be applied for future problems or predict future behaviors • Research is a systematic process of collecting, analyzing, and interpreting information in order to increase our understanding of an phenomenon

3. Research Mindset • Research mindset: iterative process, small steps, critical thinking • Verbal reasoning - persuasive writing • Argument analysis - reasons/data support or do not support the conclusions • Probability reasoning - likelihood, uncertainty related to various events • Decision making - identifying and evaluating alternatives, and selecting alternatives most likely to lead to a successful outcome. • Hypothesis testing

4. Research process Question / problem to solve -> guess (hypothesis)-> apply research methodology to collect and analyze data to answer the questions: support hypothesis -> more questions generated

5. Research Activities Write proposals Peer review Dissemination of the work: workshop (discuss ideas), conference (present your work) and journal papers (add more details) Critical thinking Implementation Mathematical proof

6. Good Researchers • New ideas and insights • Plan out a coherent story • Time management • Self-learning • Communication skills ….

7. How to Find a Research Problem • Problems from daily working • Go to talks, even not your areas • Go to conferences • Brainstorm with people, papers …

8. .bib Database and References Build a .bib database References section in the paper (only include the ones you cited) Commonly seen citation source (key: reliable) • Books, papers, articles, reputable web, personal communications • No: wiki

9. How to Locate Related Work • Google and key words • From known papers and their references • Abstract from conference proceedings and journals • When to quit: when you are not able to find more new info such as no more new papers, approaches, findings

10. Write a Paper Summary • Active reading (reading with questions) • What’s the goal of the paper? (problem solved, a survey, a study) • Previous approaches • What do they do (methodologies: an algorithm, steps for study) • Results • Pros and cons (why) • Ideas for future work • How this related to your work • a type of representation I surveyed • a different approach for the same problem - pros/cons

11. Literature Review • Related work mind map • Organize a cohesive review: where our work stands and what is its value • Describe general trends • Show how approaches to the topics are changed over time • Compare and contrast theoretical perspective and technical keys

12. Research Proposal Organization • Introduction (problem and setting) • Literature review • Overview and research steps • Evaluation plan • Preliminary work • Qualification of PIs • Merit and Broader Impact • References • Budget

13. Qualitative Research and Quantitative Research • Qualitative: find observations and insights. depths during an exploratory stage • Quantitative: is the observation generally applied? Breaths • Collect quantitative data

14. Theory • A theory is an organized body of concepts and principles intended to explain a particular phenomenon (how and why this occurs,  and allows the prediction) • A scientific theory identifies and defines a set of phenomena and makes assertions about the nature of those phenomena and the relations between them • terms - a community of scientists can observe and measure them compare to other related work • multiple theories might explain the observations, the simpler is the better

15. Grounded Theory theory building process:  a technique for developing theory iteratively from qualitative data. Initial analysis of the data begins without any preconceived categories. An interesting patterns emerge, the researcher then repeatedly compare these with existing data, and collects more data to support or refute the emerging theory.

16. How to Ask a Research Question • stage 1: vague, exploratory questions to clarify our understandings - can help get rich and qualitative data so that we can refine ideas - these questions might already be asked in the literature • existing questions: Does X exist? • description and classification questions: • What are the types of mobile phone bugs? • How the graphical representations are used in program analysis? • descriptive-comparative questions: • how mobile phone bugs are different from traditional ones? • stage 2: better defined research questions • frequency and distribution questions? How often the types of bugs occur? • process questions? How the bugs are produced? • relation questions?  Are A and B related? How performance bugs/memory leaks related to battery drain? • a good paper will give relations discovered and why • Causality: Does A cause B • Causality comparative questions: Does A cause B more than C causes B? • Causality comparative interactions: Does A cause B more than C causes B under condition X?

17. Research Hypothesis and Assumptions • Hypothesis: reasonable prediction, logical supposition. It provides a tentative explanation for a phenomenon under investigation • Assumption: a condition is taken for granted    two assumptions underlie all research (generalization): • the phenomena under investigation is somewhat lawful and predictable; it is not comprised of completely random events • Certain cause-and-effect relationships can account for the patterns observed in the phenomenon

18. Statistical Hypothesis • Testing a hypothesis: construct statistical hypothesis from research hypothesis H0 – null hypothesis H1 – Alternative hypothesis • Collect data to support research hypothesis: • you are aiming to reject null hypothesis, find counter examples • Statistical generalization and analytical generalization

19. Research Methods: Controlled Experiments and Case Studies • Controlled experiments Experimental investigation of a testable hypothesis, in which conditions are set up to isolate the variables of interest ("independent variables") and test how they affect certain measurable outcomes (the "dependent variables") • Case study: an empirical inquiry that investigates a contemporary phenomenon within its real-life context

20. When Case Study When Controlled Experiment • When you can’t control the variables • When there are many more variables than data points • When you cannot separate phenomena from context • Phenomena that don’t occur in a lab setting • E.g. large scale, complex software projects • Effects can be wide-ranging. • Effects can take a long time to appear (weeks, months, years!) • When the context is important • E.g. When you need to know how context affects the phenomena • When you need to know whether your theory applies to a specific real world setting

21. Research Methods: Action Research and Survey • Survey Research: • to identify the characteristics of a broad population of individuals • selection of a representative sample from a well defined population (challenges: how to select and justify why representative  - classify populations, frequently used, important) and data analysis and reach a conclusion about the populations • Action research: • solve a real-world problem while simultaneously studying the experience of solving the problem • Mixed Research Method

22. Validity: how to evaluate a research method • Construct Validity: • Concepts being studied are operationalized and measured correctly • Internal Validity • Establish a causal relationship and distinguish spurious relationships • e.g., confounding : control - variables other than the chosen independent variables must not be allowed to affect the experiments • External Validity • Establish the domain to which a study’s findings can be generalized • Empirical Reliability • Demonstrate that the study can be repeated with the same results

23. Data Measurement • A measure is a number or symbol assigned to an entity to characterize an attribute • an entity: table • attribute: length • numbers: 3 meters • Each different mapping from an attribute to a measurement value is a scale: nominal, ordinal, interval, ratio • Subjective measure and objective measure • Direct measure and indirect measure

24. 3 Steps in Data Analysis • Descriptive statistics - presentation and numerical processing of a data set:  summarize the main feature of the data • Inferential statistics: significant tests, make decisions • Data set reduction: abnormal and false data points are excluded • Hypothesis testing

25. Descriptive Statistics: 3 ways to Summarize the data • Measures of central tendency: mean, median, mode • Measures of dispersion: frequency, standard deviations, range • Measures of dependencies: linear, correlations (Pearson, Spearman) • Graphical visualizations: scatter plot, histogram, pie chart

26. Collaborations With Industry • Identify industrial challenge • Problem formulation • Acandidate solution is proposed -- develop in close with the industrial partners • Validation in academia • Static validation • Dynamic validation • Release solution • Conclusions:

27. Research Ethics • Non-disclosure agreement • Human subjects: • Informed consent • Confidentiality of the data • Sensitivity of the results

28. Persuasive Writing • Details are important: format, font, space, figures • Coherent story • Logical • Specific

29. General Problems in Final Report Draft • Call for papers • Anonymous • Format problems • Frequency of Subtitles and paragraph length • Disconnect between research goals and results • Correctness of the citation: x et al. proposed approach on x [1] • Appendix: you cannot discuss details in the text using figures/tables in the appendix

30. Peer Review • Tone needs to be less offensive • More technical details and critiques on data collection, analysis, generalization, logic … • A sample paper and reviews: you can come to take a look