IS 4800 Empirical Research Methods for Information Science Class Notes January 11, 2012

1. IS 4800 Empirical Research Methods for Information Science Class Notes January 11, 2012 Instructor: Prof. Carole Hafner, 446 WVH hafner@ccs.neu.edu Tel: 617-373-5116 Course Web site: www.ccs.neu.edu/course/is4800sp12/

2. Topics for Today • Why does this course exist ? • Evidence-based research in IT • Scientific method • What is an empirical research method ? • Purposes and Products of empirical research • Examples, Assignment 1

3. I. Why does this course exist? • Background facts • Empirical research means gaining new knowledge through systematic observation and analysis of data • Empirical methods usually taught only to graduate student • Usually taught only in behavioral/social sciences • In the past, little use of empirical research in CS (notable exception is HCI/UI design). This is slowly changing.

4. Why does this course exist? • The need for an empirical approach to research • IS is concerned with users and clients as well as the computer and the software • Research involving people requires empirical studies, since logical proofs are not possible • Many decisions today are based on “anecdotal evidence” or opinions Example: proponents of software design and development methods (functional programming, object-oriented programming, formal methods, design patterns, eXtreme Programming) claim their method improves productivity and/or quality based on their opinion or anecdotal evidence.

5. Why does this course exist ? • Why teach empirical research to undergraduates who are usually not aiming for careers as researchers? • For careers in system admin/IT management • For careers in system design/development • For careers in IT planning/policy (“governance”) • All educated people should be able to analyze and evaluate claims based on empirical evidence.

6. Examples: sexting, e-readers • Articles in WSJ August 25 2010 • “it’s difficult to pinpoint how common the practice (sexting) is. Various surveys have estimated . . . from roughly 4% to 25%.” • “ a study of 1,200 e-reader owners . . found that 40% said they now read more than they did with print books.” • Companies with IT product lines, legislators, etc. need their questions answered with evidence, not just opinions. Empirical research skills will be increasingly valuable.

7. II. Evidence-based research in IT:the MIS researcher’s perspective • Research on (organizational) impact • What is the impact of computerized systems on productivity, group communication, power relationships and organizational practices ? • Does the system make the organization more effective in carrying out its mission ? • Do the economic benefits of computerized systems exceed their costs? • Research on IT management practices • What approaches work best for IT governance ? • How can risks be identified and minimized ?

8. Evidence-based research in IT:the computing researcher’s perspective • Research on system lifecycle (requirements analysis, design, development) • How can system designers translate people’s needs and preferences into requirements ? • What are the advantages and disadvantages of proposed software/system development methods ? Both MIS and computing perspective • What problems should be expected when systems are deployed in their target environment • What are the best approaches (technical and managerial) to protect the security of systems

9. Evidence-based research in IT:How the Internet fits in • We can study who uses the Internet for what purposes • We can study how the Internet affects us as individuals and as a society • We can use the Internet as a tool in our research (e.g., for surveys or interviewing) • We can study how Internet-based research is different from other types (e.g., different ethical issues arise).

10. III. The Scientific Method • Methods of inquiry • Method of authority • Method of logic (deductive reasoning from known facts) • Scientific Method (inductive reasoning) • “The Name of the Rose” by Umberto Eco: best selling novel about the transition from authority to reason in 1300’s. (movie starring Sean Connery)

11. Scientific Explanation • Empirical • Rational • Testable • Tentative • Parsimonious • Rigorously evaluated (vs. “anecdotal”)

12. Steps of the Scientific Method • The scientific method consists of four steps • Noticing a phenomenon • Formulating testable explanations (hypotheses) • Observing and/or experimenting • Refining and retesting explanations • The scientific method can be tedious and time consuming • But it is the most widely accepted methodology for acquiring knowledge about the world.

13. Noticing a Phenomenon • After noticing a phenomenon, you identify VARIABLES or THEMES that appear important in explaining what happens • Formulating Tentative Explanations • Initial observations allow you to develop a HYPOTHESIS, or tentative statement, about the relationships among the variables identified

14. Observing and Experimenting • You DESIGN and carry out systematic OBSERVATIONS of the behavior of interest, • These observations are directed at testing your hypothesis • Refining and Retesting Explanations • Supported hypotheses are often REFINED and subjected to further exploration • Disconfirmed hypotheses may be reworked and RETESTED

15. The importance of skepticism • Ask whether the observed data can be explained another way? • Seek data that disconfirms a theory • View all theories as tentative (accepted until disproved) How do I know whether this is true?? How could it be proved false??

16. Challenges for Scientific Method in Behavioral/Social Research • Finding hypotheses that are both testable and that answer your question • The feasibility of collecting data to test your hypothesis • The ethics of collecting data to test your hypothesis Ex: social impact of parental neglect: experiments carried out on chimpanzees

17. Philosophical assumptions Research strategy/method Researchdesign IV. What is a “research method”? A strategy of inquiry, which moves from underlying philosophical assumptions to a research design Philosophical schools regarding research (“epistemology”): positivist – research should seek objective truth interpretive – research should take human experience into account, truth can be subjective critical – research should be aimed at making the world better

18. Categories of Empirical Data • Artifacts • Laboratory experiments • Natural observations • Questionnaires • Interviews Attributes of research methods • What kinds of questions can be answered? • What sources of data are utilized? • How is the data generated/collected? • How is the data analyzed? • How do the results of analysis result in answers?

19. Overview of the research methods • Quantitative • correlational studies • surveys • experiments • quasi-experiments • Qualitative • ethnography • case studies • grounded theory • Mixed

20. Classification schemes for research methods • Epistemological – what kind of claims can be made • Descriptive/Interpretive • Correlational/Relational • Causal • Methodological: how we go about it (process) • Level of generality • Scientific research aims to discover general principles or test a general theory • Evaluation research aims to support decision making or test a hypothesis in a specific context

21. Deploying Empirical Research Methods Idea Observation Prior Knowledge Research hypothesis or question Research Method Research Design What to observe and appropriate measures Subject Population/ Sampling technique How the data will be analyzed Step-by-step research plan

22. Steps of Empirical Research • Select a topic or phenomenon to study • Perform background investigation/literature review • Identify specific research questions or hypotheses • Select a research method • Develop a detailed research plan/proposal • For identifying or recruiting subjects • For collecting data • Define variables or phenomena to be observed and a coding scheme • Determine how the data will be obtained • For analyzing data • Execute the plan • Document the research you performed (ongoing)

23. V. Purposes and Products • Reasons for doing empirical research • Disconfirm or support a theory • Solve a design or technical problem • Evaluate a product or solution • Inform practice (develop “best practices”) • Understand why people do what they do • Predict the future • Contribute to a better world • Advance personal goals

24. V. Purposes and Products • Products of Empirical Research • Evidence of a relationship • Disconfirm a theory (or some aspect of it) • Comparison of existing products or techniques • A new or improved product or technique • In-depth study of a particular outcome • Explanation of unanticipated outcomes • A new theory • A new improved research tool or technique • A critical analysis (pros and cons)