Exploring Computer Science Research: Theoretical vs. Empirical Approaches

1. CSCD 506Research Methods Winter 2018 Doing Research, Research Method and Other Approaches Lecture 10 1

2. Topics • Computer Science Research • Theoretical • Empirical • Simulation • Differences and Examples 2

3. Types of Research in Computer Science 3

4. Theoretical Computer Science Theoretical computer science (TCS) is a subset of computer science that focuses on mathematical topics as they relate to practical and modern computing A widely accepted view of TCS is study of algorithms and protocols in modern computing

5. Empirical Computer Science Empirical analysis is an evidence-based approach to the study and interpretation of information. Empirical approach relies on real-world data, metrics and results rather than theories and concepts. Empirical analysis is key to scientific method Used to study subjects for through quantified observations of empirical evidence. Empirical analysis never gives an absolute answer, only a most likely answer based on probability

6. Research Methodologies in CS • There is some disagreement about topics within computer science research • No good answer to “What is CS Research?” • Well Accepted Areas • Theoretical CS Research • Empirical CS Research • Newer Research Method • Simulation CS Research 6

7. Theoretical and Empirical Research • Theoretical research - use mathematics/logical reasoning to prove what you are saying is definitely true • Empirical research- research through observation • Run many experiments to show that what you are saying is probably true 7

8. Theoretical vs. Empirical Research • Example: Casino Gambling Strategy • Theoretical proof that a particular casino gambling strategy will give you the highest possible winnings, • Provide mathematical proof of this • Demonstrate through experiments that a particular strategy works better than others • You could become rich in the process! • Note: It’s not always easy/possible to prove something mathematically 8

9. Empirical Computer Science • Experimentation in Computer Science aims to answer questions like these • Is program A really more efficient than program B, in practice? • Does A's runtime vary more widely than B's on different inputs? • Is program A really more accurate than program B, when applied to some task? • What are the best parameters for compiling or running a particular program? 9

10. Empirical Software Engineering There are two kinds of targets for empirical software studies: Products and Processes Product studies assess actual software artifacts, as found in code repositories, bug databases and documentation, to infer general insights Project studies assess how software projects proceed and how their participants work; as a consequence, they can share some properties with studies in other fields that involve human behavior, such as sociology and psychology

11. Some Empirical Problems in Software Engineering • Formal experiments • Often done with students (considered guinea pigs)‏ • Can control artifacts, process, and context • Similar experiments • In industry, costly and many logistical problems. Do not have control of environment Techniques Used • Surveys Often by questionnaires • Interviews More personal than surveys • Observation Being a “fly on the wall” 11

12. Some Empirical Problems in Software Engineering • Formal experiments • Often done with students (considered guinea pigs)‏ • Is there a problem with using students in experiments? • They are not representative of developers • Experimental conditions (labs in school) not the same as in industrial setting • Can't conduct long-term experiment with students who are temporary 12

13. Theoretical Computer Science Research • The field of theoretical computer science is interpreted broadly to include • Algorithms, data structures, computational complexity theory, • Distributed computation, parallel computation, VLSI, machine learning, computational biology, computational geometry, • Information theory, cryptography, quantum computation, computational number theory and algebra, program semantics and verification, • Automata theory, and the study of randomness • Work in this field often distinguished by emphasis on mathematical technique and rigor 13

14. Empirical Research vs. Theoretical inComputer Security Theory • Prove the correctness of cryptographic protocols given certain assumptions • Prove the correctness of security software by hand or automated theorem provers Creating a Protocol Composition Logic (PCL)‏ • Logic for proving security properties of network protocols that use symmetric key cryptography • The logic is designed around a process calculus ... Example 14

15. Empirical Research vs. Theoretical ComputerSecurity • Empirical Research • Capture network traffic and analyze it • Create controlled experiments of network traffic • Write own malware and run it against experimental networks (hope they don’t escape!!)‏ • Study people who use security mechanisms • Look at usability, errors, susceptibility to compromise • Use surveys, observation, tests • Test and compare security software for effectiveness 15

16. Empirical Research in Human Computer Interaction

17. When we do Research, we… Observe Measure Describe Compare Infer Relate Predict etc.

18. When we do Research, we… Observe … human behavior and response Measure … using numbers Describe … using numbers Compare … using numbers Infer … using numbers Relate … using numbers Predict … using numbers etc. Empirical Empirical - capable of being verified or disproved by observation or experiment (Websters dictionary)‏ So, what is non-empirical research?

19. Non-Empirical Research Non-empirical research (aka qualitative research) is generally concerned with the reasons underlying human behaviour (i.e., the why or how, as opposed to the what, where, or when) Tends to focus on human… Thought, feeling, attitude, emotion, reflection,, sentiment, opinion, mood, outlook, manner, approach, strategy, etc. These human qualities are not directly observable or measurable and, therefore, necessitate a different method of inquiry (e.g., case studies, focus groups, cultural probes, etc.) Ask different questions, How did you like the Interface?

20. Observe Observations are gathered… Manually Human observers using log sheets, notebooks, questionnaires, etc. Automatically Sensors, switches, cameras, etc. Computer + software to log events + timestamps

21. Measure A measurement is a recorded observation An empirical measurement is a number When you cannot measure, your knowledge is of a meager and unsatisfactory kind. Kelvin, 1883

22. Research Questions Considerfollowing questions about a new or existing UI design or interaction technique : Is it viable? Is it as good as or better than current practice? Which of several design alternatives is best? What are its performance limits and capabilities? What are its strengths and weaknesses? Does it work well for novices, for experts? How much practice is required to become proficient?

23. Testable Research Questions Preceding questions, while relevant, are not testable Try to re-cast as testable questions Scenario… You have an idea for a new [technique for entering text on a mobile phone] and you think it’s pretty good. In fact, you think it is better than [the commonly used multi-tap technique]. You decide to undertake a program of empirical inquiry to evaluate your idea. What are your research questions? Replace […] as appropriate for other research topics

24. Research Questions (2)‏ Very weak (in an empirical sense)‏ Is the new technique any good? Weak Is the new technique better than multi-tap? Better Is the new technique faster than multi-tap? Better still Is the new technique faster than multi-tap within one hour of use? Even better If error rates are kept under 2%, is the new technique faster than multi-tap within one hour of use?

25. A Tradeoff Internal validity External validity If error rates are kept under 2%, is the new technique faster than multi-tap within one hour of use? High Low Accuracy of Answer Is the new technique better than multi-tap? Narrow Broad Breadth of Question

26. Internal Validity Definition: The extent to which effects observed are due to test conditions (e.g., multitap vs. new)‏ Statistically… Differences (in the means) are due to inherent properties of the test conditions Variances are due to participant differences (‘pre-dispositions’) Other potential sources of variance are controlled or exist equally and randomly across the test conditions

27. External Validity Definition: The extent to which results are generalizable to other people and other situations Statistically… People The participants are representative of the broader intended population of users Situations Test environment and experimental procedures are representative of real world situations where the interface or technique will be used

28. Computer Simulation http://www.mrtc.mdh.se/publications/0446.pdf • Computer simulation makes it possible to investigate regimes that are beyond current experimental capabilities • Study phenomena that cannot be replicated in laboratories • Evolution of the universe • High speed real-time simulation of systems • Simulation of Internet Worms • Traffic analysis on the Internet • Widespread outbreaks of disease 28

29. Computer Simulation Steps to Conducing Simulation Research Create a model of a target that is simpler to study than the target itself Model will always be simpler than the target We can represent the model itself as a specification – a mathematical equation, a logical statement or a computer program Examine how model behavior develops over time Derive model’s future structure from specification by reasoning, perhaps using logic or more often by using mathematics Simulation means ‘running’ model forward through (simulated) time and watching what happens

30. Computer Simulation Steps to Conducting Simulation Research Need to ensure that model is correctly implemented and working as intended. This is verification – in effect, a ‘debugging’ step Next, there is validation , ensuring that behavior of model corresponds to behavior of target Finally, one needs to know how sensitive model is to slight changes in the parameters and initial conditions: sensitivity analysis

31. Summary Computer Science research has a rich set of methodologies Everything from mathematical proof to user experiments and case studies Broad nature of Computer Science requires us, computer scientists to figure out new methods and techniques to study it I expect CS research will continue to evolve as the discipline itself evolves …

32. Next Assignment • Expansion of your research proposal • Working with Research Methods End 32