Research Methods

1. Research Methods Howell Istance Department of Computer Science De Montfort University © De Montfort University, 2001

2. This session…. • Why do you need this…. • Online resources • Literature reviews • The nature of research • Questionnaire design • Structured investigations © De Montfort University, 2001

3. Why do you need this…? • To understand content of directed reading within taught modules • To be able to write essays or reviews of published literature • To be able to plan and conduct your project where there is an element of research in the investigation or the evaluation of what you have done © De Montfort University, 2001

4. Library student study skills books… • http://www.library.dmu.ac.uk/Skills/Study/ • Collection of self study books relevant to research and projects • Most available in .pdf or .html format (free download) • Series includes… • Research Methods • Designing a questionnaire • Information Citation and Control • Dissertation preparation and presentation • Research interviews • Planning a sample survey • Thinking it through: an introduction to critical thinking © De Montfort University, 2001

5. MSc projects • All projects are required to have at least one academic objective, usually an investigation of literature relevant to some aspect of your project • Most projects will include some form of evaluation • E.g the production of a computer-based tool to teach letters of the alphabet to nursery level children • A piece of software which purports to do this will be of limited value without some evidence of how effective it is in doing so. • Demonstrating its effectiveness requires a structured study © De Montfort University, 2001

6. Project Skills… • There is a non-assessed module in the second semester, which covers • Personal time management • Project management • Project Selection • Written and verbal communication • Giving demonstrations • Learning Strategies © De Montfort University, 2001

7. Literature reviews • Conducted to ensure a researcher is familiar with ‘all’ of the what is known about a particular field • Often published in order to bring other researchers (and MSc students) up to speed quickly in an unfamiliar field • Need to have the scope of the review carefully defined • Not too big such that adequate coverage is infeasible, and there is too much literature to review and the review becomes unfocussed (and thereby not useful) • Not too narrow such that there are too few papers to include © De Montfort University, 2001

8. Form of a literature review… • Requires reading literature from a variety of sources • Forming some form of taxonomy or structure for your review • Identifying where in your taxonomy the various contributions from the literature fall • Critically reviewing the literature • Identifying different approaches, contradictions between contributions, analysis of strengths and weaknesses • Not simply pasting quotes from different papers • Drawing your own conclusions, particularly concerning completeness of coverage • Highlight implications for your work (if appropriate) © De Montfort University, 2001

9. Example of a literature review Hand, C. "A Survey of 3D Interaction Techniques". Computer Graphics Forum, 16(5): 269-281. (Dec 1997) (Abstract) Recent gains in the performance of 3D graphics hardware and rendering systems have not been matched by a corresponding improvement in our knowledge of how to interact with the virtual environments we create; therefore there is a need to examine these further if we are to improve the overall quality of our interactive 3D systems. This paper examines some of the interaction techniques which have been developed for object manipulation, navigation and application control in 3D virtual environments. The use of both mouse-based techniques and 3D input devices is considered, along with the role of feedback and some aspects of tools and widgets. © De Montfort University, 2001

10. Example of a literature review Hand, C. "A Survey of 3D Interaction Techniques". Computer Graphics Forum, 16(5): 269-281. (Dec 1997) (Abstract) Recent gains in the performance of 3D graphics hardware and rendering systems have not been matched by a corresponding improvement in our knowledge of how to interact with the virtual environments we create; therefore there is a need to examine these further if we are to improve the overall quality of our interactive 3D systems. This paper examines some of the interaction techniques which have been developed for object manipulation, navigation and application control in 3D virtual environments. The use of both mouse-based techniques and 3D input devices is considered, along with the role of feedback and some aspects of tools and widgets. rationale © De Montfort University, 2001

11. Example of a literature review Hand, C. "A Survey of 3D Interaction Techniques". Computer Graphics Forum, 16(5): 269-281. (Dec 1997) (Abstract) Recent gains in the performance of 3D graphics hardware and rendering systems have not been matched by a corresponding improvement in our knowledge of how to interact with the virtual environments we create; therefore there is a need to examine these further if we are to improve the overall quality of our interactive 3D systems. This paper examines some of the interaction techniques which have been developed for object manipulation, navigation and application control in 3D virtual environments. The use of both mouse-based techniques and 3D input devices is considered, along with the role of feedback and some aspects of tools and widgets. Scope of review © De Montfort University, 2001

12. Example of a literature review Hand, C. "A Survey of 3D Interaction Techniques". Computer Graphics Forum, 16(5): 269-281. (Dec 1997) (Abstract) Recent gains in the performance of 3D graphics hardware and rendering systems have not been matched by a corresponding improvement in our knowledge of how to interact with the virtual environments we create; therefore there is a need to examine these further if we are to improve the overall quality of our interactive 3D systems.This paper examines some of the interaction techniques which have been developed for object manipulation, navigation and application control in 3D virtual environments. The use of both mouse-based techniques and 3D input devices is considered, along with the role of feedback and some aspects of tools and widgets. Indication of taxonomy © De Montfort University, 2001

13. Sources of information • Scientific journals • Conference proceedings (refereed and unrefereed) • Magazines, newspapers • WWW • Important to differentiate between reported investigations and reported opinion • Conclusions of any investigation contain some element of informed opinion or judgement • Sources of information must be cited appropriately © De Montfort University, 2001

14. Project Academic Objectives • For example.. ‘Review methods of usability evaluation appropriate to the evaluation of internet-based groupware systems for use by students undertaking a course by distance learning’ • Requires review of methods described in the literature, using the original sources (not just someone else’s review) • Implications of problems/needs associated with evaluation of groupware systems • Comparisons and contrasts between methods in light of identified needs of groupware applications • Your conclusions about which method(s) to use and how © De Montfort University, 2001

15. How to do it… • Collect and read current papers and reviews of a field • From the list of references cited in these, get copies of those which appear relevant • Start to classify the papers you collect in one or several ways to form the basis of a taxonomy • Identify authors who are prominent and check what else they have published (WWW useful here) • Read contents of recent relevant journals in library • Subscribe to mailing lists for coming conferences and look at contributors • search on-line bibliographic archives (e.g bids.ac.uk) © De Montfort University, 2001

16. Constructively criticising research.. • Researchers have a vested interest in making a piece of work appear significant and worthy of publication • They may not be as forthcoming about the limitations of the work as about the benefits • Results should be reported in sufficient detail to enable the reader to draw his/her own conclusions and thereby judge the validity of the conclusions drawn by the author • Are the conclusions drawn justified by the evidence provided? • Is the method of investigation appropriate or flawed? • Are the investigators aware of other similar contributions in the field? © De Montfort University, 2001

17. What is research? • A systematic enquiry, which is reported in a form that allows the research methods and outcomes to be accessible to others • Concerned with seeking solutions to problems or answers to meaningful questions • Meaningful questions are expressed in a way that indicates what you will accept as an answer • Non-meaningful (in research terms) questions are not answerable as a result of enquiry alone (eg judgemental or metaphysical questions) • Positivism versus phenomalism © De Montfort University, 2001

18. Nature of research - positivist • Deals with positive facts and observable phenomena • Subscribes to the ‘scientific method’ • Primary goal is not only description but prediction and explanation • Classification of substances and events, and observation of these, provide the basis for descriptive laws based on consistencies in patterns and properties • Characterised by absolute or varying degree of generalisability • Quantitative, as it draws on measurable evidence © De Montfort University, 2001

19. Postulates in Positivist Research • postulate of natural kinds: all instances of classes and categories of phenomena exhibit the same properties • postulate of constancy: all phenomena remain the same or change only very slowly over time • postulate of determinism: there is orderliness and regularity in nature, constancy in terms of cause and effect © De Montfort University, 2001

20. Nature of research - phenomenalist • Considers that each phenomena is unique and is controlled by variables such as time, location and culture • No two situations are identical • No reliance on postulates of natural kinds, constancy or determinism • Essentially subjective, where the content of research and the way it is pursued is indicative of researchers intention • Outcomes are descriptions which are expressed in narrative and mainly in qualitative terms © De Montfort University, 2001

21. Stockholm archipelago © De Montfort University, 2001

22. Example of both approaches… Can the study of critical incidents (as opposed to accidents) in marine navigation in the Stockholm archipelago provide the basis for improvements in sea safety in the area? • Positivist approach: collect data via interview, classify types of incidents, produce analyses, make recommendations based on analysis • Phenominalist approach: analyse interviews in depth, seek to draw conclusions about causal factors © De Montfort University, 2001

23. Positivist research methods include... Descriptive research • Anything that is variable, varies to a defined degree, and thus can be measured • Surveys, case studies, causal comparative studies, correlational studies, developmental studies, trend studies Experimental research • Deliberate manipulation of certain factors under highly controlled conditions • Purpose is to identify causal connections through keeping the levels of some variables constant and manipulating others © De Montfort University, 2001

24. Questionnaire design • Relevant to requirements gathering for, and evaluation of, projects • Self study pack by Arthur Rothwell covers: • Planning and logistics of questionnaire design • Layout issues • Forms of questions • Contents of questions © De Montfort University, 2001

25. Planning and logistics of questionnaire design • Quantitative or qualitative? • Legal requirements: the Data Protection Act • Confidentiality and anonymity • Sample size • Volunteer respondents • Identifying subject areas • Determining appropriate length • Typical time scale • Main components of questionnaires © De Montfort University, 2001

26. Forms of question • Open and closed questions • Dichotomous items • Scaled items • Mid-point or no mid-point? • Ranking to show preference © De Montfort University, 2001

27. Content of items • Avoiding response set • Components of attitudes • Common types of faulty items • leading questions • context effects • double barelled questions • vague and ambiguous terminology • hidden assumptions • social desirability © De Montfort University, 2001

28. Leading questions and context effects Would you agree that the governments policies on health are unfair? • Item wordings should not contain value judgements How many pints of beer did you drink last night? • Think how the context of the study would affect the response, say in a • survey of young peoples life styles • survey of health behaviour and heart disease © De Montfort University, 2001

29. Double barreled questions Do you believe the training programme was a good one and effective in teaching you new skills? • avoid questions that involve multiple premises © De Montfort University, 2001

30. Vague and ambiguous terminology How often do you clean your teeth? • Frequently • often • infrequently • never • what does ‘frequently’ mean? • Give quantifiers to ensure all respondents understand the same thing by the response categories © De Montfort University, 2001

31. Hidden assumptions, social desirability When did you last borrow a video tape? • Avoid hidden assumptions - what are these? Do you ever give to charity? • May lead to a positive response as otherwise something negative about the respondent is being conveyed © De Montfort University, 2001

32. Structured Investigations - 1 • An MSc project has produced a computer-based tool to teach letters of the alphabet to nursery level children • Talk to the person next to you and discuss how you would demonstrate the effectiveness of the tool through some form of structured investigation • Consider the resources required and the feasibility of conducting the study within a 3-month project © De Montfort University, 2001

33. Ways of approaching this…. • You need access to people with expertise and, hopefully, a group of nursery children • The product needs to be tested, and has to be ready in time for this to happen • Subjective assessment by teachers comparing the tool with • Other computer based tools • Paper-based methods • Objective approaches • Before and after letter recognition tests with two matched groups of children • Use of non-parametric statistical techniques © De Montfort University, 2001

34. Structured Investigations - 2 • A project has produced a web-based presentation of a college offering educational courses both as a 2D collection of pictures of the building, and as a 3D virtual building which you can ‘walk through’ • How would you compare the relative efficiencies of each approach? • Again, discuss with the person next to you • What you are trying to assess • How to carry out the assessment © De Montfort University, 2001

35. Ways of approaching this…. • What to assess? • How much information a visitor to the site obtains with each type of presentation • Usability issues – e.g ease of navigation through the representations of the college building • How to assess this? • Unstructured user trials, where the amount of building explored is recorded • Structured user trials, where a task is given to find pieces of information • Subjective assessment of visit by participant • Observed usability problems © De Montfort University, 2001