CHAPTER 5 Planning the Research Project

1. CHAPTER 5Planning the Research Project • Successful research is a planned research • Fail because plunged into activity with only a partially thought-out plan & inconclusive design • Research requires a conceptualisation of overall organization and detailed plan • Should be no less totally visualized and precisely detailed

2. Planning the Research Project • A general strategy = research design (RD) • RD provides • Overall structure for the procedures • The data the researcher collects • The data analyses the researcher conducts • Must plan the overall design carefully to be successful • Wasted effort if half-prepared (vague ideas) • Must identify RESOURCES, PROCEDURES, and DATA always with the goal from the very beginning

3. All Research has a Basic Format • No matter what the academic discipline • Not parochial (narrow, limited) and seldom fits into neatly packaged academic disciplines • Software engineer may be compelled to stray far from the field of SE in pursuing a research problem in user interface design • The ramifications of problem may lead into fields far from that of the original problem • Err in thinking too narrowly, in restricting problems for research to one academic area

4. The Basic Format of the Research Process • 1. A question is posed. In the mind of the researcher, a question arises that has no known resolution • 2. It’s a matter of words. The researcher converts the question to a clearly stated research problem. • 3. It’s worth a guess! The researcher poses a temporary hypothesis or series of hypotheses. • 4. The search is on! The researcher searches the literature for ideas that shed light on the problem and for strategies that may help to address it.

5. The Basic Format of the Research Process • 5. Data! Hard Data! And nothing BUT the data. The researcher collects data that potentially relate to the problem. • 6. How do the data fit together? The researcher arranges the data into a logical organizational structure. • 7. The data speak! The researcher analyzes and interprets the data to determine their meaning • 8. It’s either … or … Either the data seemingly resolve the research problem or they do not. Either they support the hypotheses or they do not.

6. All Research has a Basic Format • Might do better to think of problems as arising out of broad generic areas within whose boundaries all research falls: • People– research problems relating to individuals, groups, populations, nationalities, families, ... • Things – biological life, matter, … • Records – letters, legal documents, lists, journals, …

7. All Research has a Basic Format • Thoughts and ideas – opinions, reactions, concepts, theories, … • Dynamics and energy – human energy and activity, metabolism, gravity, hydrologic cycles, … • Above suggestions are not all-inclusive, merely suggest the broad ramification of research possibilities • Substratum (an underlying layer or foundation) of research principles is all-inclusive

8. All Research has a Basic Format • Physicist exploring subatomic particles and sociologist exploring social behaviour are both employing the same research principles, although tools may differ, interpretational techniques may have little in common – approach identical • Research approaches all problems through certain methodological channels appropriate to the nature and type of data • Must differentiate research design VS research methodology

9. Research Planning Versus Research Methodology • Many people are confused between the two • Example, statement “research in physics is different from research in philosophy or history” • Confusion between research as a process and the methodology employed by separate academic disciplines in collecting and processing data • Genuine research follows the scientific method and exhibits the EIGHT characteristics

10. Research Planning Versus Research Methodology • Data vary widely, therefore cannot employ the same methodology, e.g. blood cells vs historical facts • In planning, the researcher must choose a viable research problem and consider the nature of data and the feasible means of collecting and interpreting those data • Comparing the brain wave patterns of gifted and nongifted children may be an engaging project for research, BUT consider the following issues:

11. Research Planning Versus Research Methodology • Who are willing to participate? • Do we have the equipment? • If so, do we have the technical skills to use it? • Are we able to interpret the tracings? • Do we know how to draw conclusion? • It must be a practical research – something that we must be able to handle

12. General Criteria for a Research Project • Universality – Could be carried out by any competent person other than the researcher, same results without prejudice to project and validity • Replication – Should be repeatable by other researchers under the same circumstances, within identical parameters and achieve comparable results • Control – All researches are conducted within an area sealed off by given parametric limitations

13. Criteria for Research Project • Measurement – Data should be susceptible to measurement. Easily accomplished in the physical sciences. In humanities and social sciences research, it is more difficult to quantify, measure, or evaluate • Many research efforts are wasted by going-off HALF-PREPARED • Must bring all together, early inventory of resources, your problems, and the source of data, in order to save time, money and effort

14. Data: Their Relationship to Design and Methodology • Without facts (data), inductive reasoning vanishes and scientific method collapses • The lifeblood of research • Data is a Latin verb dare, meaning “to give” – give information, impression or other factual data to an observer • Facts from Latin word facere, meaning “to make” – what the situation makes or manifests to the observer

15. What is Data? • Data are manifestation of the truth rather than the truth itself – no one has ever look upon truth itself – pure, undisguised, naked Truth • Data are merely representative, intermediate, elusive surrogate of Truth – reflects Truth as a mirror which reflects sunlight • Researchers are like those who live in dungeon seeing a beam of sunlight passes across the floor and get the idea of what the sun must be like

16. What are Data? • Factual dungeon – never be able to see the original source of the data – see what other people do – the behaviours they exhibit, the things they create, and the effects of their actions on others. But the actual people “inside” – those individuals we shall never know! • Research seeks, through data, what is true absolutely – the complete meaning of the data • Experienced researchers are constantly aware that the ultimate goal of research (Truth) is forever just beyond what is represented by the data, and, hence, just beyond human grasp

17. What are Data? • The relationship between data and the ultimate truth is shown in the diagram - various states of data • Realm of ultimate truth, the most inaccessible – can be approached by means of passing through two intermediate areas labeled The Realm of the Data

18. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? The Realm of the Inquisitive Mind of the Researcher o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o The Barrier of the Human Senses Skills of Reading and Writing, Language Capability, and the Channels of Communication The Region of the Secondary Data The Realm of the Data The Dividing Line Between Primary and Secondary Data      The Region of the Primary Data The Impenetrable Barrier Beyond Which Lies the Absolute Truth THE REALM OF ABSOLUTE TRUTH

19. Data are Transient and Ever Changing • Data are not only elusive but also transient – exist in a split second • Example, an opinion survey on a population – tomorrow’s result will not be the same as now, people move out, move in, died, …. • Pretest and posttest conducted by a teacher in a class may not be the same class at all in six weeks, they have been growing, learning, maturing, changing

20. Data are Transient and Ever Changing • Example, asking people about their interests – can we be sure what they tell us is true? • They may answer honestly that represents the truth as they perceived it at that instant • May change because of factors: age, maturity, newscast, or items in newspaper • Researcher should recognize that even the most reliable, refined, carefully controlled data may have elusive (hard to find, describe or remember) quality – very volatile, evaporate

21. Primary Data • Lie closest to the source of The Absolute Truth underlying the phenomenon • Reflect truth more faithfully than any other approach – most valid, most illuminating, the most truth-manifesting • Impenetrable barrier exists between the Realm of Absolute Truth and the region of primary data • A beam of sunlight does not compare with seeing the sun itself

22. Car Accident • I witnessed the full occurrence of a car veer off the highway and careen into the ditch • The driver told me that he had no awareness of the possibility of an accident • Neither of us will ever be able to determine the absolute truth underlying the accident • What are the factors? Momentary seizure? Mechanism imperfections? • Answers lie beyond an impenetrable barrier, may never be known

23. Secondary Data • Beyond the region of primary data lies the region of secondary data • In the dungeon – not a direct beam of sunlight but simmering light upon the wall, reflected by mirror, distorted by imperfections • In the case of the car accident, I wrote an account stating precisely what I observed and confirmed by the driver – readers of the next day’s newspaper get the similar “reflection of the sunlight”

24. Barriers of The Truth • The data/facts are of necessity distorted albeit by the channels of communication, my writing skills, reading skills, and the inability of language to produce every detail • Many more barriers than in the diagram • The acuity of human senses • The sensitivity of instruments • The failure of language to communicate thought exactly • The inability of two persons to witness the same event and report it precisely as duplicate accounts

25. Unwarranted Conclusion • Even in exuberant moment of discovery, no one has ever glimpsed the Absolute Truth, nor can we come to a knowledge of the data that reflect that Truth except through the gross and shadowy channels of our dull and imperfect senses • Be cautious and respect for words such as perhaps, it seems, one might conclude and itwould appear to be

26. Criteria for Admissibility of Data • Not all data are acceptable – can be defective – affect the validity of the conclusion • Stem from imperfections and irregularities of nature • Research should be replicable under precisely the same conditions • To regulate the precision of conditions, criteria must be adopted, limits established, standards set up that all data must meet

27. Criteria for Admissibility of Data • Non-conformance excluded • These controls are easy in sciences • Standardize data, admit only compliance – can control the research effort and conclude with greater certainty • The criteria must be set forth clearly in proposal as well as report to ensure integrity of research • So the consumer can come to an intelligent understanding of what is being studied

28. Identifying Appropriate Measurement Instruments • Sometimes we will be able to use one or more existing measurement instruments OR have to develop our own instruments • Measurement instruments provide a basis on which the entire research effort rests • Describe any instrument used in explicit, concrete terms • Provide evidence that the instruments you use have a reasonable degree of validity and reliability for your purposes • Validity = the extent to which the instrument measures what it is actually intended to measure • Reliability = the extent to which it yields consistent results when the characteristic being measured hasn’t changed

29. Determining the Validity of Measurement Instruments Face validity • The extent to which, on the surface, an instrument looks like it is measuring a particular characteristic • Useful for ensuring the cooperation of people who are participating in a research study • Since it relies entirely on subjective judgment, it is not, in and of itself, terribly convincing evidence that an instrument is truly measuring what the researcher wants to measure

30. Determining the Validity of Measurement Instruments Content validity • The extent to which a measurement instrument is a representative sample of the content area (domain) being measured • Used to assess people’s achievement in some area – for instance, the knowledge they have learned during classroom instruction or the job skills they have acquired in a rehabilitation program • High content validity means that its items or questions reflect the various parts of the content domain in appropriate proportions and requires the particular behaviours and skills that are central to that domain

31. Determining the Validity of Measurement Instruments Criterion validity • The extent to which the results of an assessment instrument correlate with another, presumably related measure (the criterion) • Example, a personality test designed to assess a person’s shyness or outgoingness (introvert or extrovert) has criterion validity if its scores correlate with other tests of introversion versus extroversion • An instrument designed to measure a salesperson’s effectiveness on the job should correlate with the number of sales the individual actually makes during the course of a business day

32. Determining the Validity of Measurement Instruments Construct validity • The extent to which an instrument measures a characteristic that cannot be directly observed but must instead be inferred from patterns in people’s behaviour (such a characteristic is a construct) • Motivation, creativity, racial bias, bedside manner – all of these are constructs, in that none of them can be directly observed and measured • Sometimes there is universal agreement that a particular instrument provides a valid instrument for measuring a particular characteristic • We could all agree that a ruler measures length, a thermometer measures temperature, and a barometer measures air pressure • But whenever we do not have such universal agreement, we must provide evidence that an instrument we are using has validity for our purpose

33. Determining the Validity of Measurement Instruments • To demonstrate that the measurement instruments used by researchers have validity for their purposes, the following examples are used: - A multitrait-multimethod approach Two or more different characteristics are each measured using two or more different approaches. The different measures of the same characteristic should be highly correlated. The same ways of measuring different characteristics should not be highly correlated - A table of specifications To construct a measurement instrument that provides a representative sample of a particular content domain – in other words, to establish content validity – the researcher often constructs a two-dimensional grid (table of specifications) listing the specific topics and behaviours that reflect achievement in the domain. In each cell of the grid, the researcher indicates the relative importance of each topic-behaviour combination. He or she then develops a series of tasks or test items that reflects the various topics and behaviours in appropriate proportions - Judgment by a panel of experts Several experts in a particular area are asked to scrutinize an instrument to ascertain its validity for measuring the characteristic in question Although none of the approaches just described guarantees the validity of a measurement instrument, each one increases the likelihood of such validity.

34. Determining the Reliability of Measurement Instruments • Interrater reliability – the extent to which two or more individuals evaluating the same product or performance give identical judgments • Internal consistency reliability – the extent to which all the items within a single instrument yield similar results • Equivalent forms reliability – the extent to which two different versions of the same instrument (e.g., “Form A” and “Form B” of a scholastic aptitude test) yield similar results • Test-retest reliability – the extent to which the same instrument yields the same result on two different occasions

35. Reliability and Validity • A researcher can enhance the reliability of a measurement instrument in several ways. 1) The instrument should always be administered in a consistent fashion; in other words, there should be standardisation in use of the instrument from one situation or person to the next 2) To the extent that subjective judgments are required, specific criteria should be established that dictate the kinds of judgments the researcher makes 3) Any research assistants who are using the instrument should be well trained so that they obtain similar results

36. Reliability and Validity • We can measure something accurately only when we can also measure it consistently • In other words, in order to have validity, we must also have reliability • The more valid and reliable our measurement instruments are, the more likely we are to draw appropriate conclusions from the data we collect and, thus, to solve our research problem in a credible fashion

37. Linking Data and Research Methodology • Data are like ore – contain pieces of the truth but are in a rather unrefined state • To extract meaning from the data, we employ what is commonly called research methodology (RM) • Data and methodology are inextricably interdependent • The RM to be used for a particular research problem must always take into account the nature of the data that will be collected in the resolution of the problem • RM is merely an operational framework within which the facts are placed so that their meaning may be seen more clearly

38. Data and RM • Different questions yield different types of information • Different research problems lead to different research designs and methods, which in turn result in the collection of different types of data and different interpretations of those data • Many kinds of data may be suitable only for a particular methodology

39. Data and RM • To some extent, the data dictate the research method • Historical data from written records of past event - cannot extract meaning by laboratory experiment. An experiment is simply not suited to the nature of the data • Not only a true experiment constitutes a “research”

40. Nature of Data • Two types, writings and observations • Written records and accounts – of past happenings • Observation for whose transmission description is the best vehicle - at the scene of occurrence • Observation that are quantified and exist in the form of numerical concepts – in the language of mathematics • Observation of certain differences and likeness that arise from comparison or contrast of one set of observations with another set of similar observations

41. Research Methodology (RM) • Four kinds of data demand four discrete and different research methodologies • The data dictate the RM • Unless fits a given methodology, it fails to be a research

42. Research Methodology (RM) • 1-1 correspondence between data and RM • Historical method – for documentary data or literary • Descriptive survey method (normative survey method) – data from observational situation – physically observed or through questionnaire or poll techniques • Analytical survey method – quantitative data that need statistics to extract their meaning • Experimental method – appropriate for data derived from experimental control or pretest-posttest design, two separate groups or one group from which data are derived at two separate intervals

43. Quantitative vs. Qualitative Research • Quantitative research – is used to answer questions about relationships among measured variables with the purpose of explaining, predicting, and controlling phenomena. This approach is sometimes called the traditional, experimental, or positivist approach • Qualitative research – is used to answer questions about the complex nature of phenomena, often with the purpose of describing and understanding the phenomena from the participants’ point of view. This approach is also referred to as the interpretative, constructivist, or postpositivist approach

44. Quantitative vs. Qualitative Research • Both approaches involve similar processes (e.g., formation of one or more hypotheses, review of the related literature, collection and analysis of data). Yet these processes are often combined and carried out in different ways, leading to distinctly different research methods • Quantitative researchers usually start with a specific hypothesis to be tested, isolate the variables they want to study, control for extraneous variables, use a standardized procedure to collect some form of numerical data, and use statistical procedures to analyse and draw conclusions from the data

45. Quantitative vs. Qualitative Research • Qualitative researchersoften start with general research questions rather than specific hypotheses, collect an extensive amount of verbal data from a small number of participants, organise those data into some form that gives them coherence, and use verbal descriptions to portray the situation they have studied

46. Qualitative vs. Quantitative Research • A quantitative study usually ends with confirmation or disconfirmation of the hypotheses that were tested • A qualitative study is more likely to end with tentative answers or hypotheses about what was observed. These tentative hypotheses may form the basis of future studies (perhaps quantitative in nature) designed to test the proposed hypotheses • In this way,qualitative and quantitative approaches represent complementary components of the research process – appropriate for answering different kinds of questions • As a result, we learn more about the world when we have both quantitative and qualitative methodologies

47. Qualitative vs. Quantitative Research • Summary of Qualitative vs. Quantitative Approaches • Please refer to Table 5.1 on page 96 of the textbook • Deciding whether to use a quantitative or qualitative approach • Please refer to Table 5.2 on page 106 of the textbook

48. Other RMs • Action Research • Case and Field Study Research • Correlational Research • Developmental Research • Ex Post Facto or Causal-Comparative • Quasi-Experimental Research

49. Other RMs • Please refer to Table 5.3 on page 108 of the textbook