Survey Research Methods

1. Survey Research Methods Presentation to the class of CIS 9280 “Quantitative Methods in IS Research” at Georgia State University By Punit Ahluwalia Date: Oct 23rd2002

2. History • Systematic enumerations used to conduct census for centuries • Social surveys between late 1800s and 1920s on conditions of urban poor in many cities in USA, Britain, Germany and other countries • Resembled modern surveys but did not involve sampling • Late 1940s – called early adulthood in life of survey methods [Ornstein]: basic elements of questionnaire design, sampling, interviewing and data analysis came together as a method of social research

3. History Cont.. • Incremental evolution of questionnaire design paralleled by introduction of computers accelerating generation of cross tabulations and facilitating factor analysis and regression analysis. • Early 1970s: Criticism that the method was empiricist and untheoretical led to project by US national Academy of Science known as CASM (Cognitive aspects of survey methodology) • Lazarsfeld, Gallup and Centril are identified as inventors of modern survey research method. • Led by applications in social and political science research, survey as research method is now established across many disciplines. • An introduction about surveys and a repository of survey instruments on IS World website http://www.ucalgary.ca/~newsted/surveys.html

4. Some statistics of survey studies in MIS • Between 1980 and 1990[Pinsoneault and Kraemer] • Surveys in exploratory or descriptive studies • Before 1985 :70% After 1985 :54% • Surveys in explanatory studies • Before 1985 :30% After 1985 :46% • Longitudinal Studies • 3 out of 122 (2%) • Multiple research methods • 6 out of 122 (5%) • Pre-test • 93 out of 122 (76%) either did not pre-test the questionnaire or did not report about it. • Response rate • 90 out of 122 studies had response rate less than 51% • Exploratory and descriptive studies are of moderate to poor quality overall. Explanatory studies are of good quality.

5. Surveys with Class A instruments This slide was used with permission from the author)

6. Technically acceptable instruments • A Technically Acceptable Survey Instrument (Class A): Zmud and Boynton (1991) considered that an instrument met this criterion if it had at least the four following characteristics: • Its construct(s) were each measured by multiple items. • An attempt was made to assess the instrument's psychometric properties (i.e. reliability and validity). • Its use was described in a referred journal. • It could be reconstructed.

7. Definition • Epistemologically surveys provide one way of obtaining and validating knowledge • Webster’s Dictionary defines a survey as “a general study or inspection” • A survey is a means of "gathering information about the characteristics, actions, or opinions of a large group of people, referred to as a population" [Pinsonneault and Kramaemer]

8. Survey Research • Survey for research is distinctly different • Focus on surveys that are conducted to advance scientific knowledge, which we refer to as survey research.

9. Distinctiveness of survey research • The purpose of the survey is to produce quantitative descriptions of some aspects of the studied population.. requires standardized information from and/or about the subjects being studied • relationships between variables • projecting findings descriptively to a predefined population • The main way of collecting information is by asking people structured and predefined questions. Their answers constitute the data to be analyzed. • Information is generally collected about a fraction of the study population--a sample--but it is collected in such a way as to be able to generalize the findings to the population-such as service or manufacturing organizations, line or staff work groups, MIS departments, or various users of information systems.

10. Techniques • Survey research combines three techniques • Collection of answers to standardized questionnaires • Random sampling from a known population • Statistical analysis of a quantified representation of the survey answers

11. Research methods in IS • Case Study • Examination of phenomena in natural setting • No control over phenomena • Research variables may or may not be defined • Laboratory experiments • Controlled setting • Researcher manipulates independent variables to observe effects on dependent variables • Survey • Examination of phenomena in natural setting • Clearly defined independent and dependent variables • Phenomena of interest occur in current time or in recent past • Control of Research variables is not desirable

12. Purposes • Exploratory • Little is known about a population • Further information is desired about research variables • Prelude to a costlier, larger research endeavors • Descriptive • Focus on who, what, when, how • Existence of opinions and attitudes • Explanatory • Cause and effect, focus on why • Reasons for existence of facts and opinion is of interest

13. Key problems • Each focus makes different demands of the data which are to be collected. Three key problems : • Measurement • Control • Representation

14. Survey stages • Clarify the research problem • Identify the practical constraints • Decide what type of survey design is appropriate in light of 1 and 2 • Decide what type of sample design is most appropriate • Decide on the size of sample • Plan sampling procedures • What form of data collection • Design the questions and construct or select any tests to be used • Undertake the fieldwork and data collection • Process the data • Analyze the data and interpret your results • Disseminate your findings

15. Clarify the research problem • Move from general idea to focused topics • Degree to which study meets its aims is determined will be determined by relevance and completeness of research questions addressing the research problem

16. Empirical to theoretical mapping This slide was used with permission from the author

17. Identify practical constraints • Time • Finance • Personnel • Research sponsor • Specification of research topic • Constraints within which design must operate • Dissemination of findings • Effect on the field credibility of the survey

18. Decide type of survey • Cross-sectional or one-off • Longitudinal or time series ( collection of data at two or more points in time) • Before and after studies • Panel studies • Combination of Cross-sectional and longitudinal

19. Sample design • Decide which type of sample design is most appropriate • Probability samples • Simple random samples • Cluster samples • Non-Probability samples • Quota samples • Volunteer samples • Convenience samples • Snowball samples • Purposive sampling

20. Errors • Sampling errors • Difference between the results you will get from a full census and the results you will get from a sample • Non-Sampling error • Systemic • Random

21. Sample Size • Inevitably a compromise between methodological imperative and practical constraints • Determined by level of sampling error • How about Non-response? • Formula [Calder Judith, 1998] n = (desired confidence level)2 * (s.d) 2 /desired level of precision) 2

22. Decide form of data collection • Questionnaires • Interviews • Observation

23. Design the questions • Presentations by: • Stacy and Adriane • Ricardo and JJ

24. Data Collection • Some steps • Organize in advance • Agree access to your respondents • Collect the data • Monitor progress • Chasing up • Quality of the process of fieldwork and data collection • Report honest and reflect on fieldwork in research reports[Calder Judith]

25. Process and transform data • Some issues • Loss of detail due to coding • Deal with non-responses • 0 1 4 3 2 5 4 • 1 4 3 X 2 4 3 • 2 3 1 4 5 3 2

26. And Finally • Analyze the data and interpret your results • Does the data make sense? • Beware of spurious patterns • Compare like with like • Always allow role of chance in your findings • Present and disseminate your findings

27. Benefits • Theoretical propositions can be tested in an objective fashion • Enable generalizations • Easy to use (although may be an illusion) • Established and accepted research method across multiple fields • Aided by statistical tools • Immensely helped by advancement computing • Strong in external validity if sampling errors are minimized.

28. Weaknesses • Difficulty in obtaining truly random sampling • Low response rates. Business and IT staff are inundated with surveys • Weak linkages between units of analysis and respondents, and • Over-reliance on cross-sectional surveys where longitudinal surveys are really needed • single-method designs where multiple methods are needed • Inappropriateness • Different results with behavioral observation and self report

29. Usefulness • For Positivist standpoint • Good for providing actual, observed values against which to compare the predicted values • Not good for telling us what the theory or even the variables are in the first place • From Interpretive standpoint • Good for complimenting other sources of data - documents, participant observations, conversations • Not Good for single objective truth Ref: Wynne Chin (1996)

30. Attributes • Report the approach used to randomize or select samples • Report a profile of sample's frame • Report characteristics of the respondents • Use a combination of personal, telephone and mail data collection • Append the whole or part of the questionnaire • Adopt a validated instrument or perform a validity or reliability analysis • Perform an instrument pretest • Report on response rate • Perform a statistical test to justify the loss of data from non-respondents Ref. : Grover,Lee and Durand, Information and Management (1993)

31. Pragmatism • Interests of academic and governmental researchers are different.[Ornstein] • Governmental Researchers • Concerned with employment, industry, health etc. • Bias, systemic over or under estimation of population characteristics does not diminish with sample size, and is more of a concern than sampling error. • Push in direction of careful measurement and doing fewer things more exactly • Academic Researchers • Concerned with relationships between variables than estimates of population characteristics. • Smaller samples and differences in sub-groups – academics are more concerned with sampling error than bias. • Work in shorter time frames • Less concerned with making small mistakes if their research goals can be achieved.

32. References • Survey research : Ornstein, Michael, Current Sociology, V46 October 1998 • Survey Research methods: Gail Hackett, Personnel and Guidance Journal, May 1981 • Survey Research methodology in management information Systems: An assessment: Alain Pinsonneault and Kenneth L. Kraemer: Journal of Management Information System Fall 1993 • Research methods for the study of conflict and cooperation: The American Sociologist, 1976, Vol. 11, November • Survey Research methods Judith Calder, Medical education 1998, 32 636-652 • Research methods in Organizational behavior: Stone Eugene F., Scott Foresman and company

33. References • Newsted, Chin, Ngwenyama, and Lee, 1996 International Conference on Information Systems • Zmud, R. W., and A. C. Boynton, "Survey Measures and Instruments in MIS: Inventory and Appraisal." In K. Kraemer (ed.), The Information Systems Research Challenge: Survey Research Methods, Boston: Harvard Business School, 1991, pp 149-180. [has a list of organization-related instruments, too]