Work in the 21 st Century Chapter 2

1. Work in the 21st CenturyChapter 2 Research Methods and Statistics in I-O Psychology

2. Module 2.1:Science and Research • What is science? • Approach that involves the understanding, prediction, and control of some phenomenon of interest • Science has common methods • Science is a logical approach to investigation • Based on a theory, hypothesis, or basic interest • Science depends on data • Gathered in a laboratory or the field

3. Common Methods (cont'd) • Research must be communicable, open, & public • Research published in journals, reports, or books 1) Methods of data collection described 2) Data reported 3) Analyses displayed for examination 4) Conclusions presented

4. Common Methods (cont'd) • Scientists set out to disprove theories or hypotheses • Goal: Eliminate all plausible explanations except one • Scientists are objective • Expectation that researchers will be objective & not influenced by biases or prejudices

5. Role of Science in Society • Expert witnesses in a lawsuit • Permitted to voice opinions about organizational practices • Often a role assumed by I-O psychologists

6. Module 2.1 (cont'd) • Why do I-O psychologists engage in research? • Better equip HR professionals in making decisions in organizations • Provide an aspect of predictability to HR decisions

7. Common Research Designsin I-O Psychology • Experimental • Random assignment of participants to conditions • Conducted in a laboratory or the workplace • Non-experimental • Does not include manipulation or assignment to different conditions • 2 common designs: • Observational design: Observes and records behavior • Survey/Questionnaire design (most common) • Quasi-experimental • Non-random assignment of participants to conditions

8. Methods of Data Collection • Quantitative methods • Rely on tests, rating scales, questionnaires, & physiological measures • Yield results in terms of numbers

9. Methods of Data Collection: Qualitative & Quantitative Research • Qualitative methods • Include procedures like observation, interview, case study, & analysis of written documents • Generally produce flow diagrams & narrative descriptions of events/processes • Quantitative methods • Rely on tests, rating scales, and physiological measures • Yield numerical results

10. Quantitative & Qualitative Research (cont’d) • Not mutually exclusive • Triangulation • Examining converging information from different sources (qualitative and quantitative research).

11. Generalizability in Research Generalizability: • Application of results from one study or sample to other participants or situations • The more areas a study includes, the greater its generalizability • Every time a compromise is made, the generalizability of results is reduced

12. Figure 2.1: Sampling Domainsfor I-O Research

13. Control in Research • Experimental control • Eliminates influences that could make results less reliable or harder to interpret • Statistical control • Statistical techniques used to control for the influence of certain variables

14. Ethical Behavior inI-O Psychology • Ethical standards of the APA • SIOP book of 61 cases (Lowman, 1998) • Cases illustrate ethical issues that are likely to arise in I-O psychology • Joel Lefkowitz (2003) published a recent book on values and ethics in I-O psychology

15. Module 2.2:Data Analysis • Descriptive statistics • Summarize, organize, describe sample of data Frequency Distribution: • Horizontal axis = Scores running low to high • Vertical axis = Indicates frequency of occurrence

16. Describing a Score Distribution • Measures of central tendency • Mean • Mode • Median Ryan McVay/Getty Images

17. Describing Score Distribution (cont'd) • Variability • Standard deviation • Lopsidedness or skew • Mean is affected by high or low scores, median is not • Mean pulls in direction of skew Ryan McVay/Getty Images

18. Descriptive Statistics:Two Score Distributions (N = 30) Figure 2.2 Two Score Distribution (N = 30)

19. Two Score Distributions (N = 10) Figure 2.3. Two Score Distributions (N = 10)

20. Inferential Statistics • Aid in testing hypotheses & making inferences from sample data to a larger sample/population • Include t-test, F-test, chi-square test

21. Statistical Significance • Defined in terms of a probability statement • Threshold for significance is often set at .05 or lower • Significance refers only to confidence that result is NOT due to chance, not strength of an association or importance of results.

22. Statistical Power • Likelihood of finding statistically significant difference when true difference exists • The smaller the sample size, the lower the power to detect a true or real difference

23. Figure 2.4: Scatterplot of Test Scores and Training Grades Positive Linear Correlation

24. Concept of Correlation (cont'd) • Scatterplot • Displays correlational relationship between 2 variables • Regression • Straight line that best “fits” the scatterplot and describes the relationship between the variables in the graph

25. Correlation Coefficient • Statistic or measure of association • Reflects magnitude (numerical value) & direction (+ or –) of relationship between 2 variables • Ranges from 0.00 and 1.00

26. Correlation Coefficient • Positive correlation → As one variable increases, other variable also increases & vice versa • Negative correlation → As one variable increases, other variable decreases & vice versa

27. Scatterplots of Various Degrees of Correlation Figure 2.6. Scatterplots Representing Various Degrees of Correlation

28. Curvilinear Relationship • If correlation coefficient is .00, one cannot conclude that there is no association between variables • A curvilinear relationship might better describe the association

29. Figure 2.7: An Example of a Curvilinear Relationship

30. Multiple Correlation • Multiple correlation coefficient • Overall linear association between several variables & a single outcome variable

31. Meta-Analysis • Statistical method for combining results from many studies to draw a general conclusion • Statistical artifacts • Characteristics of a particular study that distort the results • Sample size is typically the most influential statistical artifact

32. Module 2.3: Interpretation through Reliability and Validity • Reliability • Consistency or stability of a measure • Test-retest reliability • Calculated by correlating measurements taken at Time 1 with measurements taken at Time 2

33. High and LowTest-Retest Reliability Figure 2.8 Examples of High and Low Test-Retest Reliability: Score Distributions of Individuals Tested on Two Different Occasions

34. Reliability (cont'd) • Equivalent forms reliability • Calculated by correlating measurements from a sample of individuals who complete 2 different forms of same test • Internal consistency • Assesses how consistently items of a test measure a single construct

35. Reliability (cont'd) • Inter-rater reliability • Can calculate various statistical indices to show level of agreement among raters • Values in the range of .70 to .80 represent reasonable reliability • Generalizability theory • Simultaneously considers all types of error in reliability estimates

36. Validity • Whether measurements taken accurately & completely represent what is to be measured • Predictor • Test chosen or developed to assess identified abilities or other characteristics (KSAOs) • Criterion • Outcome variable describing important performance domain

37. Figure 2.9: Validation Process from Conceptual and Operational Levels

38. Criterion-Related Validity • Correlate a test score (predictor) with a performance measure; resulting correlation often called a validity coefficient • Predictive validity design • Time lag between collection of test data & criterion data • Test often administered to job applicants

39. Criterion-Related Validity (cont'd) • Concurrent validity design • No time lag between collection of test data & criterion data • Test administered to current employees, performance measures collected at same time • Disadvantage: No data about those not employed by the organization

40. Content-Related Validity • Demonstrates that content of selection procedure represents adequate sample of important work behaviors & activities or worker KSAOs defined by job analysis • I-O Psychologists can use incumbents/SMEs to gather content validity evidence

41. Construct-Related Validity • Investigators gather evidence to support decisions or inferences about psychological constructs • Construct - concept or characteristic that a predictor is intended to measure; examples include intelligence, extraversion, and integrity

42. Figure 2.10: A Model for Construct Validity

43. Figure 2.11: Construct Validity Model of Strength and Endurance Physical Factors