Using Multivariate Techniques for the Analysis of Survey Data: A Case Example

1. Using Multivariate Techniques for the Analysis of Survey Data: A Case Example Isabelle Bourgeois Natural Sciences and Engineering Research Council University of Ottawa Joint Conference of the CES and the AEA Toronto, October 26th 2005

2. Performance measurement activities in R&D often use complex techniques taken from the sciences and economics Methods are needed that capture more fully the non-economic benefits of research to support decision makers The use of multivariate analysis techniques taken from the social sciences can provide useful information based on survey results Rationale for Study

3. Industry awareness of government research and researchers Satisfaction with past interactions between industry and government Level of trust in researchers and staff Types of new processes or products introduced to market Examples of Survey Measures

4. Traditional survey data analysis relies on descriptive statistics with little exploration of the relationships that exist between variables Harman (2004) example: Use of t-tests with little interpretation and no further research Use of Sophisticated Analytical Methods

5. Multivariate analysis conducted on data from the Postgraduate Scholarship (PGS) program managed by NSERC Data collection instrument: survey administered in the summer of 2005 (N=901) Attrition: 101 Response rate: 69.6% (n=557) Survey results estimated to be accurate ±3%, 19 times out of 20 Case Example

6. Independent: gender, type of award received (master’s or doctoral), and main field of study Dependent: four seven-point scales on different aspects of the student’s experience with the PGS award Variables

7. Descriptive statistics to get a general sense of the survey results Verification of statistical assumptions most critical to multivariate analysis, normality and equality of the variance-covariance matrix (homoscedasticity) Preliminary Examination of Data

8. MANOVA (Multivariate Analysis of Variance): yields information to answer questions about the combined effect of the dependent variables MANOVA conducted on each of the four scales using the gender, award type and field variables (Tables included in the paper) Multivariate Respondent Profile

9. Field variable found significant on its own in scale A2 and in combination with other dependent variables in scales A2 and C3 Suggests that the field of study has an impact on a student’s undergraduate experiences (A2), as well as his or her graduate experiences (C3) Findings of MANOVA (1)

10. Award variable found significant on scale C2 which focused on relationship with supervisor and future career plans The award variable has an impact on the student’s relationship with supervisor and future career plans – Master’s and Ph.D. students have different experiences and expectations Findings of MANOVA (2)

11. Close relative of the MANOVA Instead of looking at the influence of the independent variables on survey responses, it predicts group membership within each independent variable by using the responses to the scale items. Findings focus on items loading most highly for each variable – these provide the best discrimination between levels of the variable (examples) Discriminant Function Analysis

12. Aside from providing more sophisticated information about the results of the survey, multivariate analysis methods also provide information about the properties of the instrument itself. Exploration of Scale Properties

13. Data summarization technique Identifies underlying, or latent, dimensions in a dataset that, when interpreted, describe the data in a smaller number of concepts The responses to the items on the PGS survey scales are hypothesized to vary as a function of one or several latent variables Exploratory Factor Analysis

14. Nine meaningful factors extracted and retained for interpretation Example (Factor 1, Learning and Skill Development): Funding from NSERC will help me to complete my degree faster Theoretical knowledge of the discipline Analytical techniques/experimental methods Use of laboratory equipment or instruments Project management Etc. Factors Extracted

15. Learning and Skill Development Supervision Resources Career Orientation No Debt Encouragement Collaborative Experiences Debt Load Job Prospects Factors Extracted

16. Reliability: Internal consistency of a scale – minimizes the contribution of random error to item scores Alpha coefficient is used to measure internal consistency and is interpreted as a correlation Alpha coefficients calculated for each of the scales and the combined items, as well as for the Factors extracted in the EFA to see which model fits best Scale Reliability

17. Original Scales Scale A2: 0.59 Scale C1: 0.81 Scale C2: 0.77 Scale C3: 0.85 Combined Items: 0.82 Factors Factor 1: 0.84 Factor 2: 0.91 Factor 3: 0.81 Factor 4: 0.64 Factor 6: 0.50 Factor 8: 0.42 Factor 9: 0.24 Reliability Comparison

18. Easy to do using standard statistical software packages Interpretation is fairly straightforward Adds to descriptive data by highlighting the influence of the combined dependent variables on the survey responses Allows for a review of the instrument properties and provides suggestions for improvement Advantages of Multivariate Analysis