Multivariate Statistics

1. Multivariate Statistics Confirmatory Factor Analysis I W. M. van der Veld University of Amsterdam

2. Overview • PCA a review • Introduction to Confirmatory Factor Analysis • Digression: History and example • Intuitive specification • Digression: Decomposition rule • Identification • Exercise 1 • An example

3. PCA a review

4. PCA a review • Principal Components analysis is a kind of “data reduction” • Start with a set of observed variables. • Transform the data so that a new set of variables is constructed from the observed variables (full component solution). • Hopefully, a small subset of the newly constructed variables carry as much of their information as possible, so that we can reduce the number of variables in our study. • Full component solution • Has as many components as variables • Accounts for 100% of the variables’ variance • Each variable has a final communality of 1.00 • Truncated components solution • Has fewer components than variables (Kaiser criterion, scree plot, etc.) • Components can hopefully be interpreted • Accounts for <100% of the variables’ variance • Each variable has a communality < 1.00

5. PCA a review • Each principal component • Accounts for maximum available variance, • Is orthogonal (uncorrelated, independent) with all prior components (often called factors) • Full solution (as many factors as variables) accounts for all the variance. • Full solution Truncated solution

6. PCA a review • PCA is part of the toolset for exploring data. • When the researcher does not have, a priori, sufficient evidence to form a hypothesis about the structure in the data. • But anyone who can read should be able to get an idea about the structure, i.e. what goes with what. • Highly empirical or data driven technique. • Because one will always find components. • And when you find them you have to interpret them. • Interpretation of components measured by many variables can be quite complicated. Especially if you have no a priori ideas. • PCA at best suggests hypotheses for further research. • Confirmatory Factor Analysis has everything that PCA lacks, • Nevertheless, PCA is useful when the data is collected with hypothesis in mind, that you want to explore quickly. • And when you want to construct variables by definition.

7. Introduction to CFA

8. Introduction to CFA • Some variables of interest cannot be directly observed. • These unobserved variables are referred to as either: • Latent variables, or • Factors • How do we obtain information about these latent variables, if they cannot be observed? • We do so by assuming that the latent variables have real observable consequences. And these consequences are related. • By studying the relations between the consequences, we can infer that there is or is not a latent variable. • History and example.

9. Digression: History and example • Sir Francis Galton (1822-1911) • Psychologist, etc. • Galton was one of the first experimental psychologists, and the founder of the field of enquiry now called Differential Psychology, which concerns itself with psychological differences between people, rather than on common traits.  He started virtually from scratch, and had to invent the major tools he required, right down to the statistical methods - correlation and regression - which he later developed.  These are now the nuts-and-bolts of the empirical human sciences, but were unknown in his time.  One of the principal obstacles he had to overcome was the treatment of differences on measures as measurement error, rather than as natural variability. • His influential study Hereditary Genius (1869) was the first systematic attempt to investigate the effect of heredity on intellectual abilities, and was notable for its use of the bell-shaped Normal Distribution, then called the "Law of Errors", to describe differences in intellectual ability, and its use of pedigree analysis to determine hereditary effects.

10. Digression: History and example • Charles Edward Spearman (1863-1945) • Psychologist, with a strong statistical background. • Spearman set out to estimate the intelligence of twenty-four children in the village school. In the course of his study, he realized that any empirically observed correlation between two variables will underestimate the "true" degree of relationship, to the extent that there is inaccuracy or unreliability in the measurement of those two variables. • Further, if the amount of unreliability is precisely known, it is possible to "correct" the attenuated observed correlation according to the formula (where r stands for the correlation coefficient): r (true) = r (observed)/reliability of variable 1 X reliability of variable 2. • Using his correction formula, Spearman found "perfect" relationships and inferred that "General Intelligence" or "g" was in fact something real, and not merely an arbitrary mathematical abstraction.

11. Digression: History and example • M are measures of math skills, and L are measures of language skills. • He then discovered yet another marvelous coincidence, the correlations were positive and hierarchal. These discoveries lead Spearman to the eventual development of a two-factor theory of intelligence.

12. Digression: History and example g Verbal Ability Math Ability L1 L2 L3 M1 M2 M3

13. Introduction to CFA • Factor analysis is a very powerful and elegant way to analyze data. • Among others, because it is close to the core of scientific purpose. • CFA is a theory-testing model • [As opposed to a theory-generating method like EFA, incl. PCA.] • Researcher begins with a hypothesis prior to the analysis. • The model or hypothesis specifies • which variables will be correlated with which factors • which factors are correlated. • The hypothesis should be based on a strong theoretical and/or empirical foundation (Stevens, 1996).

14. Intuitive specification • xi are the observed variables, • ξ is the common latent variable that is the cause of the relations between xi. • λi is the effect from ξ on xi. • δi are the unique components of the observed variables. They are also latent variables, commonly interpreted as random measurement error.

15. Intuitive specification • This causal diagram can be represented with the following set of equations x1 = λ1ξ + δ1 x2 = λ2ξ + δ2 x3 = λ3ξ + δ3 • With: E(xi)=E(ξi)=0, var(xi)=var(ξi)=1, E(δiδj)=E(δx)=E(δξ)=0 • Only the x variables are known. • We don’t know anything about the other variables ξ and δ. • How can we compute λi? • We need a theory about why there are correlations between the x variables.

16. Digression: Decomposition rule • In Structural Equation Modeling (includes CFA) • The correlation between two variables is equal to the sum of • - the direct effect, • - indirect effects, • - spurious relations and • - joint effects between these variables.

17. Direct effect Indirect effect Spurious relation Joint effect Digression: Basic concepts

18. Digression: Decomposition rule • In Structural Equation Modeling (includes CFA) • The correlation between two variables is equal to the sum of- the direct effect,- indirect effects, - spurious relations and- joint effects between these variables. • The indirect effects, spurious relations and joint effects are equal to the products - of the coefficients along the path - going from one variable to the other - while one can not pass the same variable twice - and can not go against the direction of the arrows.

19. Intuitive specification • So the theory, according to the model, is that the correlations between the x variables are spurious due to the latent variable ξ. • ρ(x1x2)=λ1λ2,ρ(x1x3)=λ1λ3,ρ(x2x3)=λ2λ3. • Proof in the formal specification. • A simplified notation yields:ρ12=λ1λ2,ρ13=λ1λ3,ρ23=λ2λ3 • Where λ1, λ2, and λ3, are unknown, andρ12, ρ13, and ρ23, are known. • This is solvable, eg.

20. Intuitive specification • The correlations between the x variables are: • We know that: • ρ12 = .42 = λ1λ2ρ13 = .56 = λ1λ3ρ23 = .48 = λ2λ3 • λ2 = .42/λ1 =>λ3 = .56/λ1 => • λ1= √(.42*.56)/.48) = .7 • λ2 = .6 • λ3 = .8

21. Intuitive specification • Notice that this is much more advanced than PCA. • It is based upon theory. • It is straightforward • It allows the estimation of an effect of something that we have not measured!

22. Identification • What would happen if the latent variable had four indicators? • We already had:ρ12=λ1λ2 = .42ρ13=λ1λ3 = .56ρ23=λ2λ3 = .48 • Now we also have:ρ14=λ1λ4 = .35ρ24=λ2λ4 = .30ρ34=λ3λ4 = .50

23. Identification • Now we also have: ρ14=λ1λ4 = .35ρ24=λ2λ4 = .30ρ34=λ3λ4 = .50 • Of these three equations we need only one to determine the value of λ4 when we have solved λ1, λ2, λ3. • Therefore, this model is called over-identified with 2 degrees of freedom (df=2) • df= (# of correlations) – (# of parameters to be estimated) • If the number of degrees of freedom is larger than 1, a test is possible of the model parameters.

24. Identification • A test of the model • ρ14= λ1λ4 = .7*λ4 = .35 ρ24= λ2λ4 = .6*λ4 = .30ρ34= λ3λ4 = .8*λ4 = .50 • Let’s use the first equation to estimate λ4, then λ4 =.35/.7=.5 • Now we know all coefficients and two equations are not used yet. • These equations can be used to test the model. Using the idea that the observed correlation should equal the estimated correlation, or ρ(obs) – ρ(est)=0 • ρ24 – λ2λ4 = .30 – .6*λ4 = .30 – .6*.50 = 0 (=good) ρ34 – λ3λ4 = .50 – .8*λ4 = .50 – .8*.50 = 0.1 (less good) • These differences are called residuals. If these residuals are big the model must be wrong.

25. Identification • With df > 0, a test of the model is possible • With df = 0, no test of the model is possible • This is the case with a one factor model and 3 observed variables. • With df < 0, no test of the model is possible, and even the parameters cannot be estimated. • However, if the parameters of the model can be constraint, degrees of freedom can be gained, so that the model could be estimated. • The issue of identification is connected to the issue of number of unknowns versus number of equations. Although not entirely, because there is more. • Generally we leave this issue to the computer program. If the model is not identified, you will get a message.

26. Exercise 1 • Assume that E(xi)=E(ξi)=0, var(xi)=var(ξi)=1, E(δiδj)=E(δx)=E(δξ)=0 • Formulate expressions for the correlations between the x variables in terms of the parameters of the model. • ρ12, ρ13, ρ14, ρ23, ρ24, ρ34. • ρ12= λ11λ21 ρ13= λ11φ21λ32 ρ14= λ11φ21λ42 ρ23= λ21φ21λ32 ρ24= λ21φ21λ42ρ34= λ32λ42 • We can get the same result via the formal specification.

27. An example (PCA) • How does the factor model look like for these items? • We suggested last time that there should be 1 factor behind these items.

28. An example (PCA) Varimax rotated PCA solution. Claims a better interpretation. There are two PC’s, that can be interpreted as: positive (4,7,8) and negative (others) loneliness. But this is strange, because they should then be related!!!! PCA solution

29. An example (CFA) • We did not look at the correlations, but, some items were positive and some negative. • Therefore, if the same response scale is used, we expect negative correlations. And they should be found in the dark yellow cells.

30. An example (CFA)

31. An example (CFA) • This model does not fit (p=0). • So we must search for another model. • One possibility that the correlations are positive, where expected negative, is due to acquiescence. • Acquiescence is the tendency to agree (confirm), with a statement, regardless the content. • There are all kinds of psychological explanations for this response behavior of respondents, being nice, etc. • If people say agree to both negative and positive items, we will find positive correlations between those items. • Can we correct for acquiescence? • YES!

32. An example (CFA) • In order to identify an acquiescence factor, I have to split the positive and negative items into two factors. • Theoretically it makes sense if those two loneliness factors are correlated -1, because they are the same but opposites.

33. An example (CFA)

34. An example (CFA) • The result is a model that still does not fit, but much better than before. • This model however shows that most variation and covariation is due to the acquiescence factor. • The loneliness factors that remain after correction for random measurement error, and systematic error (acquiescence), are ‘pure’. • However, given the low loadings what do these items have in common with the “underlying factor”. That is the loneliness factor(s) does explain 0.2% of the variance of the best indicators (0.452).

35. An example (CFA) • Conclusion: • The PCA did not provide a theoretical sound solution. • Thinking about the items, and what some respondents do when answering questions, lead to a theory; • This theory could be tested, and produced very sensible results. • However the χ2 was not acceptable 74 (19), whereas with 19 df the χ2 should be 30.