Notes on Measurement Invariance

Notes on invariance

Testing measurement invariance in R/RStudio.

In a previous post I showed how one can quickly test measurement/structural invariance using R/RStudio, but I did not explain how to interpret the numbers. Here I give a brief overview of the process along with some interpretation. This is not meant to be detailed. Rather, this is meant to show how one might address the issue and write up some results (and to help me remember how to do so!). See the cited literature for more depth and some delightful complexity.

The goal here is to test how well our Confirmatory Factory Analysis solution applies to, in this case, 2 different groups, men and women. We will focus on the measurement per se. In other words, I am not interested in how men and women differ in terms of the relationships among latent variables (for the purpose of the CFA). Rather, I am interested in the extent to which the measured and latent variable relationships differ between men and women. In other words, do the same items load on the same latent factors for men and women in roughly the same manner?  

Standard approach to Measurement Invariance

A number of steps to testing measurement invariance have been semi-clearly outlined in the literature (Blunch, 2008; Byrne, 2016; Keith, 2015; Vandenburg & Lance, 2000), but can vary from between stats packages. These steps include testing:

1. Configural/unconstrained invariance, or whether or not the same factor model/structure applies to different groups
2. Metric (weak) invariance, which basically tests whether or not factor scores are roughly the same between groups. 
3. Scale/intercept (strong) invariance is the same as metric invariance, but with the model intercept constrained to equality between the groups. 
4. Residual invariance and whether it differs among the groups.

Incidentally, testing structural invariance can be done as well by testing the following. But you would not necessarily expect different groups to have equal means & covariances between latent factors. Nevertheless you can test: 

5. If factor variances are equal. 
6. If factor covariances are equal between groups. In other words are the factors related in the same way or not.
7. If factor means are equal among groups. 

Example 

In terms of testing measurement variance, Byrne (2016) has outlined the steps using AMOS. But as I noted in a previous post, AMOS is a bit tricky and it involves several steps. (I am exaggerating, it is fairly easy to do, but it does involve a few steps to set it up...I will post an example using AMOS eventually). You can see Byrne's book if you want to see the process in AMOS (or Blunch or Keith), or wait until I post an example (I will place an update here once I get that done). I am using the R package 'semTools' (semTools Contributors, 2016) to quickly examine measurement invariance with the 'measurementInvariance' command.

For this example, I have already tested a structural model in AMOS. After some preliminary investigation, the final model takes the following form in R. I am showing the output directly pasted from Jamovi, just to because it looks nice and you should try Jamovi.

The fit statistics are as follows:

Model Fit

Test for Exact Fit
χ²		df		p
304		100		.001

Fit Measures
								RMSEA 90% CI
CFI		TLI		SRMR		RMSEA		Lower		Upper		AIC
0.924		0.908		0.0731		0.0802		0.0700		0.0906		16792

 The model looks like this:

Ok, now we can run the invariance tests using the following code:


R code for running the CFA and the Measurement Invariance analysis is below (or here for code and outputs). You can replace my item and subscale names and your data with your own.:

> PRFQCFA.model <- ' PM  =~ PRFQ1 + PRFQ4 + PRFQ7 + PRFQ10 + PRFQ13 + PRFQ16
+             CMS =~ PRFQ15 + PRFQ12 + PRFQ9 + PRFQ6 + PRFQ3
+ IC  =~ PRFQ17 + PRFQ14 + PRFQ8 + PRFQ5 + PRFQ2
+ PRFQ6~~PRFQ9'
> fit <- cfa(PRFQCFA.model, data = TrimmedfullPRFQforRandOmegacalsandR)
> summary(fit, fit.measures = TRUE)
> measurementInvariance(PRFQCFA.model, data = TrimmedfullPRFQforRandOmegacalsandR,
+                       group = "Female1Male2", strict = TRUE)

And you get the following output and i have some annotations to explain below:

--------------------------------------------------------------------

Measurement invariance models: Model 1 : fit.configural (tests if factor structure differs) Model 2 : fit.loadings (tests if factor loadings differ) Model 3 : fit.intercepts (tests if both factor loadings AND intercepts differ) Model 4 : fit.residuals (test above plus residuals for differences) Model 5 : fit.means (tests above plus means) Chi Square Difference Test Df AIC BIC Chisq Chisq diff Df diff Pr(>Chisq) fit.configural 200 16248 16635 438.48 fit.loadings 213 16230 16569 446.96 8.4804 13 0.8109609 fit.intercepts 226 16216 16506 458.83 11.8696 13 0.5383647 fit.residuals 242 16204 16435 479.05 20.2133 16 0.2107653 Non-sig Difference test fit.means 245 16218 16438 499.29 20.2398 3 0.0001514 *** supports invariance --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Fit measures: cfi rmsea cfi.delta rmsea.delta fit.configural 0.907 0.088 NA NA fit.loadings 0.909 0.085 0.002 0.004 fit.intercepts 0.909 0.082 0.000 0.003 The small change in CFI (<.001) supports invariance. fit.residuals 0.908 0.080 0.002 0.002 Here CFI goes from .909 to .908, again support fit.means 0.901 0.082 0.007 0.002

________________________________________________

Meaning what?

Well, as seen above, we have a non-significant change in Chi-square, which examines whether adding additional constraints has any changes. Since the differences are non-significant, we have support for invariance until the means are constrained. The fit measures change little as well, with clear support for invariance until adding the residual constraint. A change in CFI of less than .001 supports invariance. At the residual test we are around .001, it goes from .909 to .908. But once means are constrained, we get RMSEA increasing and CFI increasing .002. However, we would not necessarily expect men and women to score the same on the measures.

So we conclude that we have good evidence of measurement invariance between men and women. In other words, the measure performs roughly the same between men and women.


References

Blunch, N. (2008). Introduction to Structural Equation Modelling Using SPSS and Amos. Sage

Byrne, B.M. (2016). Structural Equation Modeling With AMOS: Basic Concepts, Applications, and Programming, Third Edition. Routledge

jamovi project (2017). jamovi (Version 0.7.5.2) [Computer Software]. Retrieved from https://www.jamovi.org

Keith, T.Z. (2015). Multiple Regression and Beyond: An Introduction to Multiple Regression and Structural Equation Modeling. Routledge

semTools Contributors. (2016). semTools: Useful tools for structural equation modeling. R package version 0.4-14. Retrieved from https://CRAN.R-project.org/package=semTools

Vandenburg, R.J., & Lance, C.E., 2000. A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-69. doi:10.1177/109442810031002