Testing Effects across Subpopulations

Multiple-group evaluation (MGA) is a statistical technique that enables researchers to research differences across subpopulations, or demographic segments, by enabling specification of structural equations models (SEMs) with group-specific estimates or with equal estimates across groups.

Differences in means, regressions, loadings, variances, and covariances of variables might be investigated using MGA, as all these parameters might be modeled in SEM. Thus, regardless that other modeling techniques (e.g., evaluation of variance or regression with interaction effects) make it possible to research the role of a grouping variable, those techniques are less flexible than MGA in SEM.

Figure 1. General overview of multiple-group evaluation and the strategy for making inferences. Image by creator.

Anytime there may be interest in exploring group differences, MGA generally is a helpful tool. When data are gathered on individuals, groups are most frequently defined based on aspects with few levels (e.g., gender, ethnicity, occupation, family status, health status, etc.) but will also be defined based on a wide range of other aspects depending on the sector, data, and analytic context. Some examples of questions that might be answered with MGA in just a few different fields are,

Consumer research

• Is product satisfaction (or quality) different across demographic segments?

People analytics

• Is worker performance (or motivation) equal across company branches or divisions?

Health care

• Do patient reported outcomes differ based on drug manufacturer?

Marketing

• Is recent marketing campaign effective at increasing brand status in several geographical areas?

Psychology

• Are there cross-cultural differences in emotional experience?

Education

• Is growth in academic achievement equal across females and males?

All questions listed above involve variables which are unobserved (e.g., satisfaction, performance, etc.), also generally known as latent variables. Because these variables can’t be observed directly, they’re difficult to measure.

Figure 2. Comparing measurement of unobserved (latent) versus observed variables. Image by creator.

One such difficulty is that different groups can have different conceptualizations of those variables. Ask yourself:

What is satisfaction?

What is good performance?

Is it likely that your responses to those questions could be different than those with different life experiences?

Repeatedly, the reply is yes.

Thankfully, we are able to test empirically whether different groups conceptualize latent variables in an identical way. This test is carried out with MGA within the SEM framework and is generally known as factorial invariance (aka measurement invariance). Factorial invariance tests are critical to make sure comparisons across groups are valid; subsequently, these tests should be done prior to comparing regressions or means across groups (aka structural parameters) if latent variables are present.

Figure 3. The challenge of modeling unobserved variables is that they won’t be measuring the identical thing across subpopulations. Image by creator.

To check for differences in parameters across groups, researchers normally fit SEMs with and without equality constraints across groups. Then, the 2 resulting models are compared using a likelihood ratio test (equivalently, a chi-square difference test) and differences in other fit statistics (e.g., the comparative fit index and root mean square error of approximation) to evaluate whether imposing constraints produce statistically significant worsening of model fit. If the fit of the model doesn’t worsen significantly, then the model with equality constraints is retained, and one concludes that the populations into account don’t differ significantly on the parameter(s) tested. In contrast, if the fit of the model worsens significantly, the model without constraints (i.e., where each group is allowed to have its own estimate(s)) is retained, and one concludes that the populations into account differ significantly on the parameter(s) tested.

The figure below illustrates the strategy behind MGA in a two-group example where a straightforward linear regression is fit. This figure shows equality constraints placed on one parameter. Model 1 has zero degrees of freedom (i.e., it’s fully saturated), whereas Model 2 has one degree of freedom resulting from the equality constraint. These models are compared based on the difference of their chi-squares, which can be chi-square distributed with degrees of freedom equal to at least one (the difference between degrees of freedom across models). A less specific test might be conducted by placing equality constraints on multiple parameters at a time.

Figure 4. Strategy behind MGA in a two-group example with a straightforward linear regression. Image by creator.

SEMs were developed as confirmatory models. That’s, one devises hypotheses, translates them right into a testable statistical model, and inferences are used to find out if the information support the hypotheses. This approach can be applied in MGA and is critical to avoid large type I error rates, which result in finding statistical effects that should not truly present within the population(s) of study. For that reason, conducting all-possible comparisons across groups is not advisable.

Disclaimer: The paragraphs below are for methodologists that want to deepen their understanding of MGA. This section assumes readers understand the full-information maximum likelihood estimator. Furthermore, the steps outlined listed here are just for explaining the logic behind MGA. In point of fact, conducting MGA with these steps could be inefficient because statistical software should leverage algorithms that simplify this process.

The estimation of MGA shouldn’t be different from that of a straightforward SEM with missing data. In a normal implementation of MGA-SEM, users submit the information they need to research together with a grouping variable, which indicates the group that every statement belongs to. An easy data manipulation step — using the grouping variable — is required to establish the evaluation for multiple groups. The figure below illustrates the information which are supplied for evaluation and the restructuring of information for MGA.

Figure 5. Data inputted by users and data after restructuring for doing multiple-group evaluation. Image by creator.

The resulting data can now be used with full information maximum likelihood because the estimator to make sure all rows in the information are submitted for evaluation despite there being missing data. A number of convenient results from the restructured data are:

• The log likelihood of any given row is just influenced by the non-missing cells, such that adding the log likelihood of all of the ‘Group 0’ rows yields the log likelihood for that group. Similarly, adding the log likelihood of all ‘Group 1’ rows yields the log likelihood for group 1. Each group’s log chances are used to estimate a chi-square statistic for the general model, which quantifies the misfit for every group.
• The pattern of missing values prohibits estimation of any parameter across the groups’ variables (e.g., the covariance of Var1_0 and Var1_1 shouldn’t be estimable), which is inconsequential because MGA is anxious with comparison of effects across groups fairly than cross-group estimates.
• ‘Vanilla SEM’ allows one to set equality constraints on parameters. Thus, using the restructured data in SEM, one can specify two similar models with each group’s subset of variables, and equality constraints might be placed on equivalent parameters across the groups. To reiterate, all of this might be done in standard SEM without asking the software to conduct MGA explicitly.

Thankfully, these steps don’t should be performed by users who wish to do MGA-SEM! SEM software makes fitting multiple-group models quite simple by allowing users to specify a grouping variable. Nevertheless, doing the information manipulation (Figure 5) and using standard SEM to conduct MGA-SEM will deepen your understanding of this topic. To learn much more, take a look at the resources cited below.

Step-by-step example of applied multiple-group evaluation in JMP.

Book chapter on multiple-group evaluation for factorial (measurement) invariance:

Widaman, K. F., & Olivera-Aguilar, M. (2022). Investigating measurement invariance using confirmatory factor evaluation. Handbook of Structural Equation Modeling, 367.

Journal article on using alternative fit indices to check for invariance:

Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural equation modeling: a multidisciplinary journal, 14(3), 464–504.

This text was originally published within the JMP user community on February 27, 2023.