Blog | Francisco Wilhelm

A workflow for automating scale scoring in R

Sun, 19 Oct 2025 00:00:00 +0000

Scoring scales is easily one of the most boring and repetitive tasks in the social sciences. In this post, we are going to learn how we can largely automate the process using R. The major advantages are both convenience, time-saving, and quick retrieval of the array of useful statistics for the items and scales in a dataset.

The R package psych features powerful functions for scoring survey scales. In this tutorial, we will use a wrapper for the scoreItems function to automate scale scoring.

Data management: Naming variables in a consistent manner

A crucial initial step is to name the variables of a dataset consistently. This will later come in handy when we automate the scale construction, because we will construct an algorithm that looks for a certain pattern in the name of variables to identify them as items. Our goal is to create variable names that make clear that a specific item belongs to a specific scale.

bad example: “happy”, “serene”, “calm”
good example: “posaffect_1"”,“posaffect_2”, “posaffect_3”

I recommend using the following pattern: “scalename_itemnumber”. For example

agr_1 for the first item of an agreeableness scale, or
neu_1 for the first item of a neuroticism scale.

Non-scale variables should not follow the same pattern in order to make it easy for the algorithm to distinguish items we want to score from variables we don’t want to score. So, it’s best not to use the “characters_number” pattern for any non-scale variable. For example,

gender_1 should be renamed to gender

Example dataset

This is the structure of our dataset:

head(df)

# A tibble: 6 × 28
agr_1 agr_2 agr_3 agr_4 agr_5 con_1 con_2 con_3 con_4 con_5 ext_1 ext_2 ext_3
<int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>
1 2 4 3 4 4 2 3 3 4 4 3 3 3
2 2 4 5 2 5 5 4 4 3 4 1 1 6
3 5 4 5 4 4 4 5 4 2 5 2 4 4
4 4 4 6 5 5 4 4 3 5 5 5 3 4
5 2 3 3 4 5 4 4 5 3 2 2 2 5
6 6 6 5 6 5 6 6 6 1 3 2 1 6
# ℹ 15 more variables: ext_4 <int>, ext_5 <int>, neu_1 <int>, neu_2 <int>,
# neu_3 <int>, neu_4 <int>, neu_5 <int>, ope_1 <int>, ope_2 <int>,
# ope_3 <int>, ope_4 <int>, ope_5 <int>, gender <int>, education <int>,
# age <int>

We use a redacted bfi dataset from psych, which consists of 25 personality self-report items taken from the International Personality Item Pool (ipip.ori.org). We have measured each of the Big Five traits with five items.

Now, instead of calculating the scores of each of these five scales manually, we are going to automate the process¹.

Extracting all items

As you may have seen, the dataset also contains three demographic variables in addition to the scale items we want to use. Thus, we first have to select and extract all variables that are scale items from our dataset. This is where we need the unique naming structure of item-type variables, that is, variables which follow the structure scalename_itemnumber.

bigfiveitems <- dplyr::select(df, matches("_\\d$|_\\d.$"))
names(bigfiveitems)

 [1] "agr_1" "agr_2" "agr_3" "agr_4" "agr_5" "con_1" "con_2" "con_3" "con_4"
[10] "con_5" "ext_1" "ext_2" "ext_3" "ext_4" "ext_5" "neu_1" "neu_2" "neu_3"
[19] "neu_4" "neu_5" "ope_1" "ope_2" "ope_3" "ope_4" "ope_5"

What we make use of here is the select function from the dplyr package. It allows to select variables based upon regular expressions ². Regular expressions allow us to describe patterns in strings, such as variable names here. The regular expression we use here tells the function to look for variable names that have an underscore ("_") followed by a single digit number (e.g., 1). Since all items follow this naming pattern, and, importantly, no non-item does, the functions quickly selects all the item variables.

Next, we automatically look for the scale names:

scalelist <- str_extract(names(bigfiveitems), "\\w+(?=_)")
scalelist <- unique(scalelist)
scalelist

[1] "agr" "con" "ext" "neu" "ope"

As we can see, this next bit of code extracts the names of the scales, again using a regular expression. We could also supply the scalelist manually, e.g. scalelist <- c("agr", "con", "ext", "neu", "ope")

We now use the scalelist to tell scoreItems from the psych package to create scale scores for all of the scales listed in scalelist. This is handled via a function called scoreItemsMulti from the franzpak package³.

Installing franzpak

If you haven’t already installed franzpak, you can do so from GitHub:

# Install remotes if you don't have it
# install.packages("remotes")
remotes::install_github("franciscowilhelm/franzpak")

Then load the package:

library(franzpak)

bigfivescores <- scoreItemsMulti(scalelist, bigfiveitems)

The beauty of this function is that it takes the names of the scales and creates not only the scale scores, but also several useful statistics such as internal reliabilities, correlation matrices and others.

The scores themselves are available in the scores element of the object.

head(bigfivescores$scores)

 agr con ext neu ope
1 4.0 2.8 3.8 2.8 3.0
2 4.2 4.0 5.0 3.8 4.0
3 3.8 4.0 4.2 3.6 4.8
4 4.6 3.0 3.6 2.8 3.2
5 4.0 4.4 4.8 3.2 3.6
6 4.6 5.6 5.6 3.0 5.0

The reliabilities are given in the alpha element of the object.

bigfivescores$alpha

 agr con ext neu ope
alpha 0.7031003 0.7273554 0.761868 0.8140709 0.6007539

A word of caution: If your scales contain reverse-worded items, scoreItemsMulti will try to automatically detect these⁴. However, in such cases you should proceed with caution and consider checking manually whether it got it right. You can do this by inspecting the $negative_index element of your object. In the example case, some items were automatically reversed. You can also overwrite the automatic scoring by manually providing key instructions in the form of a named list, formatted like in the original scoreItems.

bigfivescores <- scoreItemsMulti(scalelist, bigfiveitems, manual_keys = list(con = c("con_1", "con_2", "con_3", "-con_4", "-con_5")))

Some items were negatively correlated with total scale(s) and were (automatically) reversed.
Scales with reversed items: agr, ext, ope
Please Check $negative_index for details.

Alternative: Using scoreItemsMultiFast

For larger datasets or when you don’t need reliability statistics, franzpak also provides scoreItemsMultiFast, which is a faster alternative that uses psych::scoreFast() under the hood. Unlike scoreItemsMulti, this function:

Does not calculate Cronbach’s alpha or perform PCA
Does not automatically detect reverse-coded items
Is significantly faster for large datasets
Is suitable for cases with few observations where PCA or reliability estimation might fail

# Using scoreItemsMultiFast (no automatic reverse detection)
bigfivescores_fast <- scoreItemsMultiFast(scalelist, bigfiveitems,
 manual_keys = list(
 con = c("con_1", "con_2", "con_3", "-con_4", "-con_5"),
 ext = c("ext_1", "ext_2", "ext_3", "ext_4", "ext_5"),
 neu = c("neu_1", "neu_2", "neu_3", "neu_4", "neu_5"),
 ope = c("ope_1", "ope_2", "ope_3", "ope_4", "ope_5"),
 agr = c("-agr_1", "agr_2", "agr_3", "agr_4", "agr_5")
 ))

Important: When using scoreItemsMultiFast without manual_keys, the function will automatically find items based on naming conventions (items starting with the scale name followed by ‘.’ or ‘_’) but will assume no items are reverse-coded. If you have reverse-coded items, you must specify them using the manual_keys parameter.

Summary: The workflow

This workflow vastly speeds up the process of scale scoring and documentation of psychometrical attributes such as Cronbach’s alpha.

Here is the recommended workflow in short:

Install franzpak from GitHub using remotes::install_github("franciscowilhelm/franzpak").
Name variables such that all items follow the pattern ‘scalename_itemnumber’ (or ‘scalename.itemnumber’).
Optional: Make sure that no other variables follow the pattern of ‘characters_number’.
Create a character vector holding all scale names either automatically or manually.
Choose between scoreItemsMulti (includes reliability statistics, automatic reverse detection) or scoreItemsMultiFast (faster, no automatic reverse detection).
Use your chosen function to create the scale scores and other attributes.
If reverse-worded items are part of your scales:
- For scoreItemsMulti: Check the $negative_index element of your object to verify whether the function identified these correctly. If not, or to ensure consistency when data changes, manually specify scale keys with the manual_keys parameter.
- For scoreItemsMultiFast: You must manually specify all reverse-coded items using the manual_keys parameter, as this function does not perform automatic detection.

with just five scales, this may seem like of over-engineering. Usually however, you will have a much larger dataset, where such an approach saves a lot of time. ↩︎
More info on regular expressions in R, especially the tidyverse packages: ↩︎
The franzpak package is available on GitHub at and can be installed using remotes::install_github("franciscowilhelm/franzpak"). ↩︎
The automatic detection of reverse-worded items is accomplished by running a PCA and reverse scoring those items that load negatively on the factor. This may not always work correctly. ↩︎

Fornell-Larcker Criterion with R using lavaan

Mon, 17 Jan 2022 00:00:00 +0000

Update 2025

The Fornell-Larcker criterion has been criticized in an influential paper on discriminant validity (Rönkko & Cho, 2022). The authors have kindly created a function to make use of their suggested procedure, available in the semTools package. The function is called discriminantValidity().

Main Post

The Fornell-Larcker criterion (Fornell & Larcker, 1981, p. 41) is a popular technique to check the discriminant validity of constructs in structural equation models. The criterion states that the average variance extracted (AVE) of items by a construct (factor) should be larger than the squared correlation of the latent construct with the discriminant construct. This article shows how to compute the Fornell-Larcker criterion in R with lavaan-based SEM analyses.

We use the classic dataset also used in the lavaan examples, the Holzinger and Swineford (1939) data of mental ability test scores. The factor model consists of three intercorrelated factors (visual, textual, speed), with 9 different tests making up the indicators.

library(tidyverse)
library(lavaan)
library(semTools)

 HS.model <- ' visual =~ x1 + x2 + x3
 textual =~ x4 + x5 + x6
 speed =~ x7 + x8 + x9 '
fit <- cfa(HS.model, HolzingerSwineford1939)

The Fornell-Larcker criterion can be applied to test whether the three factors can be discriminated from each other. Let’s consider the two sides of the equation that makes up the criterion: Average Variance Extracted (AVE) and the (squared) correlations of the latent constructs.

AVE is the variance extracted of each indicator by its factor, as indicated by the squared standardized loadings, divided by the total variance of each indicator, averaged over all indicator that are specified to load on the factor. The package semTools provides the AVE of a lavaan model:

AVE(fit)

 visual textual speed
0.371 0.721 0.424

The squared correlations of the latent constructs can be computed by extracting the correlation of the latent constructs from the fitted lavaan object and squaring them.

lavInspect(fit, what = "cor.lv")^2

 visual textul speed
visual 1.000
textual 0.210 1.000
speed 0.221 0.080 1.000

To relate the AVE to the squared correlations, I have written a function that we will load next.

source("https://raw.githubusercontent.com/franciscowilhelm/r-collection/master/forn_larcker_test.R")

Let us assume we are interested in whether the visual factor can be discriminated from the textual and speed factors according to the Fornell-Larcker criterion. We supply the fitted lavaan object, as well as x (“our” construct) and y (the constructs that we want to test against) constructs to the function.

forn_larcker_test(fit, x = c("visual"), y = c("textual", "speed"))

# A tibble: 2 × 6
x y criterion latcor_sq ave_x ave_y
* <chr> <chr> <lgl> <dbl> <dbl> <dbl>
1 visual textual TRUE 0.210 0.371 0.721
2 visual speed TRUE 0.221 0.371 0.424

The function returns the names of the x and y constructs, whether the Fornell-Larcker criterion is met, the squared latent correlation of the x-y pair, as well as the AVE of X and Y. We can see that the Fornell-Larcker criterion is met, as the latent squared correlations are lower than the AVEs of X and Y.

Some papers use a modified version of the Fornell-Larcker criterion, where only the AVE of the X construct, not the AVE of the Y construct, is compared against the latent squared correlation. We can use this version by supplying the x.only = TRUE argument.

forn_larcker_test(fit, x = c("visual"), y = c("textual", "speed"), x.only = TRUE)

# A tibble: 2 × 6
x y criterion latcor_sq ave_x ave_y
* <chr> <chr> <lgl> <dbl> <dbl> <dbl>
1 visual textual TRUE 0.210 0.371 0.721
2 visual speed TRUE 0.221 0.371 0.424

References

Fornell, C., & Larcker, D. F. (1981). Evaluating Structural Equation Models with Unobservable Variables and Measurement Error. Journal of Marketing Research, 18(1), 39. Rönkkö, M., & Cho, E. (2022). An Updated Guideline for Assessing Discriminant Validity. Organizational Research Methods, 25(1), 6–14.

Beautiful Tables for Exploratory Factor Analysis in R

Mon, 22 Feb 2021 00:00:00 +0000

The psych package for R provides great utilities for exploratory factor analysis (EFA). However, the way psych displays the results does not take advantage of visual cues to make grasping the factor solutions easier, nor is it straightforward to display the results in a way that can be shared easily with others. Several solutions to the problem have been proposed, such as the LaTeX-based or an implementation in the sjPlot package sjPlot::fa_tab(). Because these solutions have some drawbacks, here’s another take on the issue.

The solution I present here is consistent with the tidyverse and builds on the highly flexible and powerful gt package to provide beautiful and highly customizable tables that can be used in R Markdown approaches.

Factor Analysis Example

We need several packages to get started:

library(psych)
library(tidyverse)
library(gt)

As an example we use the IPIP Big Five Inventory from the psych package. First we run a factor analysis; I will not go into details here as these are discussed in-depth in the psych package and EFA best practice papers.

res <- fa(psychTools::bfi[1:25],5)

The standard way to display the EFA results are not very clean or legible inside a markdown output.

print(res)

Factor Analysis using method = minres
Call: fa(r = psychTools::bfi[1:25], nfactors = 5)
Standardized loadings (pattern matrix) based upon correlation matrix
MR2 MR1 MR3 MR5 MR4 h2 u2 com
A1 0.21 0.17 0.07 -0.41 -0.06 0.19 0.81 2.0
A2 -0.02 0.00 0.08 0.64 0.03 0.45 0.55 1.0
A3 -0.03 0.12 0.02 0.66 0.03 0.52 0.48 1.1
A4 -0.06 0.06 0.19 0.43 -0.15 0.28 0.72 1.7
A5 -0.11 0.23 0.01 0.53 0.04 0.46 0.54 1.5
C1 0.07 -0.03 0.55 -0.02 0.15 0.33 0.67 1.2
C2 0.15 -0.09 0.67 0.08 0.04 0.45 0.55 1.2
C3 0.03 -0.06 0.57 0.09 -0.07 0.32 0.68 1.1
C4 0.17 0.00 -0.61 0.04 -0.05 0.45 0.55 1.2
C5 0.19 -0.14 -0.55 0.02 0.09 0.43 0.57 1.4
E1 -0.06 -0.56 0.11 -0.08 -0.10 0.35 0.65 1.2
E2 0.10 -0.68 -0.02 -0.05 -0.06 0.54 0.46 1.1
E3 0.08 0.42 0.00 0.25 0.28 0.44 0.56 2.6
E4 0.01 0.59 0.02 0.29 -0.08 0.53 0.47 1.5
E5 0.15 0.42 0.27 0.05 0.21 0.40 0.60 2.6
N1 0.81 0.10 0.00 -0.11 -0.05 0.65 0.35 1.1
N2 0.78 0.04 0.01 -0.09 0.01 0.60 0.40 1.0
N3 0.71 -0.10 -0.04 0.08 0.02 0.55 0.45 1.1
N4 0.47 -0.39 -0.14 0.09 0.08 0.49 0.51 2.3
N5 0.49 -0.20 0.00 0.21 -0.15 0.35 0.65 2.0
O1 0.02 0.10 0.07 0.02 0.51 0.31 0.69 1.1
O2 0.19 0.06 -0.08 0.16 -0.46 0.26 0.74 1.7
O3 0.03 0.15 0.02 0.08 0.61 0.46 0.54 1.2
O4 0.13 -0.32 -0.02 0.17 0.37 0.25 0.75 2.7
O5 0.13 0.10 -0.03 0.04 -0.54 0.30 0.70 1.2
MR2 MR1 MR3 MR5 MR4
SS loadings 2.57 2.20 2.03 1.99 1.59
Proportion Var 0.10 0.09 0.08 0.08 0.06
Cumulative Var 0.10 0.19 0.27 0.35 0.41
Proportion Explained 0.25 0.21 0.20 0.19 0.15
Cumulative Proportion 0.25 0.46 0.66 0.85 1.00
With factor correlations of
MR2 MR1 MR3 MR5 MR4
MR2 1.00 -0.21 -0.19 -0.04 -0.01
MR1 -0.21 1.00 0.23 0.33 0.17
MR3 -0.19 0.23 1.00 0.20 0.19
MR5 -0.04 0.33 0.20 1.00 0.19
MR4 -0.01 0.17 0.19 0.19 1.00
Mean item complexity = 1.5
Test of the hypothesis that 5 factors are sufficient.
df null model = 300 with the objective function = 7.23 with Chi Square = 20163.79
df of the model are 185 and the objective function was 0.65
The root mean square of the residuals (RMSR) is 0.03
The df corrected root mean square of the residuals is 0.04
The harmonic n.obs is 2762 with the empirical chi square 696.08 with prob < 2.7e-60
The total n.obs was 2800 with Likelihood Chi Square = 1808.94 with prob < 4.3e-264
Tucker Lewis Index of factoring reliability = 0.867
RMSEA index = 0.056 and the 90 % confidence intervals are 0.054 0.058
BIC = 340.53
Fit based upon off diagonal values = 0.98
Measures of factor score adequacy
MR2 MR1 MR3 MR5 MR4
Correlation of (regression) scores with factors 0.91 0.82 0.84 0.82 0.81
Multiple R square of scores with factors 0.83 0.67 0.70 0.67 0.66
Minimum correlation of possible factor scores 0.65 0.34 0.41 0.34 0.31

Using gt() to build a beautiful EFA results table

Below is the fa_table function that takes the factor analysis object from psych::fa() as its input and returns a clean table. Some code elements were adapted from sjPlot::fa_tab() and .

fa_table <- function(x, varlabels = NULL, title = "Factor analysis results", diffuse = .10, small = .30, cross = .20, sort = TRUE) {
 #get sorted loadings
 require(dplyr)
 require(purrr)
 require(tibble)
 require(gt)
 if(sort == TRUE) {
 x <- psych::fa.sort(x)
 }
 if(!is.null(varlabels)) {
 if(length(varlabels) != nrow(x$loadings)) { warning("Number of variable labels and number of variables are unequal. Check your input!",
 call. = FALSE) }
 if(sort == TRUE) {
 varlabels <- varlabels[x$order]
 }
 }
 if(is.null(varlabels)) {varlabels <- rownames(x$loadings)}

 loadings <- data.frame(unclass(x$loadings))

 #make nice names
 factornamer <- function(nfactors) {
 paste0("Factor_", 1:nfactors)}

 nfactors <- ncol(loadings)
 fnames <- factornamer(nfactors)
 names(loadings) <- fnames

 # prepare locations
 factorindex <- apply(loadings, 1, function(x) which.max(abs(x)))

 # adapted from from sjplot: getremovableitems
 getRemovableItems <- function(dataframe, fctr.load.tlrn = diffuse) {
 # clear vector
 removers <- vector(length = nrow(dataframe))
 # iterate each row of the data frame. each row represents
 # one item with its factor loadings
 for (i in seq_along(removers)) {
 # get factor loadings for each item
 rowval <- as.numeric(abs(dataframe[i, ]))
 # retrieve highest loading
 maxload <- max(rowval)
 # retrieve 2. highest loading
 max2load <- sort(rowval, TRUE)[2]
 # check difference between both
 if (abs(maxload - max2load) < fctr.load.tlrn) {
 # if difference is below the tolerance,
 # remeber row-ID so we can remove that items
 # for further PCA with updated data frame
 removers[i] <- TRUE
 }
 }
 # return a vector with index numbers indicating which items
 # have unclear loadings
 return(removers)
 }
 if(nfactors > 1) {
 removable <- getRemovableItems(loadings)
 cross_loadings <- purrr::map2(fnames, seq_along(fnames), function(f, i) {
 (abs(loadings[,f] > cross)) & (factorindex != i)
 })
 }

 small_loadings <- purrr::map(fnames, function(f) {
 abs(loadings[,f]) < small
 })

 ind_table <- dplyr::tibble(varlabels, loadings) %>%
 dplyr::rename(Indicator = varlabels) %>%
 dplyr::mutate(Communality = x$communality, Uniqueness = x$uniquenesses, Complexity = x$complexity) %>%
 dplyr::mutate(across(starts_with("Factor"), round, 3)) %>%
 dplyr::mutate(across(c(Communality, Uniqueness, Complexity), round, 2))


 ind_table <- ind_table %>% gt(rowname_col = "Indicator") %>% tab_header(title = title)
 # mark small loadiongs
 for(f in seq_along(fnames)) {
 ind_table <- ind_table %>% tab_style(style = cell_text(color = "#D3D3D3", style = "italic"),
 locations = cells_body(columns = fnames[f], rows = small_loadings[[f]]))
 }
 # mark cross loadings

 if (nfactors > 1) {
 for (f in seq_along(fnames)) {
 ind_table <-
 ind_table %>% tab_style(
 style = cell_text(style = "italic"),
 locations = cells_body(columns = fnames[f], rows = cross_loadings[[f]])
 )
 }
 # mark non-assignable indicators
 ind_table <-
 ind_table %>% tab_style(style = cell_fill(color = "#D93B3B"),
 locations = cells_body(rows = removable))
 }

 # adapted from https://www.anthonyschmidt.co/post/2020-09-27-efa-tables-in-r/
 Vaccounted <- x[["Vaccounted"]]
 colnames(Vaccounted) <- fnames
 if (nfactors > 1) {
 Phi <- x[["Phi"]]
 rownames(Phi) <- fnames
 colnames(Phi) <- fnames
 f_table <- rbind(Vaccounted, Phi) %>%
 as.data.frame() %>%
 rownames_to_column("Property") %>%
 mutate(across(where(is.numeric), round, 3)) %>%
 gt() %>% tab_header(title = "Eigenvalues, Variance Explained, and Factor Correlations for Rotated Factor Solution")
 }
 else if(nfactors == 1) {
 f_table <- rbind(Vaccounted) %>%
 as.data.frame() %>%
 rownames_to_column("Property") %>%
 mutate(across(where(is.numeric), round, 3)) %>%
 gt() %>% tab_header(title = "Eigenvalues, Variance Explained, and Factor Correlations for Rotated Factor Solution")
 }

 return(list("ind_table" = ind_table, "f_table" = f_table))

}

We run the function on the results:

tables <- fa_table(res)

Warning: There was 1 warning in `dplyr::mutate()`.
ℹ In argument: `across(starts_with("Factor"), round, 3)`.
Caused by warning:
! The `...` argument of `across()` is deprecated as of dplyr 1.1.0.
Supply arguments directly to `.fns` through an anonymous function instead.
# Previously
across(a:b, mean, na.rm = TRUE)
# Now
across(a:b, \(x) mean(x, na.rm = TRUE))

tables$ind_table

Factor analysis results
	Factor_1	Factor_2	Factor_3	Factor_4	Factor_5	Communality	Uniqueness	Complexity
N1	0.815	0.103	0.004	-0.111	-0.047	0.65	0.35	1.08
N2	0.777	0.040	0.011	-0.094	0.015	0.60	0.40	1.04
N3	0.706	-0.100	-0.035	0.079	0.023	0.55	0.45	1.07
N5	0.486	-0.202	-0.004	0.207	-0.150	0.35	0.65	1.96
N4	0.474	-0.386	-0.135	0.095	0.080	0.49	0.51	2.27
E2	0.099	-0.676	-0.016	-0.048	-0.058	0.54	0.46	1.07
E4	0.014	0.591	0.024	0.287	-0.077	0.53	0.47	1.49
E1	-0.059	-0.557	0.106	-0.083	-0.103	0.35	0.65	1.21
E5	0.152	0.421	0.271	0.052	0.206	0.40	0.60	2.60
E3	0.083	0.418	-0.001	0.246	0.283	0.44	0.56	2.55
C2	0.149	-0.085	0.666	0.081	0.039	0.45	0.55	1.17
C4	0.174	0.002	-0.614	0.040	-0.048	0.45	0.55	1.18
C3	0.034	-0.061	0.567	0.092	-0.068	0.32	0.68	1.11
C5	0.189	-0.142	-0.553	0.018	0.092	0.43	0.57	1.44
C1	0.069	-0.027	0.546	-0.023	0.148	0.33	0.67	1.19
A3	-0.029	0.116	0.025	0.660	0.031	0.52	0.48	1.07
A2	-0.023	-0.002	0.077	0.640	0.032	0.45	0.55	1.04
A5	-0.112	0.233	0.006	0.532	0.044	0.46	0.54	1.49
A4	-0.057	0.064	0.193	0.433	-0.148	0.28	0.72	1.74
A1	0.213	0.166	0.067	-0.414	-0.058	0.19	0.81	1.97
O3	0.031	0.152	0.017	0.083	0.609	0.46	0.54	1.17
O5	0.132	0.098	-0.025	0.043	-0.542	0.30	0.70	1.21
O1	0.018	0.103	0.073	0.015	0.508	0.31	0.69	1.13
O2	0.195	0.057	-0.078	0.163	-0.456	0.26	0.74	1.75
O4	0.126	-0.323	-0.024	0.174	0.371	0.25	0.75	2.69

tables$f_table

Eigenvalues, Variance Explained, and Factor Correlations for Rotated Factor Solution
Property	Factor_1	Factor_2	Factor_3	Factor_4	Factor_5
SS loadings	2.570	2.199	2.029	1.985	1.586
Proportion Var	0.103	0.088	0.081	0.079	0.063
Cumulative Var	0.103	0.191	0.272	0.351	0.415
Proportion Explained	0.248	0.212	0.196	0.191	0.153
Cumulative Proportion	0.248	0.460	0.656	0.847	1.000
Factor_1	1.000	-0.213	-0.187	-0.038	-0.011
Factor_2	-0.213	1.000	0.230	0.329	0.167
Factor_3	-0.187	0.230	1.000	0.203	0.195
Factor_4	-0.038	0.329	0.203	1.000	0.193
Factor_5	-0.011	0.167	0.195	0.193	1.000

The function returns two tables: ind_table for the factor (pattern) loadings, and f_table for aspects at the factor-level. The former table displays the relevant information for each indicator; the use of italics and colors makes it easy to grasp the structure of factor (pattern) loadings. Highlighted in red, we see that N4 and 04 indicators are not represented well in the EFA because they load diffusely over multiple factors.

You can supply custom values for the conditional formatting of the table:

diffuse specifies the minimum difference an indicator needs to have between factor loadings in order to indicate a clear loading on just one factor, and not diffuse loadings over multiple factors. Diffuse indicators are labelled as red in the table.
small specifies when a loading is considered small; these loadings are printed in light gray.
cross specifies when a loading is considered to be a cross-loading; these loadings are printed in oblique black.

The fa_table() function is a work in progress and the latest version can be found at:

Import it to R via: source("https://raw.githubusercontent.com/franciscowilhelm/r-collection/master/fa_table.R")

How to compute multi-level reliability indices in R and Mplus

Sat, 11 May 2019 00:00:00 +0000

Update 2022

Since the time since I wrote this blog post, a package has been developed that calculates the multi-level reliability coefficient omega within R. See the function omegaSEM from .

Main post

Reliability estimation is one of the core tasks when working with psychological scales. However, reliability estimation with a multi-level data structure has only recently become a topic and its hard to find good materials for this. In this post, we are going to use several R packages and MPlus to compute the reliability of scales in a multi-level framework.

We use a multilevel confirmatory factor analysis (MCFA) to estimate the reliability of a psychological scale in a two-level framework. We are going to refer to level-1 as the within-level, and to level-2 as the between-level. The methods are described in Geldhof, Preacher, and Zyphur (2014) and Shrout and Lane (2012).

There are several reliability estimates available. The most common is Cronbach’s Alpha. It can be extended to a multi-level framework. The estimate can be computed for both the within-level and the between-level. Omega is another index that is generally considered better than Alpha, but is less common. When the multilevel data comes from an intensive longitudinal design, we also want to control for the (linear) time trend. This will be the case for our example.

Tools

We use Mplus to do the statistical heavy-lifting here; possibly, the R package lavaan could also do this, but at the time of writing, lavaan still misses some multilevel functionality (such as handling missing level-1 data) that make Mplus the superior tool ¹. We will utilize the R package MPlusAutomation to generate the syntax code, write the “.inp” files for Mplus and analyze the results.

library(MplusAutomation)

Version: 1.1.1
We work hard to write this free software. Please help us get credit by citing:
Hallquist, M. N. & Wiley, J. F. (2018). MplusAutomation: An R Package for Facilitating Large-Scale Latent Variable Analyses in Mplus. Structural Equation Modeling, 25, 621-638. doi: 10.1080/10705511.2017.1402334.
-- see citation("MplusAutomation").

library(tidyverse)

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.2.0 ✔ readr 2.1.6
✔ forcats 1.0.1 ✔ stringr 1.6.0
✔ ggplot2 4.0.2 ✔ tibble 3.3.1
✔ lubridate 1.9.4 ✔ tidyr 1.3.2
✔ purrr 1.2.1
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ tidyr::extract() masks MplusAutomation::extract()
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

Example Dataset

We use an example dataset provided by Bolger and Laurenceau (2013). You can get it from their website , download and extract the ch7Mplus.zip file, the data is in the psychometrics.dat file. Because the dataset comes from an Mplus setting, we first have to modify it a little bit.

psychometrics <- as_tibble(readr::read_tsv("multilevel/psychometrics.dat"))
names(psychometrics) <- c("id", "time", "item1", "item2", "item3", "item4")
psychometrics <- psychometrics %>% mutate_all(~na_if(., -999))

head(psychometrics)

# A tibble: 6 × 6
id time item1 item2 item3 item4
<dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 301 2 2 3 3 2
2 301 3 3 3 2 2
3 301 4 4 3 3 3
4 301 5 2 2 2 2
5 301 6 2 2 1 2
6 301 7 2 2 1 2

The data is long format, with multiple rows per person (id identifies the person) - one row for each time point, and four items belonging to the same scale measured at each time point.

Next we are going to source some code I created to help us prepare the Mplus Syntax for estimating the multilevel reliabilities.

source("https://raw.githubusercontent.com/franciscowilhelm/r-collection/master/modelstring.R")

Now we use MplusAutomation to create the Mplus object syntax. We are going to estimate Cronbach’s $\alpha$ (alpha), $\omega$ (omega), and Maximal Reliability ( $H$ )², each at the between- and the within-person level.

# define the names of the item variables we use
var_names <- c("item1", "item2", "item3", "item4")

# generate the MplusObject(s)
m_rel_omega <- mplusObject(
 TITLE = "MCFA RELIABILITY EXAMPLE",
 VARIABLE = "CLUSTER = id; \n WITHIN = time;",
 ANALYSIS = "TYPE = TWOLEVEL",
 MODEL = modelstring_omega_core(var_names, "time"),
 MODELCONSTRAINT = modelstring_omega_constraint(var_names),
 OUTPUT = "SAMPSTAT CINTERVAL;",
 usevariables = c(var_names, "id", "time"),
 rdata = psychometrics,
 autov = FALSE
)

m_rel_alpha <- mplusObject(
 TITLE = "MCFA RELIABILITY EXAMPLE",
 VARIABLE = "CLUSTER = id; \n WITHIN = time;",
 ANALYSIS = "TYPE = TWOLEVEL",
 MODEL = modelstring_alpha_core(var_names, "time"),
 MODELCONSTRAINT = modelstring_alpha_constraint(var_names),
 OUTPUT = "SAMPSTAT CINTERVAL;",
 usevariables = c(var_names, "id", "time"),
 rdata = psychometrics,
 autov = FALSE
)

Note that the “MODEL” and the MODELCONSTRAINT" sections of the syntax are generated using the functions we just sourced. The function takes the names of the item variables, and because we are using a longitudinal dataset, the name of the time variable. If you have data where level 1 is not longitudinal (e.g., persons in teams), you can just drop this argument.

Next, we write, run, and read the model.

m_rel_omega <- mplusModeler(m_rel_omega, modelout = "./multilevel/rel_omega.inp", run = FALSE)
m_rel_alpha <- mplusModeler(m_rel_alpha, modelout = "./multilevel/rel_alpha.inp", run = FALSE)

runModels(target = "multilevel")

m_rel_fit <- readModels(target = "multilevel")

Finally, we have to extract the reliability point estimates and then we are done!

knitr::kable(paramExtract(m_rel_fit[[1]]$parameters$unstandardized, "new"))

	paramHeader	param	est	se	est_se	BetweenWithin
29	New.Additional.Parameters	COMP_V_W	5.287	0.475	11.134	Between
30	New.Additional.Parameters	ALPHA_W	0.776	0.024	32.854	Between
31	New.Additional.Parameters	COMP_V_B	6.416	1.223	5.246	Between
32	New.Additional.Parameters	ALPHA_B	0.884	0.038	23.396	Between

knitr::kable(paramExtract(m_rel_fit[[2]]$parameters$unstandardized, "new"))

	paramHeader	param	est	se	est_se	BetweenWithin
27	New.Additional.Parameters	NUM_W	4.116	0.463	8.883	Between
28	New.Additional.Parameters	DENOM_W	5.289	0.475	11.127	Between
29	New.Additional.Parameters	OMEGA_W	0.778	0.023	34.119	Between
30	New.Additional.Parameters	H_W	0.785	0.021	37.388	Between
31	New.Additional.Parameters	NUM_B	5.689	1.264	4.501	Between
32	New.Additional.Parameters	DENOM_B	6.411	1.223	5.243	Between
33	New.Additional.Parameters	OMEGA_B	0.887	0.036	24.482	Between
34	New.Additional.Parameters	H_B	0.931	0.043	21.576	Between

In the code above we use the MplusAutomation::paramExtract function to get all the parameters created through Mplus model constraints by using the "new" argument. Note that the Alpha model gives us the Alpha at between-level (ALPHA_B) and at the within-level (ALPHA_W), whereas the Omega model gives us Omega and H at the between-level (OMEGA_B & H_B) and at the within-level (OMEGA_W & H_W). Please do not let your self get confused by the “BetweenWithin” column - manually created parameters in Mplus are always given as Between, even when they are not. The other parameters (NUM & DENOM) are used for the calculations and can be be ignored.

As we can see, the reliabilities for our four item scale are quite decent, even at the within-level with values above .70. This means that not only does the scale capture between-person differences reliably, but also within-person changes from the person’s mean level.

Sources

Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72–91.

Shrout, P. E., & Lane, S. P. (2012). Psychometrics. In M. R. Mehl & T. S. Conner (Eds.), *Handbook of research methods for studying daily life *(pp. 302-320). New York, NY, US: The Guilford Press.

If you do not have access to Mplus, you may want to use an ANOVA framework instead of the MCFA framework we use here. For more, see the excellent book by Bolger and Laurenceau (2013) and its which has the R code for it. Thanks to for pointing out this book to me. ↩︎
For more information see Geldhof et al. (2014). ↩︎