Psychometric review
A working guide for reviewing questionnaire and PROM validation work (Rasch, IRT, CFA). Lead with theory: name the problem, cite a methods paper, then recommend the fix (a method and, where relevant, software). Be direct and constructive.
Three modes
Default to self-review when the user supplies their own work. If it is not clear whose work it is, or whether it is published, ask.
- Self-review (default). The user's own analysis and manuscript, before
submission. The goal is to find and fix what a journal reviewer would
object to. Output: an internal review memo (
templates/self-review.qmd) with issues ranked by severity, each with a concrete fix. - Appraisal of a published paper. A paper already in print, read by
someone deciding whether to use the instrument, citing it, including it in
a systematic review, or presenting it at a journal club. The goal is to
judge which of the paper's measurement claims the reported evidence
supports, and what that means for anyone using the scores. Output: an
appraisal memo (
templates/appraisal.qmd) with each issue ranked as undermines a main claim (the claim does not stand on the reported evidence), weakens a claim (the claim may hold, but the evidence is thinner than presented) or minor, and with the consequence for users in place of a fix. - Referee report. Someone else's manuscript, reviewed for a journal.
Output: a reviewer report (
templates/referee-report.qmd) addressed to the authors and editor. Manuscripts under review are confidential. Do not send them to services the user has not approved. Recommend methods and software by what they do, not by who wrote them, and avoid wording that would identify the reviewer. Do not make citing the reviewer's own papers a condition. Where the user's own work is the best source, cite it alongside other sources if possible and leave the choice to the user.
The evaluation framework below is the same in all three modes. What differs is the inputs, the extra checks in self-review, and the output.
Workflow
- Settle the mode, and the model type: Rasch or IRT (dichotomous or polytomous), CFA, many-facet Rasch, or a combination. Only the sections that apply to that model are relevant.
- Gather the inputs listed below and read everything before writing.
- In self-review, start with the checks only the author can run (see "Extra checks in self-review"): numbers against output, and code against text.
- Work through the two cross-cutting concerns, the four criteria, the additional checks that apply, and the model-specific section (CFA or MFRM).
- Rank the issues and fill in the template for the mode.
Raise only issues that are present in this manuscript, and point to the evidence (section, table, quoted passage, line of code). Distinguish "not reported" from "reported and failed". Delete template sections that do not apply, and do not pad the report with general advice that the manuscript does not need.
First, gather the inputs
Self-review.
- The manuscript draft, including tables, figures and supplementary material.
- The analysis code and its output (R scripts, Quarto/R Markdown files, software output). Many problems are only visible there: the estimator actually used, the cutoffs actually applied, the items actually dropped.
- The target journal and any reporting guideline it requires.
Appraisal of a published paper.
- The published article and its supplementary material, plus any shared data and code. Most published papers have neither, so judge what is reported and say which claims cannot be checked from the paper alone. A missing test is a gap in the evidence, not evidence that the criterion holds.
- What the reader wants the paper for (choosing an instrument, a particular population or language version, a total versus subscale score). The same weakness can matter for one use and not another. If the intended use is not stated, ask for it, or appraise the uses the paper itself recommends.
- Hold the paper to the methods of its time, but say when later methods work shows a choice to be unreliable (for example a fixed fit cutoff), since that affects how far the result can be trusted today. Keep the two judgements apart, so the appraisal reads as a judgement of the evidence and not of the authors.
- For each open issue, say which analysis would settle it (for example a simulation-based fit test, a local dependence test, DIF for a group the intended use needs, a strict invariance test) and whether a reanalysis of shared data would do or new data are needed.
Referee report.
- The manuscript PDF (main text, tables, figures and any supplementary material).
- For a revision: the authors' point-by-point response and the previous review. Number the comments (e.g. R2.1, R2.2) and, on later rounds, track which are resolved, which are answered with material that addresses a different point, and which are declined. "It is conventional" or "sufficient for our aims" is not a rebuttal. Say so and restate what was asked for.
- If the paper has no page or line numbers, quote the passage being commented on so the authors can find it.
Read everything before writing. Map each measurement claim to the criterion it belongs to, then check each criterion in turn.
The evaluation framework
Structure the methods and results critique around the four criteria that underpin interval-level measurement [@christensenPsychometricValidationPROM2021; @kreinerValidityObjectivityReflections2007]. Two concerns usually cut across all four. Raise them once, up front.
Cross-cutting concern 1: sample size and power
Calibration stability is not the binding constraint. Power to detect the departures being tested for is. Small samples (and small DIF subgroups) mean low sensitivity to misfit, local dependence and DIF, so "no problem found" results are preliminary. Do not accept minimum-N rules from non-peer-reviewed sources (Rasch Measurement Transactions, rasch.org, Linacre notes). Cite [@johanssonSimulationbasedCutoffsConditional2026] on power.
Cross-cutting concern 2: rule-of-thumb versus simulation-based cutoffs
The recurring problem in this literature. Fixed critical values (item fit MSQ 0.5 to 1.5 or 0.7 to 1.3, Q3 > 0.2, first-contrast eigenvalue > 2.0, Hu and Bentler CFA cutoffs, ΔCFI 0.01 for invariance) depend on sample size and item properties and are wrong in either direction [@mullerItemFitStatistics2020; @marshSearchGoldenRules2004; @mcneishHowPsychologistsDetermine2026]. Recommend data-adaptive, simulation-based (parametric bootstrap) cutoffs [@johanssonDetectingItemMisfit2025; @johanssonSimulationbasedCutoffsConditional2026]. For iterations, decision rules and the familywise error correction (Westfall-Young, at least 400 iterations), cite [@johanssonSimulationbasedCutoffsConditional2026]. For the choice between familywise and false discovery rate control over a moderate number of tests, such as one per item, cite [@ferreiraMethodsTestingSmall2024].
For CFA fit, Groskurth et al. [-@groskurthWhyWeNeed2024, p. 3907] give the most concrete demonstration: in a one-factor model with six items and N = 200, the value that rejects 5% of correctly specified models is CFI = .813 when loadings are .40 and .979 when they are .80. One fixed value cannot serve both. The editors of the European Journal of Psychological Assessment now ask authors to "abandon all fixed cutoff rules (not only those suggested by Hu & Bentler, 1999)" [@goretzkoStopUsingFixed2026, p. 80], which a reviewer can cite when authors defend a cutoff as accepted practice. The CFA, MGCFA and bifactor tools are covered in their own section below.
Question every fixed cutoff, whatever its source, on two grounds. What is it derived from, a simulation, a derivation or an informal suggestion that later hardened into a rule? And does it generalize to the present model, estimator, number of categories, sample size and loadings? This applies equally to the fixed values a reviewer is tempted to supply in their place: a residual correlation of ±.10, a modification index above 10, an ECV of .70, a dMACS of .40. If a value has to be used, call it a heuristic and give its source, and do not let a verdict rest on it alone.
The same applies to asymptotic p-values used as if they were exact. The
item-restscore test compares the observed Goodman-Kruskal gamma between an
item and its restscore with the gamma expected under the Rasch model
[@kreinerNoteItemRestscore2011]. Kreiner [-@kreinerNoteItemRestscore2011,
p. 559] notes that the item × restscore table becomes large and sparse as the
number of items and categories grows, so the asymptotic distribution is a poor
approximation, and he suggests Monte Carlo or parametric bootstrap p-values. A
null simulation of the iarm implementation shows the size of the problem
[@johanssonCalibrationAsymptoticItemRestscore2026]. Because the expected gamma
is estimated from the same data, the z statistic is too small. The underfit
tail rejected 0.2 to 1.2% against a nominal 2.5%. A small-sample bias works
in the other direction: with Benjamini-Hochberg correction over 20
dichotomous items, a true Rasch model had at least one flagged item in 13.5%
of samples at n = 150, and 30.5% when the scale was mistargeted. Polytomous
items were conservative at every n. No single correction to the nominal level
fixes both. Ask for parametric bootstrap p-values that re-estimate the item
parameters in every replication (RMitemRestscoreCutoff() in easyRasch2).
1. Unidimensionality
- The PCA-of-residuals first-contrast eigenvalue < 2.0 is a rule of thumb that fails across settings [@chouCheckingDimensionalityItem2010]. The expected eigenvalue varies with N. Cross-study eigenvalue comparisons need expected ranges per study to be interpretable.
- RUMM PCAR/t-test: require a reliable binomial CI (Hagell 2014, doi:10.4236/ojs.2014.46044) and interpret the t-test cautiously.
- Recommend a direct test: CFA with WLSMV/DWLS, treating items as ordinal, with fit compared against simulation-based cutoffs (see "CFA, MGCFA and bifactor model fit" below), or the Martin-Löf test [@christensenTestingUnidimensionalityPolytomous2002] with Monte Carlo p-values [@christensenMonteCarloApproach2007].
- A one-factor CFA is only meaningful as a test of unidimensionality when its fit is judged against simulation-based cutoffs for that model, estimator and sample. Against fixed cutoffs such as Hu and Bentler it is no test at all, since the same index value can mean good or poor fit depending on loadings, N and the number of categories (see "CFA, MGCFA and bifactor model fit"). Even then, it tests unidimensionality under a model with free loadings, not under the Rasch model. Read it alongside item fit, which tests equal discrimination, and the local dependence checks, since residual dependence also lowers CFA fit. Constraining the loadings to be equal brings the CFA closer to the Rasch model if that comparison is wanted.
- For a scale with subscales, ask for each subscale to be fitted as its own unidimensional model before any correlated-factors, higher-order or bifactor model, so readers can see whether each holds up alone.
- Visual aid: plot standardized loadings on the first residual contrast against item location.
2. Local independence
Frequently omitted, so insist on it. Response dependency is likely whenever items are near-duplicates (e.g. pairs worded up and down). If authors pre-empt expected local dependence (e.g. by collapsing pairs), ask for an initial analysis with the items unaltered first.
- Detect with Q3 residual correlations [@yenEffectsLocalItem1984] and per-pair simulation-based cutoffs [@christensenCriticalValuesYens2017; @nason_another_2025], and with partial gamma, which assesses both directions and so flags response dependency.
- Resolve (besides deletion) with testlets/superitems (re-check threshold ordering after merging) or the graphical loglinear Rasch model [@kreinerAnalysisLocal2004] in DIGRAM.
- Local dependence inflates reliability [@raykovBiasCoefficient2001] and can manufacture a bifactor general factor. Overfit items very often signal local dependence [@tuerlinckxEffectIgnoringItem2001].
3. Invariance (DIF)
- Name the test and justify any magnitude threshold (e.g. 0.5 logit) with a methods paper, not an applied one. Partial gamma conditional on the total score [@bjornerDifferentialItemFunctioning1998; @kreinerAnalysisMultidimensionalContingency1987] and Rasch trees [@stroblRaschTreesNew2015] are methods papers for the common tests.
- Evaluate threshold-level DIF (differential step functioning), not only the average item location. Treat continuous covariates (e.g. age) continuously via Rasch trees [@stroblRaschTreesNew2015] or partial credit trees [@henningerPartialCreditTrees2025].
- Quantify DIF impact (anchored conversion tables, displacement plots), not only detection. For CFA invariance (MGCFA, longitudinal or dyadic) use simulation-based cutoffs, not ΔCFI 0.01 [@cheungEvaluatingGoodnessOfFit2002] or the Chen [-@chenSensitivityGoodnessFit2007] values. See "CFA, MGCFA and bifactor model fit" below for the tools and their limits.
- Distinguish real from artificial DIF. A flagged item can reflect DIF in
another item, so DIF must be resolved stepwise with item purification rather
than read off a single simultaneous analysis
[@andrichRealArtificialDifferential2012;
@andrichRealArtificialDifferential2015;
@hagquistExplainingDifferentialItem2019]. Henninger et al.
[-@henningerPartialCreditTrees2025] give a clear treatment of purification.
It is implemented in
RMdifTree()in easyRasch2 and its jamovi module. - Ask which anchor linked the groups' scales before reading item-level DIF results. The equal-mean-difficulty (centred) and all-other anchors produce artificial DIF whenever DIF is unbalanced (including a single DIF item) [@wangEffectsAnchorItem2004; @kopfAnchorSelectionStrategies2015], and the false alarm rate of the all-other anchor grows with sample size [@stroblAnchorPointSelection2021]. Rasch trees are affected too. Their detection rate falls when DIF is unbalanced and as the share of DIF items grows, most of all in small samples [@asamoahImpactsDIFItem2026]. Data-driven anchors such as the iterative forward method [@kopfFrameworkAnchorMethods2015] or the Gini anchor point [@stroblAnchorPointSelection2021] avoid this when most items are free of DIF, an assumption that cannot be tested. Short anchors cost power, and at least four anchor items are usually recommended. Where the anchor choice is uncertain, ask for a sensitivity analysis with alternative anchor sets [@teresiMethodologicalIssuesExamining2016].
4. Monotonicity (ordered response thresholds)
- Keep two questions apart. Monotonicity in the sense of an expected item score that increases with the latent variable holds in the PCM for any parameter values [@adamsRaschRatingModel2012], so whether the data show it is a question for item fit and observed versus expected item score plots. Threshold order asks whether the response categories work in the intended order. That is the subject of this point.
- Ask for a table of item-level threshold locations with standard errors. What disordered thresholds mean is unresolved in the peer-reviewed literature, so do not present either reading as settled. Andrich [-@andrichExpandedDerivationThreshold2013] argues that every item with disordered threshold estimates shows an anomaly in how its categories work. Adams, Wu and Wilson [-@adamsRaschRatingModel2012], Wetzel and Carstensen [-@wetzelReversedThresholdsPartial2014] and García-Pérez [-@garcia-perezAnalysisDisOrderedCategories2017] argue that disorder reflects how often the categories are used, that the category order still holds, and that collapsing loses information. The two sides partly disagree about what category order means, and neither has refuted the other on its own terms.
- What all of them support, and what the review can ask for:
- Treat disordered thresholds as a reason to examine the item (response labels, scoring key, category frequencies, item fit), neither as automatic failure nor as something to ignore. Read the estimates with their standard errors and covariance, since they can be disordered or ordered by chance [@adamsRaschRatingModel2012].
- Do not collapse categories solely because thresholds are disordered. Andrich [-@andrichExpandedDerivationThreshold2013] asks for a substantive explanation, and the other side argues against collapsing on this ground at all [@adamsRaschRatingModel2012; @wetzelReversedThresholdsPartial2014; @garcia-perezAnalysisDisOrderedCategories2017].
- Collapsing after the fact is not the same as administering a shorter response format, so a collapsed scoring is provisional until the revised format has been tested in new data [@garcia-perezAnalysisDisOrderedCategories2017; @anselmiAssessingThresholdOrdering2026].
- A supporting check is whether the mean person estimate rises across the categories of each item [@wetzelReversedThresholdsPartial2014]. It is one piece of evidence, not proof that the categories work as intended.
- Report reliability for the scoring finally used, and both values if categories were collapsed. Collapsing lowers test information and reliability [@garcia-perezAnalysisDisOrderedCategories2017], so the drop is an expected cost of the change, not a correction of an inflated value.
- Look closely at a neutral middle category on a bipolar agreement scale, such as "Neither agree nor disagree". The concern is the neutral label, not the odd number of categories as such: a middle category whose label marks a step along the trait does not invite the same use. The neutral category is the one most often found never to be the most probable [@ostiniPolytomousItemResponse2006], and it often has disordered or closely spaced thresholds across many instruments [@anselmiAssessingThresholdOrdering2026]. One explanation is that some respondents use it to mean "don't know", "can't decide" or "not applicable" rather than a position between disagreeing and agreeing [@ostiniPolytomousItemResponse2006]. Once collected, these responses cannot be separated from true neutral responses. Ask whether the authors considered this and how they handled it. Merging the middle category into a neighbour changes what the scale means, because a neutral position becomes a weak form of agreement or disagreement. On a unipolar frequency or severity scale, adjacent categories are degrees of the same attribute, so merging them is easier to defend [@anselmiAssessingThresholdOrdering2026]. Recoding middle responses as missing assumes they carry no information about the trait. Any of these changes is provisional until the revised format has been tested in new data.
- In a graded response model (GRM) the question looks different. Its thresholds are ordered by construction, so ordered GRM thresholds say nothing about whether the categories work in the intended order, and disordered GRM threshold estimates point to an estimation problem such as an empty category [@garcia-perezAnalysisDisOrderedCategories2017]. A narrow category still gives disordered crossings of the category probability curves, the same pattern that disordered thresholds show in the PCM. García-Pérez [-@garcia-perezAnalysisDisOrderedCategories2017] argues that no model which assumes ordered categories can contradict that order. Only a model that does not assume it, such as the nominal response model, can test it, and in his analyses that model often flagged disorder that was not there.
- Use the PCM as the default. The RSM makes a stronger claim, that the distances between thresholds are the same for every item, so a choice of the RSM needs evidence that this holds, for example by comparing the item-level PCM thresholds. A likelihood-ratio test can be one input, but at large N it rejects trivial differences. Show that the thresholds are ordered under the PCM, not only under the constrained RSM.
Additional required checks
Numbered so they are not lost among the four criteria. Points 8 and 9 apply only when the paper uses a bifactor model or makes claims about individual change.
-
Reliability with a confidence interval. A bare point estimate is not enough. Report two decimals with a 95% interval, bootstrap preferred. Coefficient alpha assumes uncorrelated errors and omega assumes the fitted residual structure, so undetected local dependence biases both upward [@raykovBiasCoefficient2001; @greenCommentaryCoefficient2009]. Tie this to the local independence check. The PSI excludes minimum and maximum scorers, can be very low or negative, and ignores the conditional (non-constant) reliability of the Rasch model, so treat a single PSI value cautiously. Ask for conditional reliability or the standard error across the score range, since any single coefficient hides where the scale is precise and may not describe precision at a cut point where decisions are made [@mcneishReliabilityRepresentativenessHow2025; @greenTechnicalGuidelinesAssessing1984]. Latent and manifest (observed score) reliability also differ, and the latent version is nearly always higher [@milanziReliabilityMeasuresItem2015], so ask which one is reported. For a bifactor model, report omega-hierarchical and omega-hierarchical-subscale, not one total omega.
-
Methods claims need peer-reviewed methods citations. Check every methodological choice and justification (estimator, cutoff, fit criterion, DIF threshold, testlet strategy, minimum sample size, anchor method) against two questions.
- Is the source a methods paper? A choice cannot be justified by citing applied papers that used the method. Applications may themselves be wrong, and citing them this way spreads the error, since readers rarely check the chain. "Widely used" or "accepted practice" with a list of applied papers is a description of practice, not a justification.
- Is it peer-reviewed, or documented well enough to scrutinize? Peer review is the default standard, but what matters is whether the claim can be checked. A preprint or web report of a simulation study with a methods section, results and public code can support a recommendation, particularly when it adds evidence available nowhere else. Label it as a preprint or report. A statement with no methods section, data or code cannot: Rasch Measurement Transactions and rasch.org notes, software manuals and help pages (Winsteps, Facets, RUMM), blog posts. Many rule-of-thumb cutoffs trace back to such sources, for example the 0.5 to 1.5 MSQ range and minimum-N rules. Ask whether a peer-reviewed methods paper or a documented simulation supports the choice, and if none does, ask for the choice to be presented as a heuristic or replaced by a simulation-based one.
- Can the reader retrieve it? A methods source is only useful if readers can find and check it. Back issues of some older measurement journals, such as the Journal of Applied Measurement, are now hard to obtain. Where a more accessible peer-reviewed paper makes the same point, prefer it.
- Be more lenient with books. A textbook or edited volume is acceptable for background and definitions, but a specific methodological choice is better traced to the peer-reviewed paper behind it, and the review can ask for that paper to be cited alongside or instead.
- Apply the same standard to the review itself: cite peer-reviewed methods papers where they exist, label documented preprints and reports as such, and do not cite books or applied papers as the basis for a method.
The standard is higher still for a tutorial paper.
-
Targeting. Person-item targeting via the Wright map (not "Wright's map").
-
Bifactor claims. Judge them by the strength indices (ECV, PUC, omega-hierarchical, omega-hierarchical subscale), each with a bootstrap CI, not by fit, since bifactor models fit almost any structure [@bonifayComplexityItemResponse2017; @reiseScoringModeling2013; @sellbomFactorAnalysis2019]. These are effect sizes describing the fitted model, so the right form is a point estimate read with its interval, not a simulation-based cutoff. Values such as ECV around .70 for "essentially unidimensional" are informal benchmarks from the bifactor literature and carry the same generalizability problem as fit cutoffs. Use them as one reference point, say so, and let the argument rest on the full set of indices and intervals.
- Ask what the model is for. Neither a bifactor nor a higher-order model licenses a sum score across all items on its own. The manuscript should state whether it recommends a total score, subscale scores or both, and the strength indices should support that choice.
- Check that the conclusion matches the paper's own evidence. A paper whose preferred model is bifactor, or which cites a meta-analysis finding a dominant general factor, cannot conclude that the subscales are separable without strength indices showing it.
- Fit cannot decide between bifactor, higher-order and correlated-factors models. A bifactor model is more heavily parameterized, so its "excellent fit" against fixed cutoffs is the result most likely to be an artifact. Neither DFI version has a bifactor-specific procedure (see the CFA section below).
- Unmodeled residual correlations are absorbed by the general factor, so a general factor can partly be item redundancy. Tie this to the local independence check.
-
Individual change. If the paper claims the scale detects change in individual respondents (a reliable change index, a minimal detectable change, responder rates), check how the critical value was obtained. The reliable change index refers a standardized difference to 1.96 [@jacobsonClinicalSignificanceStatistical1991]. For the Rasch version, the difference between two person estimates divided by their pooled standard error, that reference is wrong [@johanssonExactNullRasch2026]:
- The standard errors are functions of the same scores as the difference, and the scores are discrete, so the null is not standard normal. The correct critical value is lower, about 1.7 for six items and still about 1.93 at 60 items. It also depends on targeting, so it has to be computed for the sample at hand, not reused from another study.
- For complete data the null can be enumerated exactly from the item
parameters, without simulation (
RMpersonChange()in easyRasch2). - The index does not order respondents by how much they changed, because the denominator moves with the scores. Read it as a distance from the null, not as a measure of change, and prefer a critical change in logits that depends on the starting location.
- Carry-over between occasions (response dependence) makes the test conservative and creates a spurious severity gradient in estimated change. Ask for a Q3 screen of the racked data, with each item at each occasion as its own column, before change verdicts are relied on.
CFA, MGCFA and bifactor model fit
The main recommendation is the same as for Rasch item fit: interpret fit indices against cutoffs simulated for the model, estimator, sample size and loadings at hand, not against Hu and Bentler [-@huCutoffCriteria1999] or any other fixed values. For most authors this is a large change, so make the case for it first and only then turn to which simulation method fits.
Why fixed fit cutoffs fail
- Hu and Bentler studied one three-factor, 15-item model with continuous normal data, ML estimation and complete data, and warned against generalizing their values [@mcneishHowPsychologistsDetermine2026; @marshSearchGoldenRules2004].
- Fit indices respond to sample size, loadings, number of items, estimator and missing data, often more strongly than to the misspecification itself [@groskurthWhyWeNeed2024; @goretzkoEvaluatingModelFit2024].
- The reliability paradox: weak loadings make misfit easier to pass. Under the Hu and Bentler logic, the RMSEA cutoff at N = 500 ranges from about .03 (seven items, loadings .60) to .16 (three items, loadings .90) [@mcneishGeneralizabilityDynamicFit2023, p. 198]. Fixed cutoffs therefore reward short scales and weak items.
- With ordinal data and DWLS/ULS estimators, fit indices are attenuated toward perfect fit, so fixed cutoffs are more lenient still [@mcneishDynamicFitIndexCategorical2023]. Missing data also make fit look better [@goretzkoStopUsingFixed2026].
- In a re-analysis of models in Psychological Assessment, tailored cutoffs were stricter than Hu and Bentler on average. Only 14.3% of models passed the DFI cutoff for CFI and RMSEA, and 33.3% for SRMR [@goretzkoEvaluatingModelFit2024, pp. 134-135].
- Equivalence testing (T-size RMSEA and CFI) is not a substitute. Its adjusted bins depend only on N and df and ignore loading size, and it was as over-optimistic as Hu and Bentler in simulation [@mcneishGeneralizabilityDynamicFit2023].
Match the tool to the model and data
Ask which tool produced the cutoffs, and check that it covers the model, estimator and data. Every tool below is out of its scope somewhere, and authors often use one outside it.
| Model | Tool | Not covered |
|---|---|---|
| One-factor or correlated factors, continuous, ML | cfaOne, cfaHB [@mcneishDynamicFitIndex2021] | Missing data, non-normality, bifactor, ESEM |
| Same, ordinal items with WLSMV/ULSMV | catOne, catHB [@mcneishDynamicFitIndexCategorical2023] | Binary items untested. The author calls the evidence preliminary |
| Higher-order (second-order) | hier1HB for the measurement part, hier2 for the structural part [@mcneishDynamicFitIndexHierarchical2024] | Continuous ML only. The structural part cannot be tested with three or fewer first-order factors |
| Bifactor, mediation, any covariance structure | DDDFI, the direct-discrepancy DFI [@mcneishDirectDiscrepancy2024] | Single group only. No mean structures, invariance or multilevel models |
| Invariance, two independent groups | Dynamic measurement invariance (DMI) [@mcneishDynamicMeasurementInvarianceTwoGroup2026] | WLSMV, partial invariance, more than two groups |
| Invariance, longitudinal or dyadic | Dependent-groups DMI [@mcneishDynamicMeasurementInvariance2026] | WLSMV, partial invariance |
The functions are in the dynamic R package [@wolfDynamicPackage2023]. The
invariance tools are separate web apps (see Software). Where a model falls
outside every tool, a custom parametric bootstrap from the fitted model is
the fallback, and the authors should describe it fully.
Reading simulation-based fit results
- No cutoff means inconclusive, not a pass. The DFI software reports "NONE" when the correct-model and misspecified-model distributions overlap too much. This is common with few categories, skewed items, weak loadings, small N and large models [@mcneishDynamicFitIndexCategorical2023; @mcneishDynamicFitIndexHierarchical2024]. Ask for local fit and, where possible, a larger sample instead.
- A pass rules out only misfit as large as the simulated one. Multifactor DFI adds one omitted cross-loading the size of the weakest loading. Goretzko and Wetzel [-@goretzkoStopUsingFixed2026, pp. 78-80] show a true cross-loading of 0.30 passing because DFI simulated one of 0.393. Read the DFI levels as a graded scale of severity, not an accept-reject rule, and say what size of misfit a pass excludes.
- The simulated misspecification is a yardstick, not a diagnosis. Exceeding a cutoff does not mean the model has the omitted cross-loading or residual correlation that was simulated [@mcneishDynamicFitIndex2021].
- Large samples make tailored cutoffs strict. Cutoffs shrink as N grows, so trivial misfit is rejected. Goretzko et al. [-@goretzkoEvaluatingModelFit2024, p. 135] found individual cutoffs performed reasonably only for n between about 300 and 700. With large samples, the local fit evidence carries the decision.
- Recompute after modifying the model. Cutoffs belong to a model. When items or factors are added or removed, or paths freed, new cutoffs are needed [@mcneishDynamicFitIndex2021].
- Report the full model, not only global indices. Ask for the full loading matrix, factor correlations, residual variances and residual correlations [@goretzkoEvaluatingModelFit2024, p. 139].
Local fit
Global indices average over the whole model and can hide a single large residual. Local fit is recommended by reporting guidelines but rarely reported [@mcneishHowPsychologistsDetermine2026]. Ask for the residual correlation matrix and for modification indices for residual covariances, and for any model changes to be justified substantively and validated on new data, since data-driven modification capitalizes on chance [@maccallumModelModifications1992]. Fixed screens such as residual correlations above ±.10 or modification indices above 10 are themselves heuristics. Ask for the values, sorted, and for simulation-based reference values where available, rather than a count of values above a fixed number.
Disagreement between indices is a clue to what is misspecified. When RMSEA fails and CFI and SRMR pass, forced zero cross-loadings (independent clusters) are a likely cause [@goretzkoEvaluatingModelFit2024, pp. 136-138]. With correlated factors, omitted cross-loadings can make fit look better, because the estimated factor correlation absorbs them [@groskurthWhyWeNeed2024, pp. 3903-3906].
Measurement invariance
- Fixed difference rules (ΔCFI ≤ .01 or the Chen values) were reproduced in the DMI work only when averaged across conditions. In single conditions the cutoff varied widely, for ΔCFI from −.005 to −.019 [@mcneishDynamicMeasurementInvarianceTwoGroup2026, pp. 145-146].
- Treating timepoints or partners as independent groups violates their dependence and can reverse the verdict. Use a single-group, wide-format model with the dependent-groups DMI [@mcneishDynamicMeasurementInvariance2026].
- Scalar invariance is the level needed to compare means across groups or time. Strict (residual) invariance is needed in addition when the paper compares observed score variances or reliability, or applies one cut score across groups. Ask for the level that matches the comparisons the paper makes.
- Ask for invariance across the groups the scale will be compared on (for example gender), not only across time, when the paper's use of the scale involves such comparisons.
- DMI tests exact invariance (the cutoff is the most extreme 1% of the simulated invariant distribution), so its cutoffs shrink toward zero as N grows and behave like the Δχ² test. McNeish advises against relying on DMI alone and recommends pairing it with an effect size such as dMACS, with correlation residuals and with probing of the mechanism [@mcneishDynamicMeasurementInvarianceTwoGroup2026, p. 152].
- Neither DMI version covers categorical estimation (WLSMV) or partial invariance. For those, and for more than two groups or continuous covariates, ask for MNLFA or MIMIC models, or say that no simulation-based route exists yet and ask for effect sizes with intervals.
Higher-order and bifactor models
- For a higher-order model, ask for measurement and structural misfit to be evaluated separately [@mcneishDynamicFitIndexHierarchical2024].
- Neither DFI version has a bifactor-specific procedure. The direct-discrepancy DFI covers bifactor models only because it ignores the model's form. In its own example the cutoffs barely differed across five competing models (RMSEA "Close" .032 to .035), so the comparison came down to raw fit, where the bifactor model won [@mcneishDirectDiscrepancy2024, p. 853]. Simulated cutoffs do not remove the bifactor model's fit advantage. Judge it by the strength indices (point 8).
- The direct-discrepancy misfit sizes (mean absolute discrepancy .038, .050, .060) are "admittedly heuristic", and the authors encourage researchers to set their own [@mcneishDirectDiscrepancy2024, pp. 847, 856]. Ask which values were used and why.
Two ways of calibrating a cutoff
This is secondary to the move away from fixed cutoffs, but worth knowing when reading what a manuscript did. The DFI cutoffs are set from the simulated misspecified model, as the value that rejects 95% of misspecified replications, then checked against the correct model. They therefore target power against a chosen misspecification. DMI, and simulation-based Rasch cutoffs such as those in easyRasch2, are set from the simulated correct model and control the false positive rate. Groskurth et al.'s tables are also correct-model quantiles. A null-calibrated cutoff says nothing about which misfit it would detect, and a misspecification-calibrated one says nothing about misfit smaller than the simulated one. Ask the authors to state which was used.
Many-facet Rasch models and rater-mediated designs
Many-facet Rasch measurement (MFRM, usually fitted in Facets) adds facets such as raters, criteria or occasions to the persons or objects being rated. The four criteria still apply, but the design creates problems of its own.
What the study can conclude
- A paper that compares MFRM with conventional summaries (means, endorsement rates, agreement coefficients) on one dataset, with no known truth, can show that the approaches give different summaries, not that the MFRM summary is more accurate. Ask for one of two framings. Either a worked illustration with the data and full code shared, or a small simulation under the study's own design (number of raters, objects, criteria and the observed category use) with known rater and object effects, showing what each approach detects.
- Small rating datasets are rarely too sensitive to share, and a methods paper should share them. If the analysis used a commercial or student version of a program, an open-software analysis alongside it makes the paper usable for readers without a license.
- Check the comparators. If the topic is content validity from an expert panel, the conventional comparator is the item- and scale-level content validity index [@politContentValidityIndex2007]. Reasonable modeling alternatives for a crossed design are generalizability theory, which gives variance components and a dependability coefficient [@brennanGeneralizabilityTheory2001], an ordinal mixed model with crossed random effects (with an adjacent-category logit link this is a random-effects rating scale model), and the hierarchical rater model [@patzHierarchicalRaterModel2002]. Ask why MFRM was chosen over them.
Dependence built into the design
- When one rater scores several criteria for the same object, the ratings within that rater × object cell are not independent (halo). The model treats them as independent given the facet parameters, so standard errors are too small, and separation reliability and the fixed-effects chi-square are overstated. Dependence also widens the sampling distribution of the fit statistics, so critical values that assume independence flag too many elements in both directions.
- Ask for a direct check. Residual correlations between each pair of criteria across the rater × object cells are the Q3 analogue for this design. Judge them against a reference distribution from a parametric bootstrap of the fitted model, not a fixed value [@christensenCriticalValuesYens2017]. A simpler check is the observed number of cells with the same rating on every criterion against the number the model expects. Facets' interaction (bias) analyses for rater × criterion and rater × object also bear on it.
- Ask how much the precision changes under a model that represents the nesting, such as a testlet model with a rater × object effect [@wang_rasch_2005] or the hierarchical rater model [@patzHierarchicalRaterModel2002], or under a per-criterion analysis.
- Dependence can extend further. Objects in the same domain, rated by the same rater in sequence, may share a rater × domain effect.
Dimensionality of the facets
A rating scale model puts every rating on one dimension. Heterogeneous objects and conceptually different criteria (relevance, clarity, usefulness) make this an assumption to test, not a given. A low criterion separation does not show that the criteria measure one construct. Ask for rater × criterion and object × criterion bias analyses, and for any substantive reading of an extreme object ("hardest to endorse") to be checked against the possibility that one criterion drives it.
Fit of raters, objects and criteria
- Mean-square ranges such as 0.5 to 1.5 come from Rasch Measurement Transactions notes, not from sampling theory, and ZSTD with ±1.96 depends on the number of observations [@mullerItemFitStatistics2020; @johanssonDetectingItemMisfit2025]. With a few raters, each contributing few ratings concentrated in two or three categories, the fit statistics have a wide sampling distribution and low power.
- Recommend critical values from a parametric bootstrap of the fitted model under the observed design. One underfitting rater pushes the others' fit statistics down. In Guo and Wind's [-@guoIterativeParametricBootstrap2021] motivating example, one misfitting rater among 15 caused six well-fitting raters to be flagged as overfitting. Their iterative procedure removes the underfitting raters and recomputes the critical values for the rest. Their simulations used 15 to 60 raters, so their detection rates should not be assumed for a small panel. Correct for the number of elements tested [@johanssonSimulationbasedCutoffsConditional2026].
- Facets uses joint maximum likelihood, and its fit statistics are unconditional. With few elements per facet or near-extreme raters, ask for the main results to be checked with marginal or conditional estimation [@robitzschComprehensiveSimulationStudy2021].
- Ask for the criterion behind "unexpected ratings" (the standardized residual threshold) to be stated.
- A non-significant global chi-square is not evidence of fit when the test has little power, and the percentage of variance explained by Rasch measures is not a fit statistic. Ask for both to be reported without a verdict.
- The Facets random (normal) chi-square tests whether the elements can be regarded as a random sample from a normal distribution, not whether they differ. Ask for it to be explained or dropped.
Rater measures, ceilings and agreement
- Raters with near-maximum averages have imprecise measures that a handful of ratings can move, and they can drive the headline rater spread in logits. Ask for the spread on the fair-average (raw score) metric as well, since that is what a reader relates to.
- Ask for one term for the rater facet (severity or leniency), with the orientation stated in the Methods.
- Under a ceiling, chance-corrected agreement is low even when raters rarely differ by more than one category [@feinsteinHighAgreementLow1990]. Krippendorff's alpha [@hayesAnsweringCallStandard2007] also counts systematic leniency differences as disagreement, so a low alpha and a large MFRM rater spread can be one finding seen twice, not two. Ask for the level of measurement used for alpha, the rating units, a bootstrap confidence interval and adjacent (within one category) agreement.
- Unused low categories do not by themselves show a poor response scale. A favourably rated object, social desirability, and endpoint-only labels are alternative explanations with different implications. Ask for them to be discussed before scale changes are recommended. Open comments from the raters can help explain misfitting raters and extreme objects.
Reliability and Bayesian alternatives
- Separation reliability inherits the standard errors, so it is overstated under dependence. With few elements per facet, an interval is more informative than a point value. In a Bayesian random-effects version of the model, relative measurement uncertainty gives a reliability estimate with a credible interval for each facet [@bignardi_general_2025]. With few elements, the posterior depends on the prior for the facet standard deviation, so ask for a prior sensitivity check.
- For model checking of a Bayesian ordinal model, bar plots of predicted against observed category frequencies say little, since almost any model reproduces the marginals. Recommend PAV-adjusted calibration plots of the cumulative event probabilities with leave-one-out predictions [@sailynojaRecommendationsVisualPredictive2025].
Software for MFRM
TAM fits many-facet rating scale models by marginal maximum likelihood, and
immer implements the hierarchical rater model and related rater models
[@robitzschItemResponseModels2018]. G-study variance components can be
estimated with lme4. Ordinal mixed models can be fitted with ordinal, or
with brms [@burknerBrmsPackageBayesian2017] using family = acat(). The
parametric bootstrap cutoffs in easyRasch2 can serve as a template for the
facets model.
Methodological hygiene
- For ordinal items, start from an estimator made for ordinal data: WLSMV,
ULSMV or DWLS on polychoric correlations, or a full-information ordinal
(IRT) estimator [@rhemtullaWhenCanCategorical2012;
@liConfirmatoryFactorAnalysis2016; @sellbomFactorAnalysis2019]. Report the
scaled or robust fit indices. Treating the items as continuous (ML or MLR)
assumes that the response categories are equally spaced, so a choice of MLR
needs justifying. Close to equally spaced thresholds reduce the loading bias
but do not remove it, and with high loadings and a large sample even a small
bias leaves confidence intervals too narrow. With symmetric thresholds and
five or more categories, ML coverage of the loadings was around 90% for a
nominal 95% [@rhemtullaWhenCanCategorical2012, p. 366]. Reject plain ML.
Common defences, and why they fail:
- "Five or more categories, so the items can be treated as continuous." The rule is usually credited to Rhemtulla et al. [-@rhemtullaWhenCanCategorical2012], but their recommendation of ML has three conditions: "5 or more categories, sample size is small, and category thresholds are approximately symmetric" (p. 354). Even with five to seven categories, the categorical estimator gave more accurate loadings (p. 370). Their text reports that ML loading bias was never greater than 10% with five or more categories (p. 370), but the bias table in their supplement (p. 4, absolute bias) shows otherwise when converted to relative bias. With extreme asymmetry, ML loadings were biased by about −16% with five categories (λ = .6) and by more than −10% in some six- and seven-category conditions. The bias does not shrink as N grows while the standard errors do, so coverage worsens with sample size: with asymmetric thresholds and N = 600, ML coverage of the loadings fell as low as 46% (p. 366). Ask for the category distributions of each item. A count of categories does not justify MLR.
- "No excessive skew or kurtosis, so ML is fine." The criterion is not the shape of the distribution. Categorization attenuates Pearson covariances unevenly, depending on how far apart the thresholds are, whatever the marginal shape [@muthenComparisonMethodologies1985; @floraEmpiricalEvaluation2004]. Check the claim against the item distributions too, since floor-loaded symptom scales often contradict it.
- "WLSMV always improves fit." It can move fit either way. In Sellbom and Tellegen's [-@sellbomFactorAnalysis2019] example, MLR rejected a model that WLSMV and ULSMV found acceptable, with weaker ML loadings. Which estimator gives nicer indices is not a criterion for choosing one.
- The simulation comparisons concern robust ML (MLR, [@yuanThreeLikelihoodbasedMethods2000]). Plain ML has no scaling correction and is weaker still.
- A side-by-side ML versus WLSMV table is not a sensitivity analysis. The indices come from different input matrices and are not on the same scale, so agreement does not reassure and disagreement does not diagnose. Ask for one categorical estimator throughout.
- Missing data under WLSMV: lavaan offers listwise or pairwise deletion. Multiple imputation before WLSMV, or a full-information ordinal (IRT) estimator, are the alternatives that use all the data. Ask which was used, and for the analytic n of each fitted model.
- If models are compared and selected on the same data on which fit is then reported, the analysis is exploratory and should be described as such. With a large sample, a split-sample (exploratory and confirmatory) design is an option.
- Software reporting: the R and package versions (lavaan, dynamic), not the
IDE version, plus the model syntax and the call, including the estimator,
the
orderedsetting and the missing-data option. - The introduction should appraise earlier validation studies, not list them as equally trustworthy. For the key studies, note the estimator, whether fit was judged against fixed cutoffs, and whether local dependence was checked.
- Distinguish underfit from overfit explicitly. Relate item fit to CFA loadings or IRT discrimination.
- Unconditional item fit (Winsteps, RUMM, TAM, eRm) is unreliable from about N = 200 [@mullerItemFitStatistics2020]. Use conditional item fit (iarm, easyRasch2). Outfit rarely adds value over infit [@mullerItemFitStatistics2020; @johanssonDetectingItemMisfit2025; @johanssonSimulationbasedCutoffsConditional2026].
- Estimation method matters when the software uses joint maximum likelihood (Winsteps, Facets). For a comparison of JML variants, MML, CML and limited-information methods across sample size, test length and ability distributions, cite [@robitzschComprehensiveSimulationStudy2021]: MML with a flexible ability distribution was at least as efficient as CML, and penalized or ε-adjusted JML performed well for long tests. This is useful when asking for JML results to be checked with CML or MML, especially for short tests or few elements per facet.
Extra checks in self-review
These are things a referee cannot see but the author can, and they are where avoidable rejections come from.
- Numbers. Every number in the text, abstract and tables matches the analysis output. Check the item counts, n, dropped items and respondents, and the cutoffs actually applied.
- Code against text. The estimator, software version, cutoffs, iterations and seeds described in the Methods are the ones in the code.
- Reporting. Items are reported with their number of response categories, sample sizes are stated per analysis, and missing-data handling is described. Compare with the basic and expanded reporting recommendations in [@johanssonValidReliableBasic2026] and any guideline the journal requires.
- Reproducibility. Seeds, package versions and the analysis code are reported or shared.
- Anticipated reviewer questions. For each criterion: is there a reported test, which cutoff was used, and how is it justified? Any criterion without a test is the first thing a reviewer will ask about.
Rank each issue as must fix before submission (a reviewer would likely reject the paper or ask for major revision), should fix (a reviewer would likely ask for it) or consider (it would make the paper stronger). A criterion that applies to the model but has no reported test is a must fix. Give the concrete fix (the analysis to add, the sentence to change, the citation to add).
Software to recommend
- easyRasch2 (R, CRAN) and its jamovi module (GUI, Windows, Mac and Linux, https://jamovi.org): simulation-based cutoffs, conditional item fit, partial-gamma item-restscore, local dependence and DIF. Package site: https://pgmj.github.io/easyRasch2/.
- dynamic (R, Wolf and McNeish [-@wolfDynamicPackage2023]):
simulation-based CFA fit cutoffs, with
cfaOne,cfaHB,catOne,catHB,hier1HB,hier2andDDDFI. It was archived from CRAN on 2025-11-07, so install it from GitHub (https://github.com/melissagwolf/dynamic). Web app: https://dynamicfit.app. The invariance cutoffs are separate apps: https://www.dynamicfit.app/MI (two groups) and https://dynamicfit.app/MIDG/ (longitudinal and dyadic). - lavaan [@rosseelLavaanPackageStructural2012] for the CFA itself
(
ordered = TRUE,estimator = "WLSMV"), and semTools or BifactorIndicesCalculator for omega-hierarchical, ECV and PUC, with bootstrap intervals by resampling. - DIGRAM: graphical loglinear Rasch models (Windows, WINE on Mac and Linux).
Output
Start from the template for the mode and fill it in:
- Self-review:
templates/self-review.qmd - Appraisal of a published paper:
templates/appraisal.qmd - Referee report:
templates/referee-report.qmd
All three render to PDF with Quarto and Typst. Put a refs.bib and apa.csl in
the same folder as the .qmd, because Typst cannot read a bibliography or CSL
file outside the render folder. Copy references/apa.csl, and copy the
entries the report cites from references/refs.bib. Render with
quarto render <file>.qmd --to typst and confirm that no citation keys are
unresolved.
Without Quarto, write the same structure in Markdown, with citations as author-year text and a reference list at the end.
Writing style
Direct, plain sentences, one point per paragraph. No em dashes, no semicolons, no emphatic "exactly", and no announcing sentences ("In this section I will..."). A referee report opens with a short summary that credits real strengths and states the recommendation, and closes constructively. A self-review memo opens with the must-fix list.
Files
SKILL.md: this guide.templates/self-review.qmd: internal pre-submission review memo.templates/appraisal.qmd: appraisal memo on a published paper.templates/referee-report.qmd: reviewer report for a journal.references/refs.bib: the reference library, grouped by topic. Every key cited in this guide and the templates is in it.references/apa.csl: APA 7th edition citation style (CC BY-SA 3.0, from the Citation Style Language project).