Developing Short Psychological Scale Forms Using Simulated Data: A Comparison of Classical Selection, Item Response Theory, LASSO Regularization, Supervised Construct Scoring, and Genetic Algorithms
DOI:
https://doi.org/10.55074/hesj.vi52.1727Keywords:
LASSO Regression, Genetic Algorithms, Supervised Construct Scoring, Monte Carlo Simulation, Scale ShorteningAbstract
This study aims to develop and compare short forms of psychological scales using simulated data generated through a Monte Carlo design. Five prominent scale-shortening techniques were evaluated: Classical item selection, Item Response Theory (IRT), LASSO regularization, Supervised Construct Scoring (SCS), and Genetic Algorithms (GA). Simulated datasets were generated for a 24-item scale measuring three latent dimensions, while systematically manipulating three methodological factors: factor strength (strong vs. moderate), sample size (300, 600, 1000), and short-form length (12-item vs. 8-item versions). The performance of the five methods was assessed using four key psychometric criteria: latent trait recovery, structural validity via confirmatory factor analysis (CFA), external validity through correlations with a criterion variable, and internal consistency (Cronbach’s Alpha and Omega). Results showed that 12-item short forms consistently achieved superior psychometric performance across all methods, whereas 8-item versions demonstrated noticeable declines under moderate factor strength. LASSO and SCS exhibited clear advantages in latent trait recovery, lower estimation error, structural validity, and external validity, outperforming Classical, IRT, and GA methods—particularly in moderate-strength conditions. Sample size had minimal influence, while factor strength and short-form length were the primary determinants of overall performance. These findings highlight the value of integrating supervised learning and regularized regression techniques in developing accurate and reliable short forms of psychological scales, offering important methodological guidance for researchers seeking to optimize measurement efficiencyDownloads
References
قرة، نور الدين وشريفي، شعبان ولعربي، سمير. (2023). تقييم الخصائص السيكومترية للنسخة المختصرة لمقياس التعاطف مع الذات- دراسة على عينة من الطلبة بجامعة محمد بوضياف بالمسيلة- الجزائر. مجلة الروائز، 7(2)، 32-52. https://asjp.cerist.dz/en/article/235082
Abal, F. J. P., González, J. F. S., & Attorresi, H. F. (2024). Big five inventory abreviado desde lateoría de respuesta al ítem. Acta colombiana de psicologia. https://doi.org/10.14718/acp.2024.27.2.4
Alarcon, G. M., & Lee, M. A. (2022). The Relationship of Insufficient Effort Responding and Response Styles: An Online Experiment. Frontiers in Psychology, 12, 784375. https://doi.org/10.3389/FPSYG.2021.784375/BIBTEX
Alley, L., Kadolkar, I., Gupta, A., Cortina, J. M., & Winsler, K. P. (2024). Grammatical Redundancy in Scales: Using the “ConGRe” Process to Create Better Measures. Journal of Management. https://doi.org/10.1177/01492063241291542;PAGE:STRING:ARTICLE/CHAPTER
Bain, C., & Shi, D. (2024). Investigating variable selection techniques under missing data: A simulation study. https://doi.org/10.31234/osf.io/d4c2k
Bain, C., Shi, D., Boness, C. L., & Stevens, J. (2023). A simulation study comparing the use of supervised machine learning variable selection methods in the psychological sciences. https://doi.org/10.31234/osf.io/y53t6
Bißantz, S., Frick, S., Melinscak, F., Iliescu, D., & Wetzel, E. (2024). The potential of machine learning methods in psychological assessment and test construction. European Journal of Psychological Assessment. https://doi.org/10.1027/1015-5759/a000817
Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06
Coste, J., Guillemin, F., Pouchot, J., & Fermanian, J. (1997). Methodological approaches to shortening composite measurement scales. Journal of clinical epidemiology, 50(3), 247–252. https://doi.org/10.1016/s0895-4356(96)00363-0
Crone, D. L., Rhee, J. J., & Laham, S. M. (2020). Developing brief versions of the Moral Foundations Vignettes using a genetic algorithm-based approach. Behavior Research Methods 2020 53:3, 53(3), 1179–1187. https://doi.org/10.3758/S13428-020-01489-Y
DeVellis, R. F. (2017). Scale Development: Theory and Applications (4th ed.). Thousand Oaks, CA: Sage.
El Keshky, M., Al-Qarni, M., & Khayat, A. (2022). Adaptation and psychometric properties of an Arabic version of the smartphone addiction scale (SAS) in the context of Saudi Arabia. Addictive behaviors, 131, 107335. https://doi.org/10.1016/j.addbeh.2022.107335
Embretson, S.E., & Reise, S.P. (2000). Item Response Theory for Psychologists (1st ed.). Psychology Press. https://doi.org/10.4324/9781410605269
Fekih-Romdhane, F., Azzi, V., Obeid, S., Gerges, S., Sarray El Dine, A., Malaeb, D., Soufia, M., & Hallit, S. (2023). Psychometric properties of an Arabic translation of the short form of the metacognition questionnaire (MCQ-30) in a non-clinical adult sample. BMC psychiatry, 23(1), 795. https://doi.org/10.1186/s12888-023-05308-4
Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1), 1–22. https://doi.org/10.18637/jss.v033.i01
Gonzalez, O. (2021). Psychometric and Machine Learning Approaches to Reduce the Length of Scales. Multivariate Behavioral Research, 56(6), 903–919. https://doi.org/10.1080/00273171.2020.1781585
Haroz, E. E., Kane, J. C., Nguyen, A. J., Bass, J. K., Murray, L. K., & Bolton, P. (2020). When less is more: reducing redundancy in mental health and psychosocial instruments using Item Response Theory. Global Mental Health, 7, e3. doi:10.1017/gmh.2019.30
Jung, S. J., & Seo, J. W. (2025). A transformer-based embedding approach to developing short-form psychological measures. Frontiers in Psychology, 16, 1640864. https://doi.org/10.3389/FPSYG.2025.1640864/BIBTEX
Kam, C. C. S. (2019). Careless responding threatens factorial analytic results and construct validity of personality measure. Frontiers in Psychology, 10(JUN), 461631. https://doi.org/10.3389/FPSYG.2019.01258/BIBTEX
Kemper, C. J., Trapp, S., Kathmann, N., Samuel, D. B., & Ziegler, M. (2019). Short Versus Long Scales in Clinical Assessment: Exploring the Trade-Off Between Resources Saved and Psychometric Quality Lost Using Two Measures of Obsessive–Compulsive Symptoms. Assessment, 26(5), 767–782. https://doi.org/10.1177/1073191118810057
Kilmen, S., & Bulut, O. (2025). Shortening Psychological Scales: Semantic Similarity Matters. Educational and Psychological Measurement, 85(5), 910–934. https://doi.org/10.1177/00131644251319047
Kleka, P., & Soroko, E. (2018). How to Abbreviate Questionnaires and Avoid the Sins? Survey Research Methods, 12(2), 147–160. https://doi.org/10.18148/SRM/2018.V12I2.7224
Koğar, H. (2020). Development of a Short Form: Methods, Examinations, and Recommendations. Journal of Measurement and Evaluation in Education and Psychology, 11(3), 302–310. https://doi.org/10.21031/EPOD.739548
Lin, G., Liu, J., Lee, S. C., Wu, B., Li, S., Chiu, H., Wang, S., & Hsieh, C. (2024). Developing a machine learning-based short form of the positive and negative syndrome scale. Asian Journal of Psychiatry. https://doi.org/10.1016/j.ajp.2024.103965
Lowman, K. L., Costello, T. H., Stewart, R. A., Joyner, K. J., Bertoldi, B. M., & Patrick, C. J. (2025). Creating Scale Measures of Latent Factors: A Genetic Algorithmic Approach. Assessment. https://doi.org/10.1177/10731911251376230
Luo, Y., Chen, Q., Chen, J., & Zhan, P. (2024). Development and validation of two shortened anxiety sensitive index-3 scales based on item response theory. Humanities and Social Sciences Communications 2024 11:1, 11(1), 1078-. https://doi.org/10.1057/s41599-024-03615-z
Messick, S. (1994). Validity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. ETS Research Report Series, 1994(2), i–28. https://doi.org/10.1002/J.2333-8504.1994.TB01618.X
Obbarius, A., Ehrenthal, J. C., Fischer, F., Liegl, G., Obbarius, N., Sarrar, L., & Rose, M. (2021). Applying Item Response Theory to the OPD Structure Questionnaire: Identification of a Unidimensional Core Construct and Feasibility of Computer Adaptive Testing. Journal of Personality Assessment, 103(5), 645–658. https://doi.org/10.1080/00223891.2020.1828435
Olaru, G., & Danner, D. (2021). Developing cross-cultural short scales using ant colony optimization.. Assessment, 28 (1), 199-210. https://doi.org/10.1177/1073191120918026
Olaru, G., & Jankowsky, K. (2021). The hex-aco-18: Developing an age-invariant hexaco short scale using ant colony optimization.. Journal of Personality Assessment, 1-12. https://doi.org/10.1080/00223891.2021.1934480
Passarelli, M., Casetta, L., Rizzi, L., Chiorri, C., Cassina, F., Voi, S., & Rocco, D. (2024). Short and sweet: Comparing strategies for the reduction of questionnaires on self-criticism and social safeness while preserving construct validity. International Journal of Psychology, 59(6), 1234–1244. https://doi.org/10.1002/IJOP.13249
Pflanz, C. P., Gallacher, J., & Bauermeister, S. (2024). A psychometric evaluation of the 16-item PHQ-ADS concomitant anxiety and depression scale in the UK biobank using item response theory. Journal of Affective Disorders, 347, 335–344. https://doi.org/10.1016/J.JAD.2023.11.067
Raborn, A. W., Leite, W. L., & Marcoulides, K. M. (2020). A Comparison of Metaheuristic Optimization Algorithms for Scale Short-Form Development. Educational and Psychological Measurement, 80(5), 910–931. https://doi.org/10.1177/0013164420906600
Raborn, A., & Leite, W. L. (2018). Shortform: An r package to select scale short forms with the ant colony optimization algorithm.. Applied Psychological Measurement, 42 (6), 516-517. https://doi.org/10.1177/0146621617752993
Revelle, W. (2023). psych: Procedures for psychological, psychometric, and personality research (Version 2.3.9) [Computer software]. Northwestern University. https://CRAN.R-project.org/package=psych
Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02
Sahdra, B. K., Ciarrochi, J., Parker, P. D., & Scrucca, L. (2016). Using genetic algorithms in a large nationally representative american sample to abbreviate the multidimensional experiential avoidance questionnaire. Frontiers in Psychology, 7 (189), 189-189. https://doi.org/10.3389/FPSYG.2016.00189
Schroeders, U., Morgenstern, M., Jankowsky, K., & Gnambs, T. (2024). Short-scale construction using meta-analytic ant colony optimization. European Journal of Psychological Assessment. https://doi.org/10.1027/1015-5759/a000818
Schroeders, U., Wilhelm, O., & Olaru, G. (2016). Meta-heuristics in short scale construction: Ant colony optimization and genetic algorithm.. PLOS ONE, 11 (11),. https://doi.org/10.1371/JOURNAL.PONE.0167110
Scrucca, L. (2013). GA: A package for genetic algorithms in R. Journal of Statistical Software, 53(4), 1–37. https://doi.org/10.18637/jss.v053.i04
Speer, A. B., Delacruz, A. Y., & Wegmeyer, L. J. (2025). Measuring work attitudes with less: Supervised construct scoring to shorten work attitude measures. Journal of Occupational and Organizational Psychology, 98(1). https://doi.org/10.1111/JOOP.70001
Speer, A. B., Perrotta, J., & Jacobs, R. R. (2023). Supervised Construct Scoring to Reduce Personality Assessment Length: A Field Study and Introduction to the Short 10. Organizational Research Methods, 27(2), 223-264. https://doi.org/10.1177/10944281221145694
Steger, D., Weiss, S., & Wilhelm, O. (2023). The short inventory of creative activities (s-ica):Compiling a short scale using ant colony optimization. Creativity Research Journal, 1-16. https://doi.org/10.1080/10400419.2022.2128574
Tanaka, M., Kawakami, A., Sakagami, K., Terai, T., Fernandez, J., Keefer, L., & Ito, H. (2024). Development and validation of a 13-item short version of the inflammatory bowel disease self-efficacy scale. BMC Gastroenterology 2024 24:1, 24(1), 190-. https://doi.org/10.1186/S12876-024-03206-X
Venables, W. N., & Ripley, B. D. (2002). Modern applied statistics with S (4th ed.). Springer.
Wang, C., Zhu, R., & Xu, G. (2022). Using lasso and adaptive lasso to identify DIF in multidimensional 2pl models. Multivariate Behavioral Research, 58 (2), 387- 407. https://doi.org/10.1080/00273171.2021.1985950
Wickham, H., Averick, M., Bryan, J., Chang, W., D’Agostino McGowan, L., François, R., ... & Yutani, H. (2019). Welcome to the tidyverse. Journal of Open Source Software, 4(43), 1686.https://doi.org/10.21105/joss.01686
Wise, T., & Sidarus, N. (2024). Reducing the burden of psychological questionnaire measures through selective item re-weighting. Royal Society Open Science, 12(4). https://doi.org/10.1098/RSOS.241857
Yarkoni, T. (2010). The abbreviation of personality, or how to measure 200personality scales with 200 items. Journal of Research in Personality,44, 180 –198. http://dx.doi.org/10.1016/j.jrp.2010.01.002
Yu, Z., Jie-hui, Y., Tan, J., Zou, X., Su, J., & Xue, J. (2024). The simplification of the symptomchecklist-90 scale utilizing machine learning techniques. Journal of Affective Disorders, 366, 181-188. https://doi.org/10.1016/j.jad.2024.08.121
Zhou, D. J., Chahal, R., Gotlib, I. H., & Liu, S. (2024). Comparison of Lasso and Stepwise Regression in Psychological Data. Methodology, 20(2), 121–143. https://doi.org/10.5964/METH.11523
Zimny, L., Schroeders, U., & Wilhelm, O. (2024). Ant colony optimization for parallel test assembly. Behavior Research Methods. https://doi.org/10.3758/s13428-023-02319-7
Downloads
Published
Issue
Section
How to Cite
