Introduction: Situational Judgment Tests (SJTs) are widely used in personnel selection, training, and education, yet the process of developing SJTs remains resource-intensive and methodologically complex. Recent advances in artificial intelligence (AI) offer new opportunities to streamline and enhance SJT development. Although empirical research on the use of AI in SJT development is only somewhat recently emerging, the number of published and unpublished studies has quickly grown such that this research is now amenable to meta-analysis.


Purpose: This study conducts a systematic review and a preliminary meta-analysis of existing research on the use of AI in SJT construction. We classify the types of automation (e.g., scenario generation, response option development, scoring algorithms) and evaluate their impact on psychometric quality and validity.


Method: Using PRISMA guidelines, we identify and review peer-reviewed publications and technical reports (k = 15 studies) that apply AI methods in SJT design. Studies are coded for AI methodology, stage of SJT development, validation design, and reported outcomes (e.g., job performance, grades, other behavioral criteria, etc.). Where sufficient quantitative data are available, we conduct a meta-analysis to synthesize validity evidence across approaches.


Results: Meta-analytic findings will quantify whether AI-assisted methods can enhance efficiency and scalability in SJT development. If automated scoring models, particularly those using natural language processing and machine learning, demonstrate comparable or superior criterion-related validity to traditional scoring approaches, this would have important practical implications for organizations and researchers that may be able to develop and score SJTs more quickly and less expensively.


Conclusions: This systematic review contributes a structured framework for understanding current best practices and guiding future research on AI-enabled assessment development for SJTs in particular. We offer best practices and recommendations for improving the development of SJTs in a less resource-intensive and cost-effective manner.