1391 - AUTOMATED OSCE SCORING USING ARTIFICIAL INTELLIGENCE

Session: P_D02S002 - Poster Session 2 - Division 2
AUTHORS:
Lan Yu-Ling (National Dong Hwa University ~ Hualien ~ Taiwan) , Wang Ying-Fen (National Kaohsiung Normal University ~ Kaohsiung ~ Taiwan) , Chen Wan-Lan (Tzu Chi University ~ Hualien ~ Taiwan)
Abstract text:
Objective structured clinical examinations (OSCEs) have been gradually used in professional psychology to assess clinical competence and facilitate clinical training courses. However, the scoring of OSCE, especially the standardized patient (SP) station, usually uses psychologists with OSCE rater training to evaluate student performance, which requires substantial time and considerable costs. Artificial intelligence, combined with machine learning techniques, has been used in several psychotherapy studies to automatically evaluate the content of client-psychologist interactions and has demonstrated a certain degree of credibility. This study aimed to utilize Google Gemini 2.5 Flash with the support vector machine (SVM), a frequently used machine learning model, to automate the evaluation of student performance on SP stations of the Intake OSCE. A total of 30 students' performances on a 20-minute SP station were included in this study. The SVM model achieved a Precision of 83.3% and an Accuracy of 73.3%. However, the F1 score and Recall were 55.6% and 41.7%, respectively. These findings support the potential of using artificial intelligence and machine learning techniques for automated OSCE scoring in professional psychology.