P-122 - CAN AN ARTIFICIAL INTELLIGENCE ASSESS SURGICAL SKILLS: THE TURING TEST FOR ENDOVASCULAR SURGERY

TOPIC:
Education & Training
AUTHORS:
Saricilar E. (Royal North Shore Hospital ~ Sydney ~ Australia)
Introduction:
Endovascular surgery is an ever-growing interventional choice in vascular surgery. With the shift in technical skills, there should be a change in assessment methods. High-fidelity simulations are used in training, with limited evidence for assessment. This study assessed if raw objective metrics from high-fidelity simulators could correlate to traditional assessments and whether an artificial intelligence could replace an assessor.
Methods:
A single-centre observational cohort study was performed on 22 registrars, fellows and consultants in vascular surgery, interventional radiology and general surgery who performed an identical endovascular intervention on a left external iliac stenosis on a high-fidelity simulator. The procedure was assessed on a modified Reznick scale, then compared to extracted metric data. A feed forward neural network (AI) was trained using metrics to output traditional scores.
Results:
Significant correlations were found for all scores on the modified Reznick scale besides respect for tissue. The neural network could accurately predict scale scores for all criteria with a weighted Cohen's kappa of 0.16 to 0.56. Additionally, multiple correlations between raw metrics and modified Reznick scale scores were identified such as total procedure time correlating time and motion (p = 0.038) and residual stenosis reflecting flow of operation (p = 0.03).
Conclusion:
With ongoing upskilling of trainees to endovascular surgery, a model to assess skills more rigorously, objectively and regularly should be established. In its current state, high fidelity simulations are able to reflect the vast majority of technical competencies expected in a surgical trainee. High-fidelity simulation models have shown ability to accurately define the majority of skills required in a vascular surgery trainee. The time-change of skills in a trainee can be better qualified with high-fidelity models. AI has shown promise to be coupled with high-fidelity simulators to create complex yet accurate frameworks to assess competence in trainees and should be a significant field of future research.