Introduction
Integrating community members into the development of high-stakes assessments can ensure relevance, equity, and trust. Their insights can help align assessments with real-world needs, reduce bias, and foster shared ownership of assessment decision outcomes.
Purpose
We explored outcomes related to the incorporation of real-world patients into the development of a national high-stakes assessment used for the selection of future healthcare professionals.


Methods
Real world patients were invited to participate in a critical incident study involving the collection of situations in which they encountered medical students demonstrating ineffective or effective professionalism behaviors. Situational judgment tests (SJTs) created from these collected vignettes were compared to SJTs developed from traditional subject matter experts (SMEs; medical school faculty and staff) in terms of professionalism domains represented, number of initial item responses created, scoring survival rates, and item response effectiveness patterns.


Results
Sixty situational judgment test scenario sets were analyzed, with 25% of these developed from critical incidents provided by patient SMEs. Fisher Exact Tests revealed no differences in the type of critical incidents submitted based on SME type (patient versus medical school faculty/staff), as determined by competency domain represented. Additionally, no significant interactions emerged between SME type and competency domain for the number of initial items created, item effectiveness patterns, or item survival rates, suggesting that critical incidents provided by patients perform similarly to traditional SMEs in terms of opportunities to create an array of relevant response items and their ability to survive through independent scoring analysis .


Discussion
Involving relevant community members in the development of high-stakes selection tools yields outcomes comparable to traditional expert-driven methods. These results suggest that inclusive design does not compromise rigor, and can instead enhance relevance, variety of content, transparency , and public trust in the selection process.