Customer service employees frequently face emotionally demanding interactions, including unreasonable customer behavior and high-pressure communication, which can lead to elevated stress and reduced workplace performance. To address these challenges, we developed a multimodal training system utilizing large language models (LLMs) to simulate realistic customer interactions and provide personalized feedback. The system integrates two core components: a role-play agent that generates psychologically grounded customer scenarios, and a feedback agent that delivers individualized conversational guidance based on social psychology principles.
We conducted a four-week field study with 12 call center employees to evaluate the system's effectiveness in a real-world setting. Participants reported notable improvements in interpersonal communication skills, increased motivation to engage in further skill development, and enhanced overall productivity. To ensure psychological safety during training, we also explored a non-invasive stress measurement method using millimeter-wave signals from a contactless sensor. The collected data enabled the construction of a stress estimation model with high accuracy, allowing adaptive support during emotionally intense role-play sessions.
Our findings suggest that LLM-based multimodal role-play systems can serve as effective tools for improving customer service competencies while supporting employee well-being. This research contributes to the design of psychologically informed AI training environments and offers practical implications for scalable, personalized skill development in high-stress occupations.