Draw My Life: Using Vision Language Models in Robot-Mediated Visual Storytelling to Support Self-Disclosure and Detect Intimate Conversations

Disclosure is a key step in helping individuals facing psychosocial difficulties, yet many refrain from speaking about such difficulties due to fear, shame, or related concerns. We present “Draw My Life”, a robot-mediated visual storytelling activity designed to support self-disclosure and detect intimate conversations. The activity relies on two modalities, speech and drawings, with the aim of providing the user an unconstrained and relaxing way to express themselves. The robot uses a Vision Language Model to autonomously process both modalities and interact with the user following an active listening approach. We conducted a two-conditions withinsubject study, with 25 participants, manipulating the level of intimacy of the conversation topics to disclose about with the robot, and exploring whether features extracted from the participants' multimodal data could be used to predict back the level of intimacy of the conversation topics (RQ1). A logistic regression model achieved the best performance in this task (accuracy =0.74,ROC−AUC=0.770, permutation test p-value =0.002). A feature importance analysis (RQ2) revealed that the most predictive feature lied within the participants' speech content. ... mehrFeatures extracted from the drawing content and drawing behavior also showed good predictive power, demonstrating the importance of multimodal analysis when trying to detect intimate conversations. Features extracted from the speech behavior did not showcase important predictive power. Lastly, a thematic analysis of the participants' open feedback indicated an overall positive perception of the developed system, but also identified areas for technical improvement (RQ3).