Designing a Conversation Mining System for Customer Service Chatbots

As chatbots are gaining popularity in customer service, it is critically important for companies to
continuously analyze and improve their chatbots’ performance. However, current analysis approaches
are often limited to the question-answer level or produce highly aggregated metrics (e.g., conversations
per day) instead of leveraging the full potential of the large volume of conversation data to provide
actionable insights for chatbot developers and chatbot managers. To address this challenge, we
developed a novel chatbot analytics approach — conversation mining — based on concepts and methods
from process mining. We instantiated our approach in a conversation mining system that can be used
to visually analyze customer-chatbot conversations at the process level. The results of four focus group
evaluations suggest that conversation mining can help chatbot developers and chatbot managers to
extract useful insights for improving customer service chatbots. Our research contributes to research
and practice with novel design knowledge for conversation mining systems.