Our team has been focused on the development and optimization of the ABN AMRO customer service chatbot, Anna, which is engineered to provide clients with prompt and precise information, ensuring an efficient user experience.

To quantitatively assess user interactions with Chatbot Anna, we utilize endpoint analysis. This involves analysing conversational data to pinpoint 'natural' endpoints, such as queries about usefulness or requests for human handover. At the Objective and Key Results (OKR) level, we classify these endpoints into resolved, unresolved, handover, and unknown categories. Our analysis places particular emphasis on the 'unknown' category, where the conversational data lacks clarity regarding the interaction outcome.

With advancements in large language models, we investigated it's potential to extract insights from the 'unknown' category. By analysing this data, we aimed to obtain a comprehensive understanding of Chatbot Anna's performance and to discern patterns that contribute to unproductive interactions. Identifying non-constructive conversational elements enables us to perform more accurate analyses and enhance the chatbot's functionality. This approach also minimizes the necessity for manual conversation labelling. To accomplish this, we implemented a semi-supervised learning strategy and leveraged few-shot learning techniques to fine-tune our OpenAI model. This initiative led to the discovery of valuable insights that could be advantageous for other technical teams within or outside our organization.

In this presentation, we will share this journey, and the insights gained from this endeavour. We believe that our results will deepen our understanding of user interactions and foster future technological innovations within our team.

Our team has been focused on the development and optimization of the ABN AMRO customer service chatbot, Anna, which is engineered to provide clients with prompt and precise information, ensuring an efficient user experience.

To quantitatively assess user interactions with Chatbot Anna, we utilize endpoint analysis. This involves analysing conversational data to pinpoint 'natural' endpoints, such as queries about usefulness or requests for human handover. At the Objective and Key Results (OKR) level, we classify these endpoints into resolved, unresolved, handover, and unknown categories. Our analysis places particular emphasis on the 'unknown' category, where the conversational data lacks clarity regarding the interaction outcome.

With advancements in large language models, we investigated it's potential to extract insights from the 'unknown' category. By analysing this data, we aimed to obtain a comprehensive understanding of Chatbot Anna's performance and to discern patterns that contribute to unproductive interactions. Identifying non-constructive conversational elements enables us to perform more accurate analyses and enhance the chatbot's functionality. This approach also minimizes the necessity for manual conversation labelling. To accomplish this, we implemented a semi-supervised learning strategy and leveraged few-shot learning techniques to fine-tune our OpenAI model. This initiative led to the discovery of valuable insights that could be advantageous for other technical teams within or outside our organization.

In this presentation, we will share this journey, and the insights gained from this endeavour. We believe that our results will deepen our understanding of user interactions and foster future technological innovations within our team.