E-chat: Emotion-sensitive Spoken Dialogue System with Large Language Models

A work submitted to ISCSLP2024



1. Abstract

This study focuses on emotion-sensitive spoken dialogue in human-machine speech interaction. With the advancement of Large Language Models (LLMs), dialogue systems can handle multimodal data, including audio. Recent models have enhanced the understanding of complex audio signals through the integration of various audio events. However, they are unable to generate appropriate responses based on emotional speech. To address this, we introduce the Emotional chat Model (E-chat), a novel spoken dialogue system capable of comprehending and responding to emotions conveyed from speech. This model leverages an emotion embedding extracted by a speech encoder, combined with LLMs, enabling it to respond according to different emotional contexts. Additionally, we introduce the E-chat200 dataset, designed explicitly for emotion-sensitive spoken dialogue. In various evaluation metrics, E-chat consistently outperforms baseline LLMs, demonstrating its potential in emotional comprehension and human-machine interaction.



2. Demos

All audio was recorded using an Android phone.

Samples

Speech Question Text Speech Emotion Label Wav E-chat Response
今天一下花出去了两千多,这个月生活费都不够了(I've spent over 2,000 dollars today. I can't make ends meet this month.) fearful 我很理解你的感受,这确实是一笔不小的开支。不过不用担心,我们可以一起想办法解决这个问题,比如寻找一些省钱的方法,或者寻找一些兼职工作来增加收入。我相信我们一定能够克服这个困难,让生活变得更加美好。(I do understand how you feel, it is indeed a lot of money to spend. But don't worry, we can work together to figure out how to solve this problem, such as looking for some ways to save money or finding some part-time jobs to increase our income. I am sure we can overcome this difficulty and make life better.)
上海迪士尼周末人多吗?(Is Shanghai Disney crowded on weekends?) sad 我很理解你的感受,听到这个消息肯定会让你感到难过和失落。但是,请不要过于担心,上海迪士尼的客流量只是暂时的,相信很快就会恢复正常。同时,你也可以尝试一些其他的娱乐方式来放松心情,比如和朋友聚会、看一部电影等。请相信,一切都会好起来的。(I do understand how you feel, and hearing this news will definitely make you feel sad and lost. However, please don't worry too much, the traffic flow of Shanghai Disney is only temporary, and I believe it will be back to normal soon. In the meantime, you can also try some other forms of entertainment to relax, such as gathering with friends and watching a movie. Please believe that everything will be fine.)
我的电脑中病毒了(My computer has a virus.) fearful 我很理解你的担心,这种情况确实让人感到不安。但是请不要担心,你可以检查一下电脑的硬件和软件,看看是否存在病毒感染的迹象。如果你需要帮助,可以咨询专业的技术人员,他们会帮你解决这个问题。同时,也要注意保护好自己的电脑,不要随意下载和打开可疑文件。希望你能尽快解决这个问题,恢复电脑的正常使用。(I do understand your concern and this situation is really disturbing. But please don't worry, you can check your computer's hardware and software for signs of virus infection. If you need help, you can consult a professional technician who will help you solve the problem. Meanwhile, you should also pay attention to protect your computer and don't download and open suspicious files randomly. I hope you can solve this problem as soon as possible and resume the normal use of your computer.)

Paired Samples

Speech Question Text Speech Emotion Label1 Wav1 E-chat Response1 Speech Emotion Label2 Wav2 E-chat Response2
明年就要换新工作了(I'm getting a new job next year.) sad 我很理解你的感受,换工作确实会让人感到失落和不安。但是,这同时也是一个机会,让你可以尝试新的挑战和机遇。我相信你会找到一个更加适合你的工作,并为自己的职业生涯做出更好的选择。(I do understand how you feel, changing jobs can be really disorienting and upsetting. However, it is also an opportunity for you to try new challenges and opportunities. I am sure you will find a job that suits you better and make better choices for your career.) cheerful 太好了!换新⼯作意味着新的机遇和挑战,希望你能好好把握这 个机会,取得更好的成绩和成就。加油!(This is great! A new job means new opportunities and challenges, and I hope that you can make good use of this opportunity to achieve better results and accomplishments. I hope you will make good use of this opportunity to achieve better results and accomplishments!)
房子漏水了,应该联系谁?(Who should I contact if my house is leaking?) sad 我很理解您的感受,这种情况让人感到非常难过和无助。但是,如果您需要帮助和建议,我可以为您提供一些建议。首先,您需要联系物业管理部门,让他们尽快修复漏水问题。同时,您也可以联系您的保险公司,看看是否可以获得相应的赔偿。如果您需要其他帮助,请告诉我,我会尽力为您提供支持和帮助。(I do understand how you feel, this situation is very upsetting and helpless. However, if you need help and advice, I can offer you some suggestions. First of all, you need to contact the property management so that they can fix the leak as soon as possible. In the meantime, you can also contact your insurance company to see if you can get appropriate compensation. If you need any other help, please let me know and I will try my best to provide you with support and assistance.) angry 我很理解您的担忧和焦虑,这种情况确实让人感到不安和焦虑。如果您需要帮助,我建议您尽快联系物业或保险公司,以尽快解决问题并保护您的财产。同时,请保持冷静,不要惊慌失措,我们可以一起解决这个问题。(I appreciate your concern and anxiety, this situation is truly upsetting and anxiety-provoking. If you need help, I suggest you contact the property or insurance company as soon as possible to resolve the issue and protect your property as quickly as possible. In the meantime, please remain calm and don't panic so we can work together to resolve this issue.)
2023年马上就要结束了(2023 is almost over.) sad 我很理解您的悲伤情绪。2023年是一个重要的时间节点,它代表着一个时代的结束和另一个时代的开始。但是,请不要失去信心,因为未来还有很多机会和挑战等待着我们。让我们一起努力,迎接新的挑战,创造更美好的未来。(I do understand your sadness. 2023 is an important point in time, it represents the end of one era and the beginning of another. However, please don't lose faith because there are still a lot of opportunities and challenges waiting for us in the future. Let us work together to meet new challenges and create a better future.) cheerful 太棒了!祝贺你度过一个美好的2023年!希望你在接下来的一年里继续保持快乐和幸福!(Awesome! Congratulations on a wonderful 2023! I hope you continue to be happy and blessed in the upcoming year!)
今天晚上又要开会了(Another meeting tonight.) sad 很抱歉听到这个消息,听起来你很不开⼼。不过,也许你 可以试着放松⼀下⼼情,享受⼀下下班后的休闲时光。或者,你也可以提前做好准备,确保会议顺利进⾏。⽆论如何,希望你能度过⼀个愉快的夜晚。(I'm sorry to hear that. It sounds like you are very upset. However, perhaps you could try to relax and enjoy some down time after work. Or, you could prepare in advance to make sure the meeting goes smoothly. In any case, I hope you have a pleasant evening.) cheerful 太好了,听起来你很期待今晚的会议呢!希望会议能够顺利进⾏,让你收获满满。祝你今晚愉快!(That's great. It sounds like you're looking forward to tonight's meeting! I hope the meeting goes well and you get a lot out of it. Have a good night!)