See Our Progress.It took months of trial and error to reach the progress we have made today. Here's a behind the scenes look of our 2018 model.
|
2018 Amazon Alexa Prize
Team Gunrock of UC Davis Wins First Place! Team Gunrock of UC Davis has officially won first place in the 2018 Amazon Alexa Prize competition!
We are so humbled by the opportunity to improve the quality of conversational AI, and would like to thank all of you for your support throughout this journey. Our socialbot earned an average score of 3.1 out of 5 points, managing to carry a conversation for 9 minutes and 59 seconds. We started this competition competing against eight teams from universities around the world. During the initial phases of this experiment, our group took its first steps to create the bot’s dialogue and conversational flow. We faced challenges of getting the socialbot to stick to the topic of the conversation, and tried to get Gunrock to understand user requests. This was our first time competing in the competition, and we lacked the experience that other groups had. According to team lead Arbit Chen, there was a lot of worry amongst team members about making it through the finals because “we had less data to work with compared to the other returning teams. We needed to catch up with a whole year’s worth of work from other returning groups while still planning ways to stay competitive.” Our hard work had finally paid off. Team Gunrock now proudly represents UC Davis as first place finalists. Since winning the wildcard pick in the finals round, our team worked diligently to facilitate an informative, natural flowing conversation between users and our socialbot. It was with great optimism and efficient collaboration that we managed to come together and successfully secure our 3.1 rating. The most rewarding aspect of this journey was watching everyone’s growth. “Most of the students on the team lacked practical engineering and conversational AI experience,” mentioned Arbit. “However, after these months, everyone became so experienced and ready to tackle any kind of problem to build the socialbot. We feel like fathers and mothers to our bot. Watching our child’s growth and seeing it become better with time was the best part. When we deployed to production in the beginning, it was really awful. Lots of people were complaining, but now when we see the user feedback, there are more and more people that enjoy talking to our child, and even want to be its friend."
|
|
Our team recently had a demo paper accepted to EMNLP-IJCNLP 2019.
See the above video created by Michelle Cohn (collaborator, UC Davis Phonetics Laboratory) for a short system demonstration of the 2018 bot. |
Finally Finalists
We are excited to announce that Team Gunrock of UC Davis has made it to the final round of the competition. We are truly humbled to have made it as the 2018 Alexa Prize wildcard pick. Throughout the competition, teams were rated by everyday users on a scale of 1 to 5 (5 being the highest) based on the quality of the conversation. The two teams with the best ratings were then determined as finalists while the third finalist, the wildcard pick, was decided by Amazon. Based on ratings from Alexa users and Amazon judges, the quality of the conversational topics, and the content of the technical paper, our team qualified to progress to the final round.
It took a lot of trial and error to become an elite finalist. One of the biggest challenges we faced was working with the language ambiguity in the conversation. A lot of the time, users intended to talk about a specific topic, but our bot would recognize it as something else. An example of this would be talking about penguins. Gunrock could comprehend this as discussing the animal, or the Pittsburgh penguin hockey team. Without much context, our bot is quick to change the topic to something the user may not have previously mentioned.
Our team now has until November to keep improving our bot. In addition to making the conversation quality flow more organically, we hope to work on giving Gunrock more human like sounds. We ultimately want to elicit a more natural response like a realistic sounding laugh, or a more excited tone of voice. Our team has also recognized that we must incorporate personal opinions on a variety of topics as well to help maintain a free-flowing conversation structure. We hope to implement our own opinions on certain topics and have Gunrock discuss them with the user.
It took a lot of trial and error to become an elite finalist. One of the biggest challenges we faced was working with the language ambiguity in the conversation. A lot of the time, users intended to talk about a specific topic, but our bot would recognize it as something else. An example of this would be talking about penguins. Gunrock could comprehend this as discussing the animal, or the Pittsburgh penguin hockey team. Without much context, our bot is quick to change the topic to something the user may not have previously mentioned.
Our team now has until November to keep improving our bot. In addition to making the conversation quality flow more organically, we hope to work on giving Gunrock more human like sounds. We ultimately want to elicit a more natural response like a realistic sounding laugh, or a more excited tone of voice. Our team has also recognized that we must incorporate personal opinions on a variety of topics as well to help maintain a free-flowing conversation structure. We hope to implement our own opinions on certain topics and have Gunrock discuss them with the user.
Check it Out!
Our team attended a special reception to honor our efforts in the #AlexaPrize competition. Click the link to see how it went!
Maintaining the Flow
|
July 2018
|
As we near the end of the competition, we are making some final changes to our conversation templates. Last month, we spent a lot of time working on providing sufficient information in our responses to users testing our bot. In result, the conversational topics have expanded to include animals, science, and technology. Our main focus now is trying to make our conversations seem more natural. Even though Gunrock has a ton of information to share, we couldn’t help but notice how rigid and robotic the conversations can sound.
To help mimic the structure of a real conversation, our team has been working on improving our bot’s backstory. The purpose of this task is to extend the conversation beyond the one sentence answer users usually get after asking a question. We are working to increase the conversational turns of the conversation, and provide a series of follow up questions.
One of the coolest things Gunrock is capable of doing is giving out music and movie suggestions. For example, Gunrock could start the conversation by asking about your favorite artist or song. After taking in the information you provided, our bot can then make a few suggestions that complement the artist or song mentioned. This is a great opportunity to not only give users and our bot something to discuss, but it also allows those conversing with our bot to take something interesting to them from the conversation.
To help mimic the structure of a real conversation, our team has been working on improving our bot’s backstory. The purpose of this task is to extend the conversation beyond the one sentence answer users usually get after asking a question. We are working to increase the conversational turns of the conversation, and provide a series of follow up questions.
One of the coolest things Gunrock is capable of doing is giving out music and movie suggestions. For example, Gunrock could start the conversation by asking about your favorite artist or song. After taking in the information you provided, our bot can then make a few suggestions that complement the artist or song mentioned. This is a great opportunity to not only give users and our bot something to discuss, but it also allows those conversing with our bot to take something interesting to them from the conversation.
Keeping Up With the Conversation |
June 2018 |
It’s important to keep in mind the amount of information that is exchanged between participants of a conversation. Current events and personal information are constantly discussed, making the act of conversation seem so effortless. However, a lot of work goes into engineering a natural conversation.
Our team is currently trying to incorporate information that allows Gunrock to build a lasting dialogue between itself and the user. One of the bigger challenges we’ve faced is trying to keep up with the most recent news. Each day, there is always something new to talk about. We try to add in as much information as we can in Gunrock’s conversational templates so our bot can provide our users with thorough responses. Often times, users testing the bot will ask Gunrock about things it’s unfamiliar with. This interrupts the flow of the conversation, and our bot will often try to change topics to something it can better understand.
We currently updated some information on March Madness, and plan on including data from the World Cup. It’s not easy give our users the most realistic conversation in the testing phase, but our engineers are hard at work trying to improve each conversation session.
Our team is currently trying to incorporate information that allows Gunrock to build a lasting dialogue between itself and the user. One of the bigger challenges we’ve faced is trying to keep up with the most recent news. Each day, there is always something new to talk about. We try to add in as much information as we can in Gunrock’s conversational templates so our bot can provide our users with thorough responses. Often times, users testing the bot will ask Gunrock about things it’s unfamiliar with. This interrupts the flow of the conversation, and our bot will often try to change topics to something it can better understand.
We currently updated some information on March Madness, and plan on including data from the World Cup. It’s not easy give our users the most realistic conversation in the testing phase, but our engineers are hard at work trying to improve each conversation session.
"Lets Chat" |
May - June 2018 |
The year is coming to an end, and we finally made it to the Initial Feedback Period. Gunrock has met the Availability Criteria, which ensures content is filtered, and basic standards are met. The next step is to receive feedback from customers who have an Amazon Echo or Alexa. By simply saying “Alexa, Let’s Chat” to an Amazon system, everyday users are connected to one of the eight competing bots.
The process is rather simple. The person testing the bot can start a conversation from various topics including news, books, movies, and sports. The user can ask the bot questions, make suggestions, and learn about some of the bot's personal preferences. The conversation can last for however long the user desires. To end the testing period, one can just tell the socialbot to “stop,” and they will then rate the quality of the conversation.
Team Gunrock is currently looking for people to test our specific model. If you are interested in helping improve our social bot, we encourage you to reach out so that we can connect you to our bot, Gunrock. Best of all, this task can be done from the convenience of your home!
The process is rather simple. The person testing the bot can start a conversation from various topics including news, books, movies, and sports. The user can ask the bot questions, make suggestions, and learn about some of the bot's personal preferences. The conversation can last for however long the user desires. To end the testing period, one can just tell the socialbot to “stop,” and they will then rate the quality of the conversation.
Team Gunrock is currently looking for people to test our specific model. If you are interested in helping improve our social bot, we encourage you to reach out so that we can connect you to our bot, Gunrock. Best of all, this task can be done from the convenience of your home!
Whats New? |
May 2018 |
Just recently our team joined the seven other competing teams at Amazon’s headquarters in Seattle for the Alexa Prize Summit. During the summit, we were able to show off our progress and receive live feedback that helped improve the development of our social bot.
In addition to meeting with other competing teams and Amazon mentors, we shared the work we have done to a group of high school and university students. This was an exciting moment for us because we were able to teach those students who are unfamiliar with the developments of conversational AI more about what it entails. We hoped that this would also show the audience the "fun" side of the realm of machine learning and its relevance in our daily lives.
Our team thoroughly indulged in the wealth of information that was presented to us. In addition to all the feedback and analysis, our team attended various keynote sessions by our hosts at Amazon. This event was by far the most memorable experience of our time working on the Alexa Prize project. We hope to be back another year in yet another project to improve conversational AI.
In addition to meeting with other competing teams and Amazon mentors, we shared the work we have done to a group of high school and university students. This was an exciting moment for us because we were able to teach those students who are unfamiliar with the developments of conversational AI more about what it entails. We hoped that this would also show the audience the "fun" side of the realm of machine learning and its relevance in our daily lives.
Our team thoroughly indulged in the wealth of information that was presented to us. In addition to all the feedback and analysis, our team attended various keynote sessions by our hosts at Amazon. This event was by far the most memorable experience of our time working on the Alexa Prize project. We hope to be back another year in yet another project to improve conversational AI.
One of the bigger challenges we are currently faced with is trying to address lexical ambiguity in our conversations. Lexical ambiguity is described as words with various meanings. A word like “bank” could be interpreted in different contexts that could relate to word banks, food banks, or financial banks. By covering more lexical patterns, our social bot will have an easier time tracking the context of the conversation. In turn, our system will have more reference to the conversation, which would make it easier to extend its flow.
|
Exchanging Ideas... |
April 2018 |
Conversation is created by an exchange of ideas between two or more people. Each person takes a turn to speak, and each occurrence of speech relates to the same topic as the previous one. These turns then contribute to an overall discussion of a specific topic. There are numerous factors that need to be considered when we make a conversational socialbot. We are currently working on improving our bot’s ability to coreference, ultimately tracking the context of our conversations.
Coreferencing occurs when there are multiple references to the same subject being discussed. We want our bot to be able to talk about the same topic throughout multiple turns in the conversation. For example, say we want Gunrock to carry a conversation about the Avengers movie. When leading the discussion, it can start by saying “Let’s talk about the Avengers movie. Have you seen it yet?” We are trying to make sure Gunrock is able to continue the conversation using the information it has gathered from previous turns in the conversation.
Tracking the conversation helps ensure a smooth flow of communication. Coreferencing informs Gunrock of the topic of the dialogue, allowing it to extend its speaking period. With the current Alexa system, it is unable to track the turns being made in the conversation. Alexa can't recall that you are talking about the Avengers movie, and the conversation is cut short. Gunrock, however, recognizes the turns being made in the conversation, and can keep the conversation going.
Coreferencing occurs when there are multiple references to the same subject being discussed. We want our bot to be able to talk about the same topic throughout multiple turns in the conversation. For example, say we want Gunrock to carry a conversation about the Avengers movie. When leading the discussion, it can start by saying “Let’s talk about the Avengers movie. Have you seen it yet?” We are trying to make sure Gunrock is able to continue the conversation using the information it has gathered from previous turns in the conversation.
Tracking the conversation helps ensure a smooth flow of communication. Coreferencing informs Gunrock of the topic of the dialogue, allowing it to extend its speaking period. With the current Alexa system, it is unable to track the turns being made in the conversation. Alexa can't recall that you are talking about the Avengers movie, and the conversation is cut short. Gunrock, however, recognizes the turns being made in the conversation, and can keep the conversation going.
Preparing for the Semifinals |
March - April 2018 |
Team Gunrock has been diligently working on developing our socialbot since being approved. In the Initial Development Period, we had to incorporate basic concepts and programs to help establish the framework of our bot. Once the basics were established, we were then able to work on Gunrock's conversational skills. The most challenging task was making sure our bot was able to appropriately respond to each request in the conversation. We want the bot to stick to a specific topic in the conversation when the user speaks to it. Our goal is for Gunrock to engage with the users, and refer them to appropriate resources when needed. When sufficient progress has been made, we will then submit our socialbot to an Amazon Sponsor for certification.
We are now prepared to enter into the Internal Amazon Beta Period. This phase in the competition allows Amazon employees to test our bot and provide valuable feedback. Once their requirements are met, we will be one step closer to getting the green light to work with customers that have an Amazon Alexa. These customers will be able to have their own conversation with Gunrock. Our team will be able to monitor their conversation and review the feedback given so we can make further improvements.
We are now prepared to enter into the Internal Amazon Beta Period. This phase in the competition allows Amazon employees to test our bot and provide valuable feedback. Once their requirements are met, we will be one step closer to getting the green light to work with customers that have an Amazon Alexa. These customers will be able to have their own conversation with Gunrock. Our team will be able to monitor their conversation and review the feedback given so we can make further improvements.
Chalking Up Ideas |
February 2018 |
We have just been approved to compete in the Alexa Prize competition with seven other teams. After receiving a $250,000 grant from Amazon to help develop our bot, we planned to start implementing a backstory and adding to our user profile. The first order of business, however, was to come up with a name for our team. We settled on the name "Gunrock," after our our campus mascot. We chose the name as a way to represent the UC Davis community amongst the other competing teams. Gunrock seemed to be unique, yet meaningful.
A fun idea we hope to incorporate is the ability to joke with Gunrock. A lot of us come from different ethnic backgrounds, and we believe a good way to integrate into the modern day culture is through humor. We want our bot to be able to not only respond to the jokes in a more personable way, but also initiate more of the conversation by coming up with original jokes . Ideally, we hope to include up to date information that Gunrock could mention. We want to move past the generic conversations and make our topics more relevant. By taking those steps, we can encourage more conversations between the user and the improved system.
A fun idea we hope to incorporate is the ability to joke with Gunrock. A lot of us come from different ethnic backgrounds, and we believe a good way to integrate into the modern day culture is through humor. We want our bot to be able to not only respond to the jokes in a more personable way, but also initiate more of the conversation by coming up with original jokes . Ideally, we hope to include up to date information that Gunrock could mention. We want to move past the generic conversations and make our topics more relevant. By taking those steps, we can encourage more conversations between the user and the improved system.
The Competition Begins... |
February 2018 |
While we have made groundbreaking progress in the world of artificial intelligence, there is still so much potential for improvement. Socialbots like Alexa have been programmed to respond to a question or command, but the flow of conversation is quickly cut short. Our goal is to program our bot to track the context of the dialogue, and allow more turns in the conversation to be made. As of now, the social bots don’t have enough reference to the conversation. We could ask Alexa to tell us something interesting, and she would simply respond with a suggestion like, “do you want to talk about the weather?” After a simple interaction is made, the conversation stops. One could try to continue answering the question, but the dialogue has already ended. Alexa doesn’t know why we said “yes, or no.” We are currently working on extending the social interaction by finding similarities to incoming requests.