How Do Chatbots Work?

How Do Chatbots Work?
Article by Meiryum Ali | July 08, 2019

A chatbot is any program that mimics real conversations. This can either be embedded in a site or through a third party messaging platform like Facebook Messenger or Slack. Chatbots rely on natural language processing, the same technology behind virtual assistants such as Google Now or Apple’s Siri.

Chatbots follow three simple steps: understand, act, respond. In the first case, the chatbot processes what the user sends, then acts according to a series of algorithms that interprets what the user said, and then finally picks from a series of appropriate responses. You can actually program your chatbot to respond the same way each time or to respond differently to messages containing certain keywords.

 

Why are chatbots important for customer experience?

Chatbots allow businesses to automate customer service without employing customer service representatives or salespersons. 60% of millennials have used chatbots when shopping, and 70% of them reported positive experiences. Many businesses make chatbots available for their customers when they have a question or run into a problem, but some businesses are leveraging chatbot technology even more. Estee Lauder uses a chatbot embedded in Facebook messenger that uses facial recognition to pick out the right shade of foundation for its customers, and Airbnb has used Amazon Alexa to welcome guests and introduce them to local attractions and restaurants.

Chatbots are useful for conversational purposes, when an app can’t perform an action because multiple variable inputs are needed to solve the problem. In other cases, a chatbot will offer simplicity by providing  the most immediate and direct solution to a person’s question.

quartz-messenger-bot
With its new chatbot, Quartz hopes to deliver the news and other content to users on Facebook’s Messenger. Image by AdWeek, Illustration by Yuliya Kim; Courtesy of Quartz.

What is the current state of chatbot technology?

If you asked somebody just two years ago about chatbots, the response would have been optimistic. Apps’ retention rates were slowing down, while messaging apps were on the rise. In 2016, it was not uncommon to see articles with headlines like “This is how Chatbots will kill 99% of Apps.”

And there was data to back up the hype. Facebook messenger crossed the 1 billion users mark in 2016, which is also the year it introduced its Bots API, which now boasts 100,000+ bots. Western app developers were also enamored by apps like China’s WeChat, a much more immersive messaging platform whose bots seemed capable of myriad tasks, like calling a taxi or making appointments.

However, being excited about chatbots and actually implementing a working chatbot are two very different things. A chatbot needs to be easy to use, efficient, and get the job done for users to consistently engage with it. Developers and investors constantly run into snags when launching a chatbot. To get at why, one has to revisit what the point of a chatbot is.

 

What are the difficulties in building a chatbot?

Any system or application that relies upon a machine’s ability to parse human speech is likely to struggle with the complexities inherent in elements of speech such as metaphors and similes. Because chatbots so heavily rely on natural language processing, they are also constrained by the limitations of natural language processing. Even the best natural language processing system does not often compare to a human. That would be fine in other use cases, but not for a chatbot. A user will only tolerate a chatbot if it speeds up communication, not hinders it. But at the current stage of natural language processing, it’s very possible that a chatbot understand words, but not necessarily their meaning. There needs to be major improvements in natural language processing for users to regularly use chatbots.

The other issue is human speech itself. Chatbots try to mimic conversations, but most conversations aren’t linear: discussions restart, there are tangential topics, or multiple topics being discussed at once.  This is very tough to follow algorithmically.

 

How can I build an effective chatbot?

One of the best ways to improve a chatbot is to constantly train it and get it to respond to different human interactions. The more data you feed a chatbot, the better it can adapt to human speech and all its idiosyncrasies, and the better you get at achieving a natural,  human-like conversation level. Successful human-like chatbots rely on context understanding, intent variation, and intent recognition.

 

Context Understanding for Chatbot Training

Context understanding  is the ability to remember and track different aspects of a conversation — location, time, preferences of people — and combine all of the inputs to ‘paint a picture’ of the conversation. Just like humans use surrounding context to inform their interactions, chatbots also need information that keeps the conversation going.

 

Intent Variation for Chatbot Training
To understand intent variation, it’s important to first understand what intents are in natural language processing. An “intent” refers to the purpose of a user’s input into a model, such as a chatbot or search engine. For example, if a restaurant has a chatbot on their website, then a customer might use it to ask about business hours.

This is where intent variation comes in, because people have different ways of expressing the same intent. The customer might phrase their question as “What are your business hours,” “What are your hours of operation,” or “What time does your store open and close.” The chatbot should be able to understand that there are different ways of expressing the same intent, and inform the customer of the store’s business hours regardless of which way the question is phrased.

 

Intent Recognition for Chatbot Training

Intent recognition is the ability to extract relevant information from each sentence, word and verb, and understand the intention and the meaning behind it. This allows the use of long complex sentences by users, because the chatbot is able to understand and extract multiple intents.

These complex features require a robust natural language processing foundation, that can only be improved and sustained by high quality training data. That’s where data providers like Lionbridge AI come in. At Lionbridge AI, we provide high-quality chatbot training data, including potential questions, answers and conversation templates. Since text structure is key to understanding and answering requests properly, we can provide a large number of example sentences to show the chatbot which terms to focus on, and what the user wants to achieve.  We can also provide flexibility and customization with industry specific terminology and terms that can be used for each chatbot.

Lionbridge’s entity annotation and linguistic annotation are particularly useful for chatbot training.

 

Entity Annotation for Chatbot Training

Entity annotation refers to extracting ‘units of information’ from sentences or unstructured data – and making it structured. These units can include names, such as people, organizations, and location names, proper nouns. It can also be used to identify numeric expressions such as time, date, money, and percent expressions. This is helpful in training a chatbot to develop ‘context’ in a conversation.

 

Linguistic Annotation for Chatbot Training

Linguistic Annotation refers to assessing the subject of any given sentence. It’s a broad genre, but essentially it’s anything to do with analysis of text, whether that be sentiment analysis of the user, or using natural language processing to answer questions. Through sentiment analysis, a chatbot can understand the ‘tone’ of the user, and see if they’re comfortable, friendly, or upset (and adapt accordingly). With over 500,000 vetted language experts across 300+ languages, Lionbridge AI can provide the most accurate sentiment annotation in the market.

Ultimately, the goal of any chatbot is to maintain natural conversation fluidity, and provide consistent engagement. The more data you give to a chatbot, the smoother its natural language processing engine runs, and the better the AI experience is.

Looking for custom chatbot training data?