What is Natural Language Processing?

Article by Meiryum Ali | June 09, 2019

Let’s say you want to set an alarm on your iPhone. You command Siri, and tell it to set an alarm for tomorrow. Siri responds ‘For what time?’, and you specify 9am. The alarm is set.

In this short interaction, you activated a device, which heard your speech, processed said speech, executed an action, and responded with a sentence. This entire exchange was made possible by natural language processing (NLP). Natural language processing is the basis behind any machine or program’s ability to process human speech. It’s the technology behind recognizable voice assistants like Siri or Alexa, and chatbots in messaging apps.

 

What is natural language processing?

Natural language processing is the umbrella term for any machine’s ability to recognize what is said to it, understand its meaning, determine the appropriate action, and respond in language the user will understand.

 

Natural Language Processing Terms to Know

Natural language understanding (NLU) is a subset of natural language processing. Natural language understanding goes beyond just basic sentence structure, but attempts to understand the intended meaning of language. Human speech is peppered with nuances, subtleties, mispronunciations, and colloquialisms. Natural language understanding is designed to tackle the complexities of human speech. One of the main areas of research in language processing is to transition from natural language processing to natural language understanding. Natural language understanding deals with the much narrower facet of how to best handle unstructured inputs and convert them into a structured form that a machine can understand and act upon.

Finally, natural language generation (NLG) is what a machine writes itself. In the example above, Siri’s response ‘For what time?’ would fall under natural language generation.

 

How does natural language processing work?

Let’s use the above example of asking Siri to set an alarm for you. At the very basic level, these were the natural language processing steps.

  1. You ask Siri to set an alarm.
  2. Siri converts your audio speech to text.
  3. Siri converts this plain text request into commands for itself, using natural language processing to turn text into structured data.
  4. Siri processes this data in a decision engine.
  5. Siri responds to you by asking “what time,” using natural language generation to turn structured data into text.
  6. You specify ‘9am’ which is then again processed through natural language processing into the decision engine.
  7. Siri sets the alarm for you.

 

Data Annotation for Natural Language Processing

How are natural language processing systems built? The following are a few ways to break down and organize data so that you can train your program to improve its natural language processing.

 

Entity Annotation

Entity annotation refers to extracting units of information from sentences or unstructured data, and making it structured. These units can include names, such as people, organizations, and location names, proper nouns. It can also be used to identify numeric expressions such as time, date, money, and percent expressions.

 

Semantic Annotation

Semantic annotation helps assess search results. Essentially, companies are looking for ways to improve their search relevance so that customers can actually find their products in search engines. The problem is, most product descriptions vary greatly depending on the source, and are often not accurate. Semantic annotation helps improve search results by tagging different product titles and search queries. At Lionbridge AI, we can build datasets to help you predict which categories fit best to a given product to make ecommerce process and product classification easier, faster and less error-prone.

 

Linguistic Annotation

Linguistic annotation refers to assessing the subject of any given sentence. Its a broad genre, but essentially it’s anything to do with analysis of text, whether that be sentiment analysis of social media data, or using natural language processing to answer routine questions. Linguists and translators at Lionbridge AI provide a wide variety of services, including part-of- speech tagging, and audio speech analysis. Our team includes 500,000 qualified contributors across 300 languages.

 

What is natural language processing used for?

Natural language processing can be used in a variety of cases, such as the following.

  1. Voice assistants: as described above, voice assistants like Siri and Alexa are powered by natural language processing.
  2. Chatbots: since chatbots mimic real conversations, they heavily rely on natural language processing.
  3. Customer service: many companies transcribe and analyze customer call recordings. Natural language processing helps in analyzing such data, and responding to customer needs faster.
  4. Sentiment analysis: Natural language processing is used in figuring out the tone of any given piece of writing. This is usually very useful for companies looking to understand how their product is received on social media.
  5. Healthcare: Natural language processing has huge implications for healthcare. This includes healthcare assistants that are similar to Siri but specially trained on medical terminology, and image classification to understand a medical scan and provide diagnosis and treatment options.
The Author
Meiryum Ali

Freelance writer working at Lionbridge; AI enthusiast

Welcome!

Sign up to our newsletter for fresh developments from the world of training data. Lionbridge brings you interviews with industry experts, dataset collections and more.