Using Natural Language Processing to improve our service…

Our project

We’d noticed that customers and restaurants sometimes text back to automated text messages we send to them. These texts were going unanswered, so we wanted to change that.
We get a few hundred of these messages a month, so it could have been possible to have a human text back. However, that wouldn’t really have been a scalable approach. Using Natural Language Processing stood out as a solution as firstly, it’s something we could apply in other areas of the business and, secondly, it’s really interesting.
We knew Stanford University were a front runner in the field of Natural Language Processing. They’ve got some really excellent free courses on the topic, as well as software libraries that we could use for the ‘brain’ of our application. From here we got started designing our application.

Natural Language Processing in a nutshell

Natural Language Processing refers to human-machine interaction and the ability of software to process and understand human language. The Stanford course mentioned above is highly recommended for those wanting a bit more of an in depth understanding. Here’s how it works…

Word Selection

Regular expressions are used to pick out dates, phone numbers and other patterned expressions.
Tokens, which roughly translate to words, are extracted. In some instances, such as for the word pairing ‘New York’ a token might be a collection of words.

Reducing complexity

The dialogue is normalised, removing capitalisation and some apostrophes in order to reduce the complexity and deviation in the language input.
Stemming might be applied. This is where we effectively chop off the end of a word to further normalise them. Something like ‘ponies’ would become ‘poni’, as would ‘pony’. The process of stemming is a fairly crude approach and so some language processing tools opt not to use it.
Lemmatisation is used, which is the process of reduction verb inflections into a base term. For example, ‘am’, ‘are’, and ‘is’ become ‘be’.

Dealing with ambiguity

Where there’s a level of ambiguity, caused by a typo or a synonym, we can work out the probability of each word appearing between the two words either side of it using probability algorithms.

Parsing sentences

Sentences can be picked out using punctuation, as well as the probability of various words ending a sentence or new line spacing.
A parse tree is used to break out the specific elements of a sentence such a nouns, verbs, adjectives and noun and verb phrases. From this we can understand the purpose of specific words in a sentence. For example, if someone says ‘Find me restaurants in London’ the language processor can parse the sentence, pick ‘London’ and ‘restaurants’ as nouns. It can understand the direction ‘find me’ and attribute that to a specific action. It might then pop up a maps application and locate restaurants for the user.
Analysis of sentiment (the attitude of a speaker towards their topic), colloquialism, and ambiguous sentences such as ‘I’ve unplugged my mouse’ (computer or rodent?) can be more complicated for an application to interpret. Often, it must rely on context and the probability of the meaning of a given word.

How we build our application

We wrote a service to listen out for inbound text messages and a service to parse the messages and learn new words. We used the Stanford Natural Language Processor plugin within our application to parse the content of texts and drop them into a category based on the message content. We also built a micro-api to get details about the context of a texter, such as their name, their order, if they were a customer or a restaurant, and previous messages.
We trained our application using a selection of 100 texts. We wrote some unit tests to assert that it correctly categorised each of the texts and had it run through the process until it got all of the right. It improved each time because the brain of the application was able to recognise words what were used in place of other words.
We deployed the application into our production environment to build up a vocabulary by listening to live messages and telling us what it would say in response. We checked it now and then and told it when it had learnt words correctly and when it was wrong.

What our robot friend learned

  • It got pretty good at responding to messages about refunds, or questions about why the order could have been cancelled (because a lot of people text this kind of message) and so it had a large training set.
  • It struggled with some of the more left-field messages. For example, because it was used to recognising the word ‘free’ in relation to requests for free meals, when a texter replied by asking us to message them via a free instant messenger app, it thought they wanted free food.
  • It learnt that ‘shut’ and ‘offline’ meant ‘closed’ because it saw them used by restaurants in the same kinds of sentences.
  • It was 25% sure that ‘take’ was like ‘refund’ because it saw those two words used in conjunction frequently.
  • It thought that ‘oh’ meant that someone was satisfied with their level of service, because when it text people to say their order was on its way people often text back ‘oh’. They also text back saying just ‘oh’ when their order was cancelled. In the second example, it missed some of the sentiment of that kind of response..
  • It also learnt a lot of bad language. Ahem.


What we learned

Our training set for the application was quite small because we didn’t receive large amounts of messages, so the application had limited scope to learn. Given more time though we hope that it will build up a better idea of the language used in messages and what that language means.
The application also became a bit of a pessimist. Currently we mostly text customers if there’s an issue with their order and often, when they text back, it’s to tell us they’re not too happy. The application started to associate words like ‘I’m’ with words like ‘unhappy’ and flagged them both as words that meant someone was upset. We could change the app’s configuration so that it doesn’t bother to learn words that are less than three letters. However, this point does highlight how context dependent the language it has learnt is. It would perhaps need to start from scratch if we were to respond to texts in a different context.
It’s been a very interesting project and we’ve learned a lot on the way. Hopefully, with a bit more work, we’ll be able to have the app ready to respond to customer and restaurant texts in the near future. Even if it’s just those texts asking about refunds.