AI is a sought-after topic, but most developers face two hurdles that prevent them from programming anything with it.<\/p>\n
Especially in the up-and-coming natural language processing (nlp) sector, there is a lack of data in many areas. With this blogpost we are going to discuss a simple yet powerful solution to address this problem in the context of a conversational AI. ?<\/p>\n
Leon presented a simple solution on our blog a few weeks ago: With AI as a Service<\/a> reliable language processing systems can be developed in a short time whithout having to hassle around with datasets and neural networks. However, there is one significant drawback due to this type of technology: Dependence on the operator of the service. On one hand the service can be linked with costs, furthermore the own possibly sensitive data has to be passed on to the service operator. Especially for companies this is usually a show stopper. That’s where Rasa enters the stage.<\/p>\n Rasa<\/a> is an open source (see Github<\/a>) conversational AI that is fully free for everyone and can be used in-house. There is no dependence on a service from Rasa or any other company. It consists of a two-part stack whose individual parts seem to perform similar tasks at first glance, but on a closer look you see that both try to solve their own problems. Rasa NLU<\/a><\/strong> is the language understanding AI we are going to dig deeper into soon. It is used to understand what the user is trying to say and which additional information he provides. Rasa Core<\/a><\/strong> is the context-aware AI for conversational flow, which is used to build dialog systems e.g. chatbots like this<\/a>. It uses the information from Rasa NLU to find out what the user wants and what other information is needed to achieve it. For example, for a weather report you need both the date and the place.<\/p>\n The following paragraphs deal with the development of language understanding. Its basics are already extensively documented, which is why I will keep this brief. Instead, the optimization possibilities are to be presented more extensively. If you have never coded something using Rasa, it makes sense to work through the restaurant example<\/a> (see also Github code template<\/a>) to get a basic understanding of the framework.<\/p>\n The processing pipeline is the core element of Rasa NLU. The decisions you make there have a huge influence on the system’s quality. In the restaurant example the pipeline is already given: Two NLU frameworks spaCy and skLearn are used for text processing. Good results can be achieved with very few domain-specific training data (10 – 20 formulations per intent). You can get this amount of data easily using Rasa Trainer<\/a>. It is so small because transfer learning combines your own training data with spaCy’s own high-quality models to create a neural net. Besides spaCy, there are other ways to process your data, which we will discover now!<\/p>\n Instead of spaCy you can also use MIT Information Extraction<\/a>. MITIE can also be used for intent recognition and named entity recognition (NER). Both backends perform the same tasks and are therefore interchangeable. The difference lies in the algorithms and models they use. Therefore you are not bound to only spaCy or mitie, but you can also use scikit-learn<\/a> for intent classification.<\/p>\n Which backend works best for your project is individual and should be tested. As you will see in the next paragraph, the pipeline offers some precious showpieces that work particularly well. The already included cross validation<\/a> should be used to evaluate the quality of the system.<\/p>\n You should understand how the pipeline<\/a> works to develop a good system for your special problem.<\/p>\n In the following example (from Rasa) you can see how the single parts work together to provide information about intent and entity:<\/p>\n As mentioned by Rasa itself<\/a> intent_classifier_tensorflow_embedding<\/em> can be used for intent classification. It is based on the StarSpace: Embed All The Things!<\/a> paper published by Facebook Research. They present a completely new way for meaning similarity, which generates awesome results! ?<\/p>\n For named entity recognition you have to make a decision: Either you use common pre-trained entities, or you use custom entities like “type_of_coffee”. Pre-trained entities can be one of the following:<\/p>\n Those two algorithms perform very well in recognition of the given types, but if you need custom entities they perform rather bad. Instead you should use ner_mitie<\/strong> or ner_crf<\/strong> and collect some more training data than usual. If your entities have a specific structure, which is parsable by a regex make sure to integrate intent_entity_featurizer_regex<\/em> to your pipeline! In this Github Gist<\/a> I provided a short script, which helps you to create training samples for a custom entity. You can just pass some sentences for an intent into it and combine it with sample values of your custom entity. It will then create some training samples for each of your sample values.<\/p>\n That’s it \ud83d\ude42 If you have any questions about Rasa or this blogpost don’t hesitate to contact me! Have a nice week and stay tuned for our next post.<\/p>\nThe Rasa Stack<\/h2>\n
Digging deeper into Rasa NLU<\/h2>\n
Unlock the full potential<\/h2>\n
The processing pipeline<\/h3>\n
\n
{\r\n \"text\": \"I am looking for Chinese food\",\r\n \"entities\": [\r\n {\"start\": 8, \"end\": 15, \"value\": \"chinese\", \"entity\": \"cuisine\", \"extractor\": \"ner_crf\", \"confidence\": 0.864}\r\n ],\r\n \"intent\": {\"confidence\": 0.6485910906220309, \"name\": \"restaurant_search\"},\r\n \"intent_ranking\": [\r\n {\"confidence\": 0.6485910906220309, \"name\": \"restaurant_search\"},\r\n {\"confidence\": 0.1416153159565678, \"name\": \"affirm\"}\r\n ]\r\n}\r\n<\/code><\/pre>\n\n