1. It is a complex field in which a lot of experience is needed to achieve good results
2. Although there are good network topologies and models for a problem, there is often a lack of training data (corpora) without which most neural networks cannot achieve good results

Especially in the up-and-coming natural language processing (nlp) sector, there is a lack of data in many areas. With this blogpost we are going to discuss a simple yet powerful solution to address this problem in the context of a conversational AI. ?

Leon presented a simple solution on our blog a few weeks ago: With AI as a Service reliable language processing systems can be developed in a short time whithout having to hassle around with datasets and neural networks. However, there is one significant drawback due to this type of technology: Dependence on the operator of the service. On one hand the service can be linked with costs, furthermore the own possibly sensitive data has to be passed on to the service operator. Especially for companies this is usually a show stopper. That’s where Rasa enters the stage.

The Rasa Stack

Rasa is an open source (see Github) conversational AI that is fully free for everyone and can be used in-house. There is no dependence on a service from Rasa or any other company. It consists of a two-part stack whose individual parts seem to perform similar tasks at first glance, but on a closer look you see that both try to solve their own problems. Rasa NLU is the language understanding AI we are going to dig deeper into soon. It is used to understand what the user is trying to say and which additional information he provides. Rasa Core is the context-aware AI for conversational flow, which is used to build dialog systems e.g. chatbots like this. It uses the information from Rasa NLU to find out what the user wants and what other information is needed to achieve it. For example, for a weather report you need both the date and the place.

Digging deeper into Rasa NLU

The following paragraphs deal with the development of language understanding. Its basics are already extensively documented, which is why I will keep this brief. Instead, the optimization possibilities are to be presented more extensively. If you have never coded something using Rasa, it makes sense to work through the restaurant example (see also Github code template) to get a basic understanding of the framework.

The processing pipeline is the core element of Rasa NLU. The decisions you make there have a huge influence on the system’s quality. In the restaurant example the pipeline is already given: Two NLU frameworks spaCy and skLearn are used for text processing. Good results can be achieved with very few domain-specific training data (10 – 20 formulations per intent). You can get this amount of data easily using Rasa Trainer. It is so small because transfer learning combines your own training data with spaCy’s own high-quality models to create a neural net. Besides spaCy, there are other ways to process your data, which we will discover now!

Unlock the full potential

Instead of spaCy you can also use MIT Information Extraction. MITIE can also be used for intent recognition and named entity recognition (NER). Both backends perform the same tasks and are therefore interchangeable. The difference lies in the algorithms and models they use. Therefore you are not bound to only spaCy or mitie, but you can also use scikit-learn for intent classification.

Which backend works best for your project is individual and should be tested. As you will see in the next paragraph, the pipeline offers some precious showpieces that work particularly well. The already included cross validation should be used to evaluate the quality of the system.

The processing pipeline

You should understand how the pipeline works to develop a good system for your special problem.

1. The tokenizer: is used to transform input words, sentences or paragraphs into single word tokens. Hence, unnecessary punctuation is removed and stop words can also be removed.
2. The featurizer is used to create input vectors from the tokens. They can be used as features for the neural net. The simplest form of an input vector is one-hot.
3. The intent classifier is a part of the neural net, which is responsible for decision making. It decides which intent is most likely meant by the user. This is called multiclass classification.
4. Finally named entity recognition can be used to extract information like e-mails from a text. In terms of Rasa (and dialogue systems) this is called entity extraction.

In the following example (from Rasa) you can see how the single parts work together to provide information about intent and entity:

{
    "text": "I am looking for Chinese food",
    "entities": [
        {"start": 8, "end": 15, "value": "chinese", "entity": "cuisine", "extractor": "ner_crf", "confidence": 0.864}
    ],
    "intent": {"confidence": 0.6485910906220309, "name": "restaurant_search"},
    "intent_ranking": [
        {"confidence": 0.6485910906220309, "name": "restaurant_search"},
        {"confidence": 0.1416153159565678, "name": "affirm"}
    ]
}

As mentioned by Rasa itself intent_classifier_tensorflow_embedding can be used for intent classification. It is based on the StarSpace: Embed All The Things! paper published by Facebook Research. They present a completely new way for meaning similarity, which generates awesome results! ?

For named entity recognition you have to make a decision: Either you use common pre-trained entities, or you use custom entities like “type_of_coffee”. Pre-trained entities can be one of the following:

• ner_spaCy: Places, Dates, People, Organisations
• ner_duckling: Dates, Amounts of Money, Durations, Distances, Ordinals

Those two algorithms perform very well in recognition of the given types, but if you need custom entities they perform rather bad. Instead you should use ner_mitie or ner_crf and collect some more training data than usual. If your entities have a specific structure, which is parsable by a regex make sure to integrate intent_entity_featurizer_regex to your pipeline! In this Github Gist I provided a short script, which helps you to create training samples for a custom entity. You can just pass some sentences for an intent into it and combine it with sample values of your custom entity. It will then create some training samples for each of your sample values.

That’s it 🙂 If you have any questions about Rasa or this blogpost don’t hesitate to contact me! Have a nice week and stay tuned for our next post.

Greets,
Domi

In the course of the article we are going to build a simple chatbot that reminds its users every two days to work out. The result will look something like this:

Even with this simple example, we can take a look at a bunch of great features provided by the Telegram API and more specific via AbilityBot. In a nutshell, those features are:

• Replying to commands like /start
• Using inline keyboards like in the picture above
• making use of the embedded database in ability bot
• and as an extra: Repeated execution of tasks at a specific time

If you don’t want to work yourself through the whole article, feel free to take a look at the finished code on github.

Okay then, let’s start coding!

Project setup

For the AbilityBot to work, you need to set up a Maven project in a Java 8 environment. In case you have never done that before, head over to Maven’s getting started and setup a new maven project. In your pom.xml add the following two dependencies:

<dependencies>
        <dependency>
                <groupId>org.telegram</groupId>
                <artifactId>telegrambots</artifactId>
                <version>3.6.1</version>
        </dependency>

        <dependency>
                <groupId>org.telegram</groupId>
                <artifactId>telegrambots-abilities</artifactId>
                <version>3.6.1</version>
        </dependency>
</dependencies>

Ensure that you are using at least Java 8 by adding the following build plugin to your pom:

<build>
        <plugins>
                <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-compiler-plugin</artifactId>
                        <configuration>
                                <source>8</source>
                                <target>8</target>
                        </configuration>
                </plugin>
        </plugins>
</build>

Getting your bot to talk

At first, we create our bot class called FitnessBot. It extends the class AbilityBot thus we need to call super() on the constructor and override the method creatorId(). Furthermore we added another constructor without any arguments, thus we can easily instantiate our bot.

public class FitnessBot extends AbilityBot {

    public FitnessBot() {
        this(Constants.BOT_TOKEN, Constants.BOT_USERNAME);
    }

    private FitnessBot(String botToken, String botUsername) {
        super(botToken, botUsername);
    }

    @Override
    public int creatorId() {
        return Constants.CREATOR_ID;
    }
}

At the latest when you paste this code into your IDE, you will recognize the unresolved reference to an interface called Constants. It is a common software pattern widely used in Android for example. The interface contains all the constant values needed by your application to function properly. Later we will use it for all our text responses, database identifiers, and similar stuff. In order to fix your code for now, create a new interface and paste the following lines.

public interface Constants {
    // Initialization
    String BOT_USERNAME = "FitnessBot";
    String BOT_TOKEN = "your-super-secret-token";
    int CREATOR_ID = your-telegram-user-id;
}

You need to replace the information with the ones @BotFather gave to you. If you don’t know what I’m talking about please refer to this tutorial. It contains information on how to create a new Telegram bot.

Running your bot

Before you can run your bot you have to initialize it. Therefore we create a new class called Application and fill it with a main() method containing the code to initialize your bot. This block is copied from the Github documentation of ability bot.

public class Application {

    public static void main(String[] args) {
        // Initializes dependencies necessary for the base bot
        ApiContextInitializer.init();

        // Create the TelegramBotsApi object to register your bots
        TelegramBotsApi botsApi = new TelegramBotsApi();

        try {
            // Register your newly created AbilityBot
            FitnessBot bot = new FitnessBot();
            botsApi.registerBot(bot);

        } catch (TelegramApiException e) {
            e.printStackTrace();
        }
    }
}

Now you can run your bot but he won’t answer, yet. In the next step, we change this by developing the first ability: A response to the command /start. Therefore, you have to add the following method to your FitnessBot class.

public Ability replyToStart() {
    return Ability
        .builder()
        .name("start")
        .info(Constants.START_DESCRIPTION)
        .locality(ALL)
        .privacy(PUBLIC)
        .action(ctx ->  silent.send("Hello World!", ctx.chatId()))
        .build();
}

In Constants add:

String START_DESCRIPTION = "Start using the fitness bot to remind you doing sports";

I don’t want to go into detail on the specific methods chained here because they are pretty well explained in the already linked documentation.

But here are some information explaining action() more thoroughly. ctx is the message context and provides related information like the chatId of the message. To reply to a message you have to call a method on either object silent or sender. Both are provided by the parent class AbilityBot. As you can see in our code, currently we are using the silent object. The difference is that while silent is used to send plain text messages only, sender gives you more freedom on how to compose your reply. As you will learn later, inline keyboards, for example, need to use sender. The downside of sender is that more freedom leads to more responsibility. Exceptions might occour and you have to handle them by yourself.

Database usage and state handling

State handling and properly saving that state to a database can get pretty complex. So it makes sense to create a separate class for that concern. Therefore we build a new class ResponseHandler which will be responsible for processing requests:

public class ResponseHandler {
    private final MessageSender sender;
    private final Map<Long, State> chatStates;

    public ResponseHandler(MessageSender sender, DBContext db) {
        this.sender = sender;
        chatStates = db.getMap(Constants.CHAT_STATES);
    }
}

In it’s constructor, the ResponseHandler receives the database context called db. The instance, which is going to be created using the constructor, will become a field in our FitnessBot class. Field sender will be used to send messages back to the user. As you can see, the state of the bot is saved separately for each chat in a Map called chatStates.

The next step is to add a new entry to our Constants interface:

String CHAT_STATES = "CHAT_STATES"

This is the name of the table created internally by the embedded database. Pretty simple. You can read and even update the map chatStates as you wish and it will be synced with the database automagically.

If you wondered about State-class: This is an enum containing our set of states for the bot. Currently, just a single one which says that we are waiting for the user to reply.

public enum State {
    AWAITING_TRAINING_DAY
}

To make use of the whole new code we need to initialize our ResponseHandler in the bot class. We do that using a new field inside of FitnessBot.

private final ResponseHandler responseHandler;

public FitnessBot(String botToken, String botUsername) {
        super(botToken, botUsername);
        responseHandler = new ResponseHandler(sender, db);
    }

Now we need to replace the action() method in our replyToStart() ability with

.action(ctx ->  responseHandler.replyToStart(ctx.chatId()))

But our ResponseHandler doesn’t have the referenced method, so far. You can find it below.

public void replyToStart(long chatId) {
        try {
                sender.execute(new SendMessage()
                        .setText(Constants.START_REPLY)
                        .setChatId(chatId));
                chatStates.put(chatId, State.AWAITING_TRAINING_DAY);
        } catch (TelegramApiException e) {
                e.printStackTrace();
        }
}

And our text is saved in Constants:

String START_REPLY = "Welcome I'm FitnessBot. I'm here to remind you doing sports every second day!";

If you run the application now and issue /start our newly added text will appear!

Using inline keyboards

Chances are that you will use more than one inline keyboard. That’s why we create a new class called KeyboardFactory. This class will create the needed keyboard instances for us. The code is pretty self-explanatory but see for yourself.

public class KeyboardFactory {
    public static ReplyKeyboard withTodayTomorrowButtons() {
        InlineKeyboardMarkup inlineKeyboard = new InlineKeyboardMarkup();
        List<List<InlineKeyboardButton>> rowsInline = new ArrayList<>();
        List<InlineKeyboardButton> rowInline = new ArrayList<>();
        rowInline.add(new InlineKeyboardButton().setText(Constants.TRAINING_TODAY).setCallbackData(Constants.TRAINING_TODAY));
        rowInline.add(new InlineKeyboardButton().setText(Constants.TRAINING_TOMORROW).setCallbackData(Constants.TRAINING_TOMORROW));
        rowsInline.add(rowInline);
        inlineKeyboard.setKeyboard(rowsInline);
        return inlineKeyboard;
    }
}

And in our Constants file we add:

String TRAINING_TODAY = "Today";
String TRAINING_TOMORROW = "Tomorrow";
String FIND_TRAINING_DATE = "Do you want to have a workout today or tomorrow?";

Now we can just call the static method of our factory to make use of an inline keyboard. The most important code part is setCallbackData(). It defines an identifier to recognize which button has been clicked by the user. Hint: In a real world application it might not be too smart to use the button text as identifier for a callback but we use it here to simplify the code.

Now we need to use the keyboard in our response by adding another sender.execute() below the first one which we already defined.

public void replyToStart(long chatId) {
        try {
            sender.execute(new SendMessage()
                .setText(Constants.START_REPLY)
                .setChatId(chatId));

            sender.execute(new SendMessage()
                .setText(Constants.FIND_TRAINING_DATE)
                .setChatId(chatId)
                .setReplyMarkup(KeyboardFactory.withTodayTomorrowButtons()));

            chatStates.put(chatId, State.AWAITING_TRAINING_DAY);

        } catch (TelegramApiException e) {
                e.printStackTrace();
        }
}

Give it a try! Our bot will now offer an inline keyboard to answer the training day question.

Inline keyboard interactions

So far we aren’t able to recognize clicks on the two buttons. But we can give our FitnessBot a new ability to do so! Similar to the image reply example given by Telegram we can filter for button responses via Flag.CALLBACK_QUERY. All the identifiers of clicked buttons will be sent inside of the update object upd. This object contains a lot of data but luckily there are some helper methods to extract the important information conveniently. getChatId(upd) will find the chat id of the update for you and via AbilityUtils.getUser(upd) you can get the author of the message.

Add the following lines to your bot class.

public Reply replyToButtons() {
        Consumer<Update> action = upd -> responseHandler.replyToButtons(getChatId(upd), upd.getCallbackQuery().getData());
        return Reply.of(action, Flag.CALLBACK_QUERY);
}

The tricky part here is the implementation of replyToButtons() inside of responseHandler because every single button click will be processed in this method. That’s why we use it for separation of concerns only.

public void replyToButtons(long chatId, String buttonId) {
        try {
                switch (buttonId) {
                        case Constants.TRAINING_TODAY:
                                replyToTrainingToday(chatId);
                                break;
                        case Constants.TRAINING_TOMORROW:
                                replyToTrainingTomorrow(chatId);
                                break;
                }
        } catch (TelegramApiException e) {
                e.printStackTrace();
        }
}

The actual logic resides in the referenced methods. First, it validates whether the bot is currently in a state where it is waiting for a response. Only if this is the case, the button is clicked. Depending on which button has been pressed, the bot is moved to another state.

private void replyToTrainingToday(long chatId) throws TelegramApiException {
        if (chatStates.get(chatId).equals(State.AWAITING_TRAINING_DAY)) {
                sender.execute(new SendMessage()
                        .setText(Constants.TRAINING_TODAY_REPLY)
                        .setChatId(chatId));
                chatStates.put(chatId, State.TODAY_IS_TRAINING_DAY);
        }
}

private void replyToTrainingTomorrow(long chatId) throws TelegramApiException {
        if (chatStates.get(chatId).equals(State.AWAITING_TRAINING_DAY)) {
                sender.execute(new SendMessage()
                        .setText(Constants.TRAINING_TOMORROW_REPLY)
                        .setChatId(chatId));
                chatStates.put(chatId, State.TODAY_IS_RELAX_DAY);
        }
}

The two referenced states are new thus have to be added to our enum:

public enum State {
    AWAITING_TRAINING_DAY, TODAY_IS_TRAINING_DAY, TODAY_IS_RELAX_DAY
}

Depending on the clicked button we want to answer something different. As usual, we need to add the answers to Constants:

String TRAINING_TODAY_REPLY = "Okay then take this as a reminder ;)";
String TRAINING_TOMORROW_REPLY = "Okay I'll remind you tomorrow at nine o'clock!";

Extra: Scheduled task execution

Now we need a way to remind our users to do sports on their training days. For that, we make use of ScheduledExecutorService from Java’s concurrent package. We’re not going to go into detail here, as it is not part of the bot framework. You just need to know that with the class below we have an easy way to execute a task at a specific time on a daily basis. E.g. to write a message at 9am in the morning. Create a new class called DailyTaskExecutor:

public class DailyTaskExecutor {
    private final ScheduledExecutorService executorService;
    private final DailyTask dailyTask;

    public DailyTaskExecutor(DailyTask dailyTask) {
        this.executorService = Executors.newScheduledThreadPool(1);
        this.dailyTask = dailyTask;
    }

    public void startExecutionAt(int targetHour, int targetMin, int targetSec) {
        Runnable taskWrapper = () -> {
            dailyTask.execute();
            startExecutionAt(targetHour, targetMin, targetSec);
        };
        long delay = computeNextDelay(targetHour, targetMin, targetSec);
        executorService.schedule(taskWrapper, delay, TimeUnit.SECONDS);
    }

    private long computeNextDelay(int targetHour, int targetMin, int targetSec) {
        LocalDateTime localNow = LocalDateTime.now();
        ZoneId currentZone = ZoneId.systemDefault();
        ZonedDateTime zonedNow = ZonedDateTime.of(localNow, currentZone);
        ZonedDateTime zonedNextTarget = zonedNow.withHour(targetHour).withMinute(targetMin).withSecond(targetSec);
        if(zonedNow.compareTo(zonedNextTarget) >= 0)
            zonedNextTarget = zonedNextTarget.plusDays(1);

        Duration duration = Duration.between(zonedNow, zonedNextTarget);
        return duration.getSeconds();
    }

    public void stop() {
        executorService.shutdown();
        try {
            executorService.awaitTermination(1, TimeUnit.DAYS);
        } catch (InterruptedException ex) {
            Logger.getLogger(DailyTaskExecutor.class.getName()).log(Level.SEVERE, null, ex);
        }
    }
}

DailyTask is an interface with an execute() method. We have to create an implementation of this interface later.

public interface DailyTask {
    void execute();
}

But first, let’s create an instance of DailyTaskExecutor in our FitnessBot class as a new field. As soon as we have our instance we schedule a new task which will run every morning at 9 am using startExecutionAt().

    private final DailyTaskExecutor dailyTaskExecutor;

    private FitnessBot(String botToken, String botUsername) {
        super(botToken, botUsername);
        responseHandler = new ResponseHandler(sender, db);
        dailyTaskExecutor = new DailyTaskExecutor(new MorningReminderTask(this));
        dailyTaskExecutor.startExecutionAt(9, 0, 0);
    }

Okay, now we need a new class for our DailyTask interface. We call it MorningReminderTask and it takes a callback listener as a constructor parameter. This callback listener is part of the class itself and gets called as soon as execute() will be run.

public class MorningReminderTask implements DailyTask {

    public interface Callback {
        void onTimeForMorningTask();
    }

    private final Callback callback;

    public MorningReminderTask(Callback callback) {
        this.callback = callback;
    }

    @Override
    public void execute() {
        callback.onTimeForMorningTask();
    }
}

Now you should have a compile error in your bot class because you try to pass your bot as a reference for callback in the constructor of MorningReminderTask. Before you can do that you need to implement MorningReminderTask.Callback in your bot.

public class FitnessBot extends AbilityBot implements MorningReminderTask.Callback {

    @Override
    public void onTimeForMorningTask() {
        responseHandler.sayMorningMessages();
    }
}

Due to that interface, you need to override onTimeForMorningTask() method. As our response handler class is taking care of writing messages we just call one of its methods. Which we have to create now:

public void sayMorningMessages() {
        try {
            for (long chatId : chatStates.keySet()) {
                switch (chatStates.get(chatId)) {
                    case TODAY_IS_TRAINING_DAY:
                        processTrainingDay(chatId);
                        break;
                    case TODAY_IS_RELAX_DAY:
                        processRelaxDay(chatId);
                        break;
                }
            }
        } catch (TelegramApiException e) {
            e.printStackTrace();
        }
    }

    private void processTrainingDay(long chatId) throws TelegramApiException {
        sender.execute(new SendMessage()
                        .setText(Constants.TRAINING_REMINDER)
                        .setChatId(chatId));
        chatStates.put(chatId, State.TODAY_IS_RELAX_DAY);
    }

    private void processRelaxDay(long chatId) {
        chatStates.put(chatId, State.TODAY_IS_TRAINING_DAY);
    }

In the constants file:

String TRAINING_REMINDER = "Good Morning! Don't forget to do sports, today.";

For each chat, we will now send a reminder message if we are in TODAY_IS_TRAINING_DAY state. Furthermore, we are changing the state to the opposite, so every second day will be a training day.

And that’s it. Now your bot is able to remind you every second day to do sports. Feel free to leave a comment if you have any questions or suggestions. You can find the whole code of this getting started on our github profile.

Greets,
Domi

Artificial Intelligence as a Service (AIaaS) is third-party offering of artificial intelligence, accessible via a API. So, people get to take advantage of AI without spending too much money, or if you’re lucky and a student, no money at all.

The Microsoft AI Platform

As already mentioned above, we have used several Microsoft services to build our prototypes, including several Microsoft Cognitive Services and the Azure bot service. All of them are part of the Microsoft AI platform, where you can find services, infrastructure and tools for Machine Learning and Artificial Intelligence you can use for your personal or business projects.

we used only some of the services of the AI platform

BlackForest Hackathon

The BlackForest Hackathon has been the first hackathon ever for our team and we have been quite excited to participate.

The theme proposed by the organizers of the hackathon was “your personal digital assistant”, and after some time brainstorming we came up with the idea of creating an intelligent bot, which assists you with creating your learning schedule. Most of us are serious procrastinators (including me :P), so we thought that such a bot can help us stick to our learning schedule and motivate us along the way to the exam.
The features we wanted to implement for our prototype are

• asking the user about his habits (usual breakfast, lunch and dinner time as well as sleep schedule),
• asking the user about his due exams (lecture, date and the amount of time the user wants to learn for the exam) and
• automatic creation of learning appointments, based on the user input, within the users Google calendar.

With the topic of the hackathon in mind we wanted the bot to gather the user input via a dialog as natural as possible.

Billige Plätze in Action

The technology stack

Cem and I have experimented a little bit with the Azure Bot Service before the hackathon, and we thought it to be a perfect match for the task of writing the bot. We also wanted the bot to process natural language and stumbled upon LUIS, a machine learning based service for natural language processing, which can be integrated seamlessly into the bot framework (because it is from Microsoft, too).
Our stack consisted of, as expected, mainly Microsoft technologies. We used

• C#,
• .Net core,
• Visual Studio,
• Azure Bot Service,
• LUIS and
• Azure.

The combination of C#, the bot service and LUIS provided the core functionality of our bot and we were able to deploy it to Azure within one click.

The Azure Bot Service

The Bot Service provides an integrated environment that is purpose-built for bot development, enabling you to build, connect, test, deploy, and manage intelligent bots, all from one place. Bot Service leverages the Bot Builder SDK with support for .NET and Node.js.

Overview of the Bot Service

The bot service consists of the concepts of

• channels, which connect the bot service with a messaging platform of your choice,
• the bot connector, which connects your actual bot code with one or more channels and handles the message exchange between the channels and the bot via
• activity objects.

Dialogs, another core concept, help organize the logic in your bot and manage conversation flow. Dialogs are arranged in a stack, and the top dialog in the stack processes all incoming messages until it is closed or a different dialog is invoked.

General Conversation flow of the Bot Service

By using the bot service we were able to focus on programming the actual conversation with the Bot Builder SDK. To use the bot service you just have to create a new bot in Azure, and connect to the channels of your choice (Telegram worked like a charm) also via the web app.
After creating your bot in Azure you can start coding right away by using a template provided by Visual Studio, you just have to type in your bot credentials in the configuration file and your good to go. Because we didn’t have to worry about where to host the bot and how to set it up we were able to quickly create a series of dialogs (which involved serious copy pasting :P) and test our conversation flow right away by using the botframework emulator, and when we were happy with the results publish the bot on Azure within one click in Visual Studio.
We didn’t have to worry about getting the user input from the messaging platform and integrating natural language understanding into our bot was very easy, because we used Microsoft LUIS. We were seriously impressed by the simplicity of the bot service.

Microsoft LUIS

LUIS is a machine learning-based service to build natural language into apps, bots, and IoT devices.
You can create your own LUIS app on the LUIS website. A LUIS app is basically a language model designed by you, specific for your domain. Your model is composed of utterances, intents and entities, whereas utterances are example phrases users could type into your bot, intents are the users intention you want to extract from the utterance and entities represent relevant information of the utterance, much like variables in a programming language.
An example for an utterance from our bot could be “I write my exam on the 23rd of August 2018”. Based on the utterance, our model is able to extract the intent “ExamDateCreation” as well as the entity “Date” <– 23.08.2018 (for a more detailed explanation, visit the LUIS Documentation). Once you define all your intents and entities needed for your domain in the LUIS web application and provide enough sample utterances, you can test and publish your LUIS app. After publishing you can access your app via a REST API, or in our case through the Azure bot service.

the LUIS web application

LUIS is tightly integrated in the bot service, to integrate our model all we had to do was to add an annotation to a class and extend the prefabricated LuisDialog to get access to our intents and entities.

[Serializable]
[LuisModel("your-application-id", "your-api-key")]
public class ExamSubjectLuisDialog : LuisDialog<object>
{
  [LuisIntent("Klausurfach")]
  private async Task ExamPowerInput(IDialogContext context, IAwaitable<object> result,
  Microsoft.Bot.Builder.Luis.Models.LuisResult luisResult)
        {
        ...
        }
}

Theres nothing else to do to integrate LUIS into your bot. The bot connector service handles the message exchange between your bot code and the LUIS REST API and converts the JSON into C# objects you can directly use. Fun fact: the integration of the google calendar into our bot took us several hours, a lot of nerves and around 300 lines of code, whereas the integration of our LUIS model took around 5 minutes and the lines of code for every LuisDialog we created.

Summary

By using the Azure bot service in combination with a custom LUIS model we were able to create a functional prototype of a conversational bot assisting you in creating your custom learning schedule by adding appointments to your Google calendar in roughly 24 hours, all while being able to understand and process natural language. With the power of the bot service, the bot is available on a number of channels, including Telegram, Slack, Facebook Messenger and Cortana.

It was a real pleasure to use these technologies because they work seamlessly together. Being able to use ready-to-use dialogs for LUIS sped up the development process enormously, as well as being able to deploy the bot on azure within one click out of Visual Studio. I included a little demonstration video of the bot below, because it is no longer in operation.

demo part 2

Zeiss Hackathon Munich

Our second Hackathon we participated in took place in January 2018. It was organized by Zeiss and sponsored by Microsoft.

The theme of this hackathon was VISIONary Ideas wanted, and most of the teams did something with VR/AR. One of the special guests of the hackathon was a teenager with a congenital defect resulting in him to be left with only about 10% of his eye sight. The question asked was “can AI improve his life?”, so we sat down with him and asked him about his daily struggle being almost blind. One problem he faces regularly, and also according to our internet research other blind people, is the withdrawal of cash from an ATM.
So after some time brainstorming we came up with the idea of a simple Android app which guides you through the withdrawal process via auditory guidance. Almost everyone has a smartphone, and the disability support for them is actually pretty good, so a simple app is a pretty good option for a little, accessible life improvement.

We figured out 3 essential things we needed to develop our app:
1. Image Classification, to distinguish between different types of ATM (for our prototype we focused on differentiating ATMs with touch screen and ATMs with buttons on the sides),
2. Optical Character Recognition, to read the text on the screen of the ATM to detect the stage of the withdrawal process the user is in and generate auditory commands via
3. Text to Speech, which comes out of the box in Android.

The Technology Stack

We wanted to develop an Android app, so we used Android in combination with Android Studio. We chose to develop an Android app simply because some of us were familiar with it and I always wanted to do something with Android.

For both the image classification and the OCR we again relied on Microsoft Services.
The Microsoft Vision API provides a REST endpoint for OCR, so we got ourselves a API key and we were ready to go.

For the image classification Microsoft provides the custom vision service, where you can train your own model.

Plugging the parts together

The flow of our app is quite simple, it takes a picture every 5 seconds, converts it into a byte array and first sends it to our custom image classifier to detect the ATM type. After the succesful classification of the ATM all further taken images are sent to the vision API for optical character recognition. We get a JSON as a response with all the text the vision API was able to extract from the image. The app then matches keywords, we focused on the flow of ATMs from Sparkasse, with the extracted text to detect the current stage of the withdrawal process and then create auditory commands via text to speech. We didn’t even have to rely on frameworks by Microsoft like in the first hackathon, all we needed was to call the REST API of the services and process the response.

Summary

Much like in the first hackathon we were really impressed how good the services work out of the box. As you can imagine the images of the ATM were oftentimes pretty shaky, but the OCR worked pretty good nevertheless. And because we only matched important, but distinct, keywords of every stage of the withdrawal process, the app could handle wrongly extracted sentences to a good degree. Our custom image classifier was able to differentiate between ATMs with touch screens and ATMs with buttons pretty reliable with only 18(!) sample pictures.

our custom ATM type classifier

After the 24 hours of coding (-2 hours of sleep :/ ), we had a functioning prototype and were able to pitch our idea to the jury with a “living ATM” played by Danny :).

The jury was impressed by the prototype and our idea of a “analog” pitch (we had no PowerPoint at all) and awarded us the Microsoft price. We all got a brand new Xbox One X, a book from Satya Nadalla and an IoT kit, which was pretty cool (sorry for bragging :P).

Billige Plätze and their new Xboxes

Takeaways

I think there are 3 main points you can take away from this blog post and my experiences with AI as a service.

1. You can create working prototypes in less than 24 hours
2. You don’t have to start from scratch
3. It is not a shame to use finished models. When you need more, there is always time!

Before using these services I thought AI to be very time-consuming and difficult to get right. But we were able to create working prototypes in less than 24 hours! The models provided by Microsoft are very good and you can integrate them seamlessly into your application, be it in your conversational bot or in your Android App.
All projects are available in our GitHub organization, but beware, the code is very dirty, as usual for hackathons!

I hope I was able to inspire you to create your own application using AI services, and take away some of your fears of the whole AI and Machine Learning thing.

Have fun writing your own application and see you in our next blog post!