How to Connect Your Api.ai Assistant to the IoT

The potential of a personal assistant gets exciting when it has access to personal data and the real world via the Internet of Things. New possibilities arise --- from requesting your assistant turn on your lights to asking it how well you slept. We'll be connecting your Api.ai assistant to the Jawbone Up API as an example of this.

Note: this article was updated in 2017 to reflect recent changes to Api.ai.

What You'll Need

This article builds upon a variety of concepts we've already covered in previous articles here at SitePoint. In order to follow along with this tutorial comfortably, you'll need the following.

• An Api.ai agent connected to a simple HTML web app. See this article if you'd like to understand this process. Otherwise, you can download the code from this guide and use it too.
• An agent that has been taught the entity of "sleep". We created this in Empowering Your Api.ai Assistant with Entities. It should understand concepts like "how much sleep did I have last night?" and "how much REM sleep did I get?" If you're looking to adapt this to your own IoT device, you'll need to have created your own custom entity that understands your IoT functionality.
• A general knowledge of Node.js and running a Node server. Without it, you won't be able to get the server running!
• Knowledge of how to use the Jawbone UP API (or another API you intend to use). We've covered the Jawbone Up API previously in Connecting to the Jawbone Up API with Node.js, and I'll be referring to sections from that article throughout.
• An SSL certificate to run your site on HTTPS. You'll need this if working with the Jawbone Up API. As mentioned at the start of this series, overall it's a bit easier to do a lot of this on HTTPS. We cover how to set up a self signed certificate on the Jawbone Up API article if you're interested, but it's not the easiest option these days. You can do it really easily using Let's Encrypt as mentioned in the first article in the series. Glitch.com also provides a great prototyping environment that comes with HTTPS by default.

The Code

All code for this demo is available for you to download and use however you please! You can find it all on GitHub.

How This Works

Your Api.ai assistant is already connected to a simple web app that accepts statements via the HTML5 Speech Recognition API. From here, you need to add a new bit of functionality that listens for a specific action from your Api.ai agent. In your case, this is the action of "sleepHours".

Whenever your JavaScript detects this action, it triggers a separate call to your Node.js app to ask the Jawbone API for that data. Once the web app receives this data, your web app turns it into a nice sentence and reads it out --- giving your assistant a whole new range of intelligence!

Your Project Structure

I've adjusted the app from the initial HTML-only structure to one that uses EJS views so that you can switch pages in your web app when logging into the Jawbone Up API via OAuth. In reality, you only really have one page, but this method allows you to add more in future if needed for other IoT devices. This single view is at /views/index.ejs. You then have your Node server in the root folder as server.js and certificate files in root too. To keep things relatively simple and contained, all front-end JavaScript and CSS is inline. Feel free to move these into CSS and JS files as you prefer, minify them and make them pretty.

Responding to Api.ai Actions in JavaScript

As you might remember from the previous article, when Api.ai returns a response, it provides a JSON object that looks like so:

[code language="js"]
{
"id": "6b42eb42-0ad2-4bab-b7ea-853773b90219",
"timestamp": "2016-02-12T01:25:09.173Z",
"result": {
"source": "agent",
"resolvedQuery": "how did I sleep last night",
"speech": "I'll retrieve your sleep stats for you now, one moment!",
"action": "sleepHours",
"parameters": {
"sleep": "sleep"
},
"metadata": {
"intentId": "25d04dfc-c90c-4f55-a7bd-6681e83b45ec",
"inputContexts": [],
"outputContexts": [],
"contexts": [],
"intentName": "How many hours of @sleep:sleep did I get last night?"
}
},
"status": {
"code": 200,
"errorType": "success"
}
}
[/code]

Within that JSON object there are two bits of data you need to use --- action and parameters.sleep:

[code language="js"]
"action": "sleepHours",
"parameters": {
"sleep": "sleep"
},
[/code]

action is the name you gave to the Api.ai action that the user has triggered. In the case of your sleep example, you named it "sleepHours". parameters contain the variables in your sentence that can change a few details. In the case of sleep, your parameter tells you what type of sleep --- "sleep", "deep sleep", "light sleep" or "REM sleep" (or just "REM").

Initially, in an earlier article on Api.ai, the prepareResponse() function took the JSON response from Api.ai, put the whole thing into your debug text field on the bottom right and took out Api.ai's verbal response to display in the web app. You completely relied on what your Api.ai agent said, without adding any of your own functionality:

[code language="js"]
function prepareResponse(val) {
var debugJSON = JSON.stringify(val, undefined, 2),
spokenResponse = val.result.speech;

respond(spokenResponse);
debugRespond(debugJSON);
}
[/code]

This time around, keep an eye out for the action field and run your own function called requestSleepData() if the action contains "sleepHours". Within this function, pass in the sleep parameter so you know what type of sleep is being requested:

[code language="js"]
function prepareResponse(val) {
var debugJSON = JSON.stringify(val, undefined, 2),
spokenResponse = val.result.speech;

if (val.result.action == "sleepHours") {
requestSleepData(val.result.parameters.sleep);
} else {
respond(spokenResponse);
}

debugRespond(debugJSON);
}
[/code]

Within requestSleepData(), request all sleep data from your Node.js server and then filter it by looking at the very first value in the returned array of data (data.items[0].details): this would be last night's sleep. Within these details, you have data.items[0].details.rem with your REM sleep, data.items[0].details.sound with your deep sleep, data.items[0].details.light with your light sleep and data.items[0].details.duration with the combined amount of sleep recorded:

[code language="js"]
function requestSleepData(type) {
$.ajax({
type: "GET",
url: "/sleep_data/",
contentType: "application/json; charset=utf-8",
dataType: "json",
success: function(data) {
console.log("Sleep data!", data);

if (data.error) {
respond(data.error);
window.location.replace("/login/jawbone");
}

switch (type) {
case "REM sleep":
respond("You had " + toHours(data.items[0].details.rem) + " of REM sleep.");
break;
case "deep sleep":
respond("You had " + toHours(data.items[0].details.sound) + " of deep sleep.");
break;
case "light sleep":
respond("You had " + toHours(data.items[0].details.light) + " of light sleep.");
break;
case "sleep":
respond("You had " + toHours(data.items[0].details.duration) + " of sleep last night. That includes " + toHours(data.items[0].details.rem) + " of REM sleep, " + toHours(data.items[0].details.sound) + " of deep sleep and " + toHours(data.items[0].details.light) + " of light sleep.");
break;
}
},
error: function() {
respond(messageInternalError);
}
});
}
[/code]

toHours() is a crude and quick function that formats your times into sentences like "1 hour, 53 minutes and 59 seconds":

[code language="js"]
function toHours(secs) {
hours = Math.floor(secs / 3600),
minutes = Math.floor((secs - (hours * 3600)) / 60),
seconds = secs - (hours * 3600) - (minutes * 60);

hourText = hours + (hours > 1 ? " hours, " : " hour, ");
minuteText = minutes + (minutes > 1 ? " minutes " : " minute ");
secondText = seconds + (seconds > 1 ? " seconds" : " second");

return hourText + minuteText + "and " + secondText;
}
[/code]

As you'll see when looking into the requestSleepData() function, the end result is a call to respond() --- the same function that previously took Api.ai's voice response. You reuse your existing functionality to bring speech to your own response, allowing your assistant to tell the user this information once it's ready.

One last aspect of your front-end JavaScript to point out is error handling. If you have an issue with how Jawbone returns data (usually due to not being logged into the service), your server responds with a JSON value in the format of {"error" : "Your error message"}. The assistant sees this and automatically takes the user to your OAuth login page:

[code language="js"]
if (data.error) {
respond(data.error);
window.location.replace("/login/jawbone");
}
[/code]

Continue reading %How to Connect Your Api.ai Assistant to the IoT%

by Patrick Catanzariti via SitePoint