Friday, September 29, 2017

Extracting Website Data and Creating APIs with WrapAPI

Today, almost all services we use have some sort of API. Some web applications are even built from API points alone, being passed to some kind of front-end view. If you're a consumer of a service that provides an API, you'll sometimes need more features or find limits to what the API can offer. In this article, we'll cover a service that's useful both for API consumers and creators.

I always go with the saying that, if there's a web interface, you can build your own API over it. WrapAPI tries to make this process easier. If you're familiar with the process of web scraping/crawling (or extracting data from websites), you'll see the magic of WrapAPI.

WrapAPI offers a service that allows you to easily extract information from websites and create APIs from the data. It provides an easy, interactive way of selecting what information you want to get. With just a few clicks, you can have your API online.

To follow along with this tutorial, I recommend you head over to wrapapi.com and create an account.

How To Get Around WrapAPI

On the WrapAPI site, you'll see that you can start to build your project right away --- although, unless you create an account, your work won't be saved.

Once you've signed up, click the Try building an API button.

WrapAPI Web Builder

You'll be presented by a browser-like interface. On top of the site we're presented with a URL bar. As an example, WrapAPI uses Hacker News (http://ift.tt/YV9WJO). If you click the URL to change it to something else, you'll see more options related to the request you want to make. We'll use the default options, and only change the URL to http://ift.tt/2brOFq8. We're covering only the GET method, as we only want to get data in this example.

Below the URL bar there are four buttons that give you different information regarding the site you're viewing. Browser view displays the site as you would visit it from your browser. Code view displays the source code of the site. Headers shows the response you get from the server. This is useful if you want to see what response you get from the server: it gives you information like the HTTP status codes (200, 404, 400 etc.), content types, web servers and so on. You can also view the request's Cookies directly from the builder.

Getting the Data

By now you should be able to see SitePoint inside the Browser View frame.

Preview of SitePoint inside WrapAPI Builder

Let's create a very simple API that shows us the latest post titles of the JavaScript channel. If you hover over the titles, images or any other element in the site, you'll notice a selection color covering it. Let's scroll down a bit, to the LATEST articles part. Hover over the title from one of the articles and click on that title. You'll notice that it doesn't switch to that particular link we clicked. We see that every title in this section is highlighted. WrapAPI guessed that these are all the titles we want. Sometimes it can also select parts of the sites we don't want. That's usually the case when the CSS class selectors are not well-defined or used by other elements in the site.

Besides CSS selectors, WrapAPI supports regular expressions, JSON selectors, headers, cookies, form outputs, and a bunch more options. You can use them all together and extract exactly what you're aiming for. In this example, we'll only use CSS selectors.

Selecting titles in the builder

In the right part of the interface, you'll see three tabs. Let's take a look at the current Build tab. Outputs will show us the selectors (in our case CSS selectors), and you'll get more details on what you would like to select. We're interested only in extracting the title, which is text. There are more options on cleaning the result output, but we won't get into these details. If you'd like to create another selector, to select description, author, date, etc., just click the Create a new collection/output. Naming your selectors is also important, as this will make it easier if you use multiple selectors in the site. By clicking the pencil icon, you can edit your selectors.

Extract/Ouput options

The Preview tab will show a representation of our data in JSON, and you probably get the idea of what the API will look like. If you're happy with the results, you can click the Save button to save a version of the API.

Saving API endpoints

You'll need to enter the repository and the endpoint name of the API. It helps you manage and organize your APIs. That will also be part of your API's name in the end. After entering the information, you'll return to the builder. Our API is saved, but now we need to test and publish it.

Tips:

  • If the site has pagination (previous/next pages), you can use the query string options. (More on that here.)
  • Name your selectors correctly, as they'll be part of the JSON output.

Continue reading %Extracting Website Data and Creating APIs with WrapAPI%


by Ardian Haxha via SitePoint

#354: React 16 Released

This week's JavaScript newsRead this e-mail on the Web
JavaScript Weekly
Issue 354 — September 29, 2017
The final release of React 16 is here, with rewritten ‘Fiber’ internals. There’s a lot more React 16 stuff in today’s React Status newsletter.
Facebook Code

Recently, there have been some issues involving patent-related features, but Facebook is now moving some of its projects to a standard MIT license.
Facebook Code

JavaScript telemetry gives a timeline of browser events leading to an error, including interaction events, like clicks, inputs, navigation and console messages + more. Debug better w/ telemetry and know why your app crashed.
ROLLBAR   Sponsor

Draggable abstracts native browser events into a modular API for you to build custom, and potentially complex, drag and drop experiences.
Shopify

Learning by building in action in this step-by-step tutorial to help you understand how Promises work.
Trey Huffine

Wissam Abirached explains context and the this keyword in JavaScript, the difference between explicit and hard bindings and what the new keyword represents.
Telerik Developer Network

It’s pitched at being an ideal step up for developers comfortable with jQuery who want to move on to something else.
Mateusz Łuczak

Speakers Holly Schinsky & Jen Looper will be covering Vue & Preact next Thursday (Oct 5).
Certified Fresh Events

Jobs Supported by Hired.com

Can't find the right job? Want companies to apply to you? Try Hired.com.

In Brief

BigInt: A Proposal for Arbitrary Precision Integers in JS news
Ecma TC39

Write Less Code, Use More Tools 
Our outrageous decision to orchestrate Nomad with Kubernetes.
CircleCI  Sponsor

A Brief History of Modularity in JavaScript tutorial
Nicolás Bevacqua

ES6 Promises: Patterns and Anti-Patterns tutorial
Bobby Brennan

Exploring set(), get() and Object.defineProperty() in TypeScript tutorial
Todd Motto

Why Node 8's util.promisify Is So Awesome tutorial node
James Jeffery

Love fraud? You’re going to hate Blockchain 
SitePen  Sponsor

Web Truths: JavaScript Can’t Be Trusted opinion
Christian Heilmann

The Past, Present, and Future of the Angular CLI story
Stephen Fluin

ECMAScript, TC39, and the History of JavaScript video
Tyler McGinnis

Rapid prototyping playground for JavaScript in your editor 
Run any code and see the results immediately right in your editor. Supports VS Code, Atom and JetBrains IDEs.
Wallaby.js  Sponsor

CheerpJ, a Java to JavaScript Compiler, Now in Beta tools
Compile unmodified Java apps to JS for running in the browser.
Leaning Tech

Cerebral: Declarative State and Side Effects Management for JS Frameworks code
An intriguing idea, at least.
Christian Alfoni

Cherow: A Fast, TypeScript-Based JavaScript Parser code
KFlash

Slack JS 9.0.0: A JS Client for Slack's Web API code
Brian Leroux

vue-a11y-calendar: An Accessible, Internationalized Vue Calendar code
IBM

ppipe: Pipes Values Through Functions code
An alternative to using the proposed ‘pipe’ operator for ES.
Yavuz Ege Özcan

Sheet: A Simple Spreadsheet in 220 Bytes of HTML and JS code

Curated by Peter Cooper and published by Cooperpress.

Like this? You may also enjoy: FrontEnd Focus : Node Weekly : React Status

Stop getting JavaScript Weekly : Change email address : Read this issue on the Web

© Cooperpress Ltd. Fairfield Enterprise Centre, Lincoln Way, Louth, LN11 0LS, UK


by via JavaScript Weekly

HustleCase

Lovely clean pre-order Landing Page built on the Shopify framework for a whiteboard laptop cover called HustleCase.

Full Review | Direct Link


by Rob Hope @robhope via One Page Love

Best Video Background Plugins for WordPress

Building a Slack Bot Using Node.js

Slack is quickly becoming the new industry standard for teams to communicate with. In fact, it is so popular that when I typed slack into Google, as I expected, the first result was the definition of the word from the dictionary. This was followed immediately by Slack's website! 

This is almost unheard of for most common words in the English dictionary. Usually, Google's definition is followed by several links to the top dictionary websites.

What is Slack?

At its most basic, Slack is a messaging system. It allows for direct messages to team members and the creating of channels (private or public) that allow easy real-time team communication and collaboration. For more information on Slack, you can view Slack's Features.

At this point, you might be wondering where Node.js comes in. As I mentioned, at its most basic, Slack is a messaging system; however, it can be infinitely extended and customized. Slack provides an incredibly flexible system to customize your team's integration, including:

  • creating custom welcome messages
  • creating custom emojis
  • installing third-party applications
  • creating your own applications
  • creating custom Slack Bots

In this article, I am going to demonstrate how to create a Slack Bot with Node.js that can be added to your team's Slack configuration.

Slack Bots Defined

A Slack Bot's job is to receive events sent from Slack and handle them. There are a plethora of events that will be sent to your Bot, and this is where Node.js will come in. We must decide not only which events to handle, but how to handle each individual event.

For example, some common events that a Bot would handle are:

  • member_joined_channel
  • member_left_channel
  • message

In this article, I will create a Node.js application and a Slack Bot that can be added to your team project to perform specific actions based on the events it receives.

To begin, I need to create a Bot on Slack. Two types of bots can be created:

  • a custom bot
  • creating an application and adding a bot user

This article will create a custom bot because an application bot user would be more appropriate if you were planning to write and publish an application on Slack. Given that I wish this bot to be private to my team, a custom bot will suffice.

Creating a Custom Slack Bot

A custom bot can be created here: http://ift.tt/2fVYiPf. If you are already logged in to your Slack account, on the left select the Add Configuration button; otherwise, log in to your Slack account before proceeding. If you do not have a Slack account, you can sign up for free.

This will take you to a new page that requires you to provide a username for your bot. Enter your username now, ensuring you follow Slack's naming guidelines. Once you have selected an awesome bot name, press Add bot configuration.

After you have successfully created your bot, Slack redirects you to a page that allows for further customization of your bot. I'll leave that part to your creative self. The only thing needed from this page is the API Token that starts with xoxb-. I would either copy this token to a safe place for later use or simply leave this page open until we need the token for the Node.js application.

Configurations

Before moving on to code, two more Slack configurations are required:

  1. Create or choose an existing channel that your bot will interact with. While I'm testing my new bot, I chose to create a new channel. Be sure to remember the channel name as you will need it within your application shortly.
  2. Add/Invite your bot to the channel so it can interact with it.

Now that I've got my Slack Bot configured, it's time to move on to the Node.js application. If you already have Node.js installed, you can move on to the next step. If you do not have Node.js installed, I suggest you begin by visiting the Node.js Download page and selecting the installer for your system.

For my Slack Bot, I am going to create a new Node.js application by running through the npm init process. With a command prompt that is set to where you wish your application to be installed, you can run the following commands:

If you are unfamiliar with npm init, this launches a utility to help you configure your new project. The first thing it asks is the name. It defaulted mine to slackbot, which I'm comfortable with. If you would like to change your application name, now is the chance; otherwise, press Enter to proceed to the next configuration step. The next options are version and description. I've left both as the default and simply continued by pressing Enter for both of these options.

Entry Points

The next thing that is asked for is the entry point. This defaults to index.js; however, many people like to use app.js. I do not wish to enter this debate, and given my application will not require an intensive project structure, I am going to leave mine as the default of index.js.

After you've recovered from a debate that is probably as strong as tabs vs. spaces, the configuration continues, asking several more questions:

  • test command
  • git repository
  • keywords
  • author
  • license

For the purposes of this article, I've left all options as their default. Finally, once all options have been configured, a confirmation of the package.json file is displayed prior to creating it. Press Enter to complete the configuration.

Enter the SDK

To make interacting with Slack easier, I'm also going to install the Slack Developer Kit package as follows:

Are you finally ready for some code? I sure am. To begin, I'm going to use the example code from the Slack Developer Kit's website that posts a Slack message using the Real-Time Messaging API (RTM) with a few tweaks.

Given that the entry point I chose was index.js, it's time to create this file. The example from the Slack Developer Kit's website is roughly 20 lines of code. I'm going to break it down several lines at a time, only to allow for explanations of what these lines are doing. But please note that all these lines should be contained in your index.js file. 

The code begins by including two modules from the Slack Developer Kit:

The RtmClient, once instantiated, will be our bot object that references the RTM API. The CLIENT_EVENTS are the events that the bot will be listening for.

Once these modules are included, it's time to instantiate and start the bot:

Be sure to replace the API Token that is obfuscated above with your token obtained during the Slack Bot creation.

Calling the start function on my RtmClient will initialize the bot's session. This will attempt to authenticate my bot. When my bot has successfully connected to Slack, events will be sent allowing my application to proceed. These events will be shown momentarily.

With the client instantiated, a channel variable is created to be populated momentarily inside one of the CLIENT_EVENTS events.

The channel variable will be used to perform specific actions, such as sending a message to the channel the bot is connected to.

When the RTM Session is started (rtm.start();) and given a valid API Token for the bot, an RTM.AUTHENTICATED message will be sent. The next several lines listen for this event:

When the RTM.AUTHENTICATED event is received, the preceding code performs a for loop through the list of Slack team channels. In my case, I'm specifically looking for jamiestestchannel and ensuring that my bot is a member of that channel. When that condition is met, the channel ID is stored in the channel variable.

Debugging

To aid in debugging, a console message is logged that displays a message indicating that the bot has successfully authenticated by displaying its name (${rtmStartData.self.name}) and the team name (${rtmStartData.team.name}) it belongs to.

After the bot has authenticated, another event is triggered (RTM.RTM_CONNECTION_OPENED) that signifies the bot is fully connected and can begin interacting with Slack. The next lines of code create the event listener; upon success, a Hello! message is sent to the channel (in my case, jamiestestchannel).

At this point, I can now run my Node application and watch my bot automatically post a new message to my channel:

The results of running this command (when successful) are twofold:

  1. I receive my debug message indicating that my bot has successfully logged in. This originated from the RTM.AUTHENTICATED being triggered after starting the RTM Client.
  2. I receive a Hello! message in my Slack channel. This occurred when the RTM.RTM_CONNECTION_OPENED event message was received and handled by the application.

Before proceeding and further enhancing my application, now is a good time to recap what I have done to get this far:

  1. Created a custom Slack Bot.
  2. Created a custom Slack Channel and invited my bot to it.
  3. Created a new Node.js application called slackbot.
  4. Installed the Slack Developer Kit package to my application.
  5. Created my index.js file that creates an RtmClient using my API Token from my custom bot.
  6. Created an event listener for RTM.AUTHENTICATED that finds the Slack Channel my bot is a member of.
  7. Created an event listener for RTM.RTM_CONNECTION_OPENED that sends a Hello! message to my Slack Channel.
  8. Called the RTM Start Session method to begin the authentication process that is handled by my event listeners.

Building the Bot

Now it's time for the real fun to begin. Slack offers (I didn't count) at least 50 different events that are available for my custom bot to listen and optionally handle. As you can see from the list of Slack Events, some events are custom to the RTM API (which we are using), while other events are custom to the Events API. At the time of writing this article, it is my understanding that the Node.js SDK only supports RTM.

To finish my bot, I will handle the message event; of course, this is probably one of the most complicated events as it supports a large number of sub-types that I will explore in a moment.

Here is an example of what the most basic message event looks like from Slack:

In this basic object, the three most important things I care about are:

  1. The channel. I will want to ensure this message belongs to the channel my bot is a part of.
  2. The user. This will allow me to interact directly with the user or perform a specific action based on who the user is.
  3. The text. This is probably the most important piece as it contains the contents of the message. My bot will want to only respond to certain types of messages.

Some messages are more complicated. They can contain a lot of sub-properties such as:

  • edited: A child object that describes which user edited the message and when it occurred.
  • subtype: A string that defines one of the many different kinds, such as channel_join, channel_leave, etc.
  • is_starred: A boolean indicating if this message has been starred.
  • pinned_to: An array of channels where this message has been pinned.
  • reactions: An array of reaction objects that define what the reaction was (e.g. facepalm), how many times it occurred, and an array of users who reacted this way to the message.

I'm going to extend my previously created index.js to listen for message events. To reduce redundancy of code, the following examples will contain just the portion of code related to the message event enhancements.

The first thing that must be done is to include a new module for the RTM_EVENTS that I will be listening to. I've placed this below my two previous module includes:

The code for handling the message event I will be placing at the bottom of my file. To test that the message event is working correctly, I've created a new event listener that logs the message object to the console as follows:

I can now re-run my Node application (node index.js). When I type a message into my channel, the following is logged to my console:

So far, so good. My bot is successfully receiving messages. The next incremental step to make is to ensure the message belongs to the channel my bot is in:

Now when I run my application, I only see my debug message if the message event was for the channel that my bot is a part of.

I'm now going to extend the application to send a custom message to the channel demonstrating how a user can be tagged in a message:

Now, when anyone types a message in the channel, my bot sends its own message that looks something like: "Stop, everybody listen, @endyourif has something important to say!"

Ok, not extremely useful. Instead, I'm going to finish my bot by enhancing the message event listener to respond to specific commands. This will be accomplished by doing the following:

  1. Split the text portion of a message into an array based on a blank space.
  2. Check if the first index matches my bot's username.
  3. If it does, I will look at the second index (if one exists) and treat that as a command that my bot should perform.

To make it easy to detect if my bot was mentioned, I need to create a new variable that will store my bot user ID. Below is an updated section of code where I previously set the channel variable. It now also stores my bot's user ID in a variable called bot.

With my bot variable set, I've finished my bot by fleshing out the previously created message event listener as follows:

The following code splits the text property of the message object in an array based on a space. I next ensure that I have at least two elements in the array, ideally my bot and the command to perform.

When the first element in the array matches my bot, I perform a switch statement on the second element in the array: the command. The current commands supported are jump and help. When a message is sent to the channel that looks like "@jamiestest jump", my bot will respond with a special message to the originating user. 

If the command is not recognized, it will fall into my default case statement for my switch and respond with a generic command that will look like this: "@endyourif, sorry I do not understand the command "hi". For a list of supported commands, type: @jamiestest help".

Conclusion

At this point, my bot is complete! If you are interested in further enhancing your bot, here's a list of ideas:

  • Handle a new team member joining by listening to the team_join event. When a new team member joins, it would be a great idea to send them a variety of onboarding information and/or documentation welcoming them to your team.
  • Enhance the list of supported commands that I've started.
  • Make the commands interactive by searching a database, Google, YouTube, etc.
  • Create a bot user on an application and create your own custom slash commands.

by Jamie Munro via Envato Tuts+ Code

8 Hidden Android Features You Probably Don't Know About [video]

We've found 8 little tricks and hidden features in Android that you might not know about!

[ This is a content summary only. Visit our website http://ift.tt/1b4YgHQ for full links, other content, and more! ]

by Web Desk via Digital Information World

Echoes of the Studio

Echoes of the Studio, created by the modern artist Xavier Veilhan, is offering an immersive audio-visual experiment for the 57th International Arts Bienniale in Venice.
by via Awwwards - Sites of the day