Filtering Reality with JavaScript and Google Cardboard

The ability to run virtual reality within a mobile browser is empowering and exciting. Google Cardboard and other similar VR devices make it unbelievably simple, just place your phone into the holder and go! I previously covered Bringing VR to the Web with Google Cardboard and Three.js, where I discussed the basics of building a VR environment that pulls in web data. People really enjoyed that article (and I really enjoyed building that demo) so I thought I'd expand on it with a different idea. Rather than bringing in web APIs, why not bring in your phone's camera and turn this into an Augmented Reality experience?

In this article, I'm going to explore how we can pull in camera data, filter it and display it back using HTML5 and JavaScript. We'll do this all through a stereoscopic vision effect to create an Augmented Reality experience for Google Cardboard and other VR devices. We'll apply a few different filters to our camera stream - a cartoonish greyscale filter, a sepia film style filter, a pixelated filter (my favorite) and an inverse color filter.

If you are completely new to filtering images with HTML5, the canvas tag and JavaScript, I have a whole course on the topic over at Learnable called JavaScript in Motion! I'll be approaching this article with the assumption that you understand the canvas and video tags, along with how to stream videos into the canvas tag. Or with the assumption that you're confident enough to work it out as you go!

Demo code

If you're keen to get straight into the code and try it out, you can find it here on GitHub.

Want to try it in action? I've got a running version hosted here: Reality Filter.

How This Will Work

We'll be taking the same initial set up from the previous Google Cardboard article - a Three.js scene that we display through a stereoscopic effect. That effect allows us to have a display for each eye, making things look wonderfully 3D in VR. However, rather than floating particles and such from the previous article, we remove most elements and place one simple Three.js mesh in front of the camera that plays our camera feed.

Our Code Explained

Looking at our variable declarations, most of the variables here will look familiar to those who've gone through the previous demo. The variables for preparing our Three.js scene, camera, renderer, element for our canvas output, container to place that element in and a variable to store our stereoscopic effect are all the same.

[code language="js"] var scene, camera, renderer, element, container, effect, [/code]

Our three new variables related to our camera feed are video, canvas and context.

[code language="js"] video, canvas, context, [/code]

• video - Our actual HTML5 <video> element. That will have our camera feed playing within it.
• canvas - A virtual canvas element that will have the contents of our video element. We will read in the video data from this canvas and then add our theme filters back onto it, before placing its contents into our Three.js scene.
• context - Our canvas' 2D context which we use to perform most functions against it.

We have a few other variables under those which relate to our filter functionality.

[code language="js"] themes = ['blackandwhite', 'sepia', 'arcade', 'inverse'], currentTheme = 0, lookingAtGround = false; [/code]

• themes - An array of the names of our filters.
• currentTheme - The index we're currently viewing within the themes array.
• lookingAtGround - Whether or not we've looked at the ground (this one will make more sense soon).

We start with our init() function setting up our scene, camera and so forth as before:

[code language="js"] init(); function init() { scene = new THREE.Scene(); camera = new THREE.PerspectiveCamera(90, window.innerWidth / window.innerHeight, 0.001, 700); camera.position.set(0, 15, 0); scene.add(camera); renderer = new THREE.WebGLRenderer(); element = renderer.domElement; container = document.getElementById('webglviewer'); container.appendChild(element); effect = new THREE.StereoEffect(renderer); element.addEventListener('click', fullscreen, false); [/code]

We do not have any camera movement functionality via the DeviceOrientation event this time around. Compared to a VR experience, we won't need to change the actual camera position in this Three.js scene. We're keeping the scene in the same spot - the camera feed is what will be moving when the user looks around.

One listener we have kept from the previous example is an event listener to go fullscreen if we tap the scene. This removes the Chrome address bar from our view.

A Different Use For DeviceOrientationEvent

There is a new use for the DeviceOrientationEvent in this demo. We set it to watch for changes in the orientation of our device and use that as a trigger for switching our filter. We don't really have any physical controls to trigger events, so we control things by where the user is looking. In particular, we change the filter any time the user looks at the ground.

[code language="js"] if (window.DeviceOrientationEvent) { window.addEventListener('deviceorientation', function(evt) { if (evt.gamma > -1 && evt.gamma < 1 && !lookingAtGround) { lookingAtGround = true; currentTheme = (themes.length > currentTheme+1) ? currentTheme+1 : 0; setTimeout(function() { lookingAtGround = false; }, 4000); } }.bind(this)); } [/code]

In this code, we watch for whether the evt.gamma is between -1 and 1. If so, they're looking at the ground. This is quite a precise spot on the ground, if you find it too small and difficult to trigger, you can increase the range to between -1.5 and 1.5... etc.

When they are looking in this range and when lookingAtGround is false, we run our theme switcher code. This adjusts currentTheme to the next index number of our themes array. We set lookingAtGround to true and set it back after 4 seconds. This ensures we only change the filter once every four seconds at most.

Continue reading %Filtering Reality with JavaScript and Google Cardboard%

by Patrick Catanzariti via SitePoint