View the project » (Chrome required)
View the source code »
We at POV recently released a series of interactive documentaries that use technologies like HTML5 video to tell nonfiction stories in new ways. One of these documentaries, Empire, explores the long-term, unintended consequences of Dutch colonialism. The piece, directed by Kel O’Neill and Eline Jongsma and coded primarily by Genevieve Hoffman, began as a small prototype, hacked together under the time pressure of a POV Hackathon with other technologists, and the team built it into an ambitious and fully functional interactive experience. I had the opportunity to help out in the final stages with some of the technical challenges of deploying a media-heavy piece like this on the web.
Kel and Eline wanted to push the browser far enough to get their vision across without losing speed and reliability, so Empire was built entirely on standard web technologies: HTML, CSS, SVG and Javascript. The design, partially inspired by a previous video installation, required a lot of screen space, so development was restricted to desktop devices.
In interactive pieces like Empire, the viewer’s browser is essentially performing the work of a post-production suite, rendering a movie in real time over an unreliable, remote network connection. So small quirks in browser implementations or device limitations can become major stumbling blocks in the development and debugging process.
Pre-Loading Video, But Which One?
The first big challenge was getting all of the media components to load reliably and quickly. Empire includes eight video files, two audio files and many images, totalling hundreds of megabytes. With this much media to load, it became necessary to balance the need for a fast start-up time with the need to minimize buffering once a video has started to play. Empire consists of multiple chapters that a user can switch between at any time, so it is impossible to predict which media elements will be played next. It sounds tempting to pre-load everything, but browsers aren’t able to load many large files simultaneously. And web developers have limited control of how and when the browser loads video data from the network. For the most part, how much data to load is decided by the browser, which guesses how much you’re going to need, and the algorithm is different for each browser.
We found the best solution was to pre-load just the “metadata” of all the media files (the header of a media file that contains the duration, the pixel dimensions and enough information for the browser to know if it can play it) and then start loading the full file once the user selected a chapter. With only a few files to load at a time, we see that the start-up time is not more than a few seconds and there is minimal pausing to buffer, even over a mobile 3G connection.
The video and audio tags have an attribute called preload that we set to metadata so the browser knows not to start loading anything beyond that until we tell it to. Our video tag looks like this:
<video id="video1" preload="metadata"> <source src="videos/video1.mp4" type="video/mp4"/> <source src="videos/video1.webm" type="video/webm"/></video>
Once the user has chosen a chapter and we’re ready to start loading the rest of the video, we use the .load() method in Javascript like this:
var video1 = document.getElementById('video1');video1.load();From this point, the loading process is beyond our control as the browser progressively loads as much of the video as it thinks we need based on whether the video is actually playing and how fast the data comes over the network.
The Empire Framework
There was also a challenge in architecting the project code to manage the many asynchronous Javascript events fired in the course of running the interactive experience.
Keeping track of the state of things can get quite complicated. Certain components may be visible at different times, each of the media elements load in different ways, and there are many elements that the user can select at any time. There is no guarantee that these events will ever happen in the same order twice, and we needed to be sure that images, video and audio were only visible and audible when they’re supposed to be.
There are many Javascript frameworks for handling this sort of thing (e.g. Ember, Angular and ReactJS), but they come with steep learning curves and require build tools that can slow the development process. Also, they weren’t designed with audio and video in mind.
Genevieve and I created a small, ad-hoc framework of our own. Each of the four chapters is implemented in its own Javascript file, with all its local variables inside a closure, eliminating the risk of a chapter overwriting the variables of another. Each of the four scripts returns a single Javascript object with common “methods” (another name for Javascript functions when they are attached to objects), and each can carry out the same sets of tasks: “initialize” the action and page elements; “activate” the chapter by making it visible and start playing media; “deactivate” the chapter by pausing media and hiding all the elements; and “sizer,” which resizes each video.
By using common Javascript methods, we were able to keep the main code that switches between the chapters relatively simple, and it is makes it possible to work on the chapters independently. This makes life easier for one or two programmers to wrap their heads around the code, and it’s even more important for larger teams.
View the project » (Chrome required)
View the source code »
In my next post, I’ll share code for a key part of Empire’s “Migrants” chapter, in which we enabled synchronized video playback for shared viewing experiences.
Post a comment below if you’ve faced these same problems and how you addressed them. What other challenges regularly pop up when building interactive video?
The book “Empire: The Unintended Consequences of Dutch Colonialism” is available in the Empire store at jongsmaoneill.com. Get more documentary film news and features: Subscribe to POV’s documentary blog, like POV on Facebook or follow us on Twitter @povdocs.