Artist Marius Watz created images like this one for the interactive documentary Clouds, which was featured at the 2014 Tribeca Storyscapes exhibition in New York City. Image courtesy of James George. Photo: Marius Watz
Clouds, the immersive documentary about data, was named best of this year’s Storyscapes exhibition of interactive film at the Tribeca Film Festival. For their Storyscapes installation, project directors Jonathan Minard and James George created an interactive journey through a series of questions and answers by data artists and experts, visually represented by data-driven graphics. The work is viewable either with an Oculus Rift virtual reality headset, or as an installation that a viewer controls with their gestures as sensed by a Microsoft Kinect camera — think Tom Cruise in Minority Report. I spoke with the creators about how their project, originally conceived as a traditional documentary, evolved into an award-winning interactive media project where almost every aspect is created by or experienced through computer code.
POV: How do you describe your project to people with no technical background?
James George: I think everybody can relate to the idea of a documentary about a community of artists, but in our case the artists’ craft is code. It’s like they can draw through code, and the doc is about what that looks like and what topics they’re engaging with. In software you can model parts of our world, but a lot of the artists in the film are inspired by the idea that you can create your own new world through coding. The software stems from a scientific process but they use it in an aesthetically pleasing way that often has surprising results.
Experiencing the film is about traveling through space and discovering things. The concept is that it’s the same as how you can control an actual conversation through the questions you ask. When you’re experiencing the documentary, the questions you follow dictate your path and along the way you encounter different ideas and see different creative works.
POV: At what point did you know that this would be an interactive film, and how did that change your approach?
Jonathan Minard: We didn’t even set out to make an interactive film, but there was a practical dimension to it. It’s a complex subject and we spoke to 45 people, each of whom had something interesting to say about it. We didn’t want to edit it all down and cut good stuff. We wanted to let audiences follow their curiosity.
James George: We decided to go this route when we were already 50-75% done with interviews. We had just released a short version, Clouds Beta, and there was a lot of footage left. We had alternate edits and when we strung it all together it was dull, but when we showed it to different people with different interests, they responded to different parts. One of our artists, Karsten Schmidt, suggested that if we thought of it like a database, instead of like a video, we could generate a conversation and have an infinite number of films.
That began to change our thinking, and it changed our next interviews because we allowed them to be much longer. Our story is set of themes, a collection of ideas that reflect what a certain community cares about. Once we started figuring this out, we used these themes as a basis for conducting other interviews and then when we had a gap in themes, we used it to figure out who to interview next to fill the gap.
POV: How did you develop a storyline if you knew that the movie-watching experience was ultimately going to be controlled by the user?
Jonathan Minard: We created a story engine by marking the responses to all of our questions within Final Cut, and then generating a mind map out of them. We didn’t even really know what our story was until we did this tagging and saw the common themes. That same story engine is running the interactive experience, but the idea of marking themes could be used by any traditional documentary filmmaker to make sense of their interviews and put their stories together.
James George: It was kind of a hacked process. A huge part of Clouds is invisible. The tagging process takes a piece of code that reads the markers in a Final Cut Pro file and creates a list, then we can discover clips across all the interviews that share keywords. It’s a discovery tool. There is no traditional edit of the film. We are never going to make an edit. Every time a viewer asks a question, the code generates 5-7 minutes of footage that is tagged with certain keywords. That’s why it’s unpredictable and the viewing experience is never the same twice. The code makes connections that we didn’t make ourselves when we marked the clips.
POV: Code controls the content, but the visuals are also created by code. What does that mean and how does it work?
Jonathan Minard: We never would have thought that the documentary would expand to include work created by code, but it made sense to use code to visualize the interviewees’ work. We wanted to use the medium that the artists we spoke to use.
James George: The algorithms are just as much the subject of the documentary as the people. We wanted to show that they are expressions of creativity. What you’re seeing was generated by very specific technical words, like someone wrote “draw a line from here to here.” There are a lot of different kinds of code that do different things, like run your banking system. This code is a graphics description language. By accumulating these words, you can create really complex images. This community, featured in the film, is using language that video game designers use, but they’re using it to create art.
Why the way we do it is meaningful and different from computer-generated graphics like in Pixar films is because it is responsive. The computer is sitting in the same room as you and the images change in real time according to what you do. It’s like difference between seeing a movie and a play at the theater. The work will never look exactly like that again, and that’s important to the artists.
POV: What do you hope people will take away from the Clouds experience?
James George: Universally my desire for the viewer is more from an emotional place — wandering, feeling lost, searching, learning. I want that experience to be evocative and incite curiosity. And then, hopefully, if people have no previous exposure to the ideas in the film, they will find a meaningful connection from their own experience as a doorway into it and that will start a path and create new connections while they are still having a pleasurable cinematic experience.
Get more documentary film news and features: Subscribe to POV’s documentary blog, like POV on Facebook or follow us on Twitter @povdocs!
