Semi-Automatic Filmmaking: How to Use the Android App, Documatic, to Automatically Pre-edit Your Videos

Documatic is a participatory, android-based, mobile shooting assistant, and pre-editor. I have created this program to lay the basic groundwork for the larger concept of "semi-automatic filmmaking" as part of my Master's Thesis in Digital Media at Georgia Tech.

This instructable will teach you about how you can use this Android App, to streamline the production of your own films. The official project website is available at: 
http://www.semiautomaticfilmmaking.org/

The basic principles of Semi-Automatic Filmmaking are that, instead of gathering tons of footage with a video camera, and then later meticulously sorting and organizing in a separate editing stage, you can use an Android device to roughly annotate and organize the video clips AS THEY ARE BEING RECORDED. Then later, the system combines these annotations with the raw footage and intelligently organizes and "pre-edits" the footage into an Adobe Premiere Sequence. (This will all be explained in further detail in the later steps.

Here is a prototype documentary made with the Documatic system:



If you would like to read more about the theory and design that went into this program, please read my official design document available at:

http://dwig.lcc.gatech.edu/projects/Quitmeyer/site/papers/Quitmeyer_Andrew-Thesis_Design_Document.pdf


The actual Android application is freely available for download at:
https://market.android.com/details?id=andy.documatic&feature=search_result


***Please note that this product primarily serves as a proof of concept for a semi-automatic editing system. There are numerous bugs and flaws that are bound to pop-up, and the entire user-experience has not quite been professionally streamlined. For instance, the system should theoretically work for Final Cut Pro, but i have only been able to get it to work for Adobe Premiere.

On that note, if you are interested in helping me develop this program further to include features such as Multi-camera, or Multi-annotator support please email me at:

andrew.quitmeyer|at|gmail.com

1. an Android Device with the documatic app installed: 
https://market.android.com/details?id=andy.documatic&feature=search_result

2. A digital camcorder device that saves videos as individual files, like in a cameraphone, or video from a DSLR (note, miniDv probably won't work for this)

3. A computer with Adobe Premiere (or maybe Final Cut Pro) installed.

4. Java installed
http://www.java.com/en/download/index.jsp

5. Xuggler installed. Because of the way the current project generator functions, you will need to quickly download and install this free software library that quickly analyzes video files:
http://www.xuggle.com/xuggler/downloads/

***Skip this step, if you just want to get started***

Impetus: Give a digital hand-up to small budget and beginning filmmakers

The advent of digital video recording devices has opened the doors to many small scale filmmakers by greatly reducing the costs associated with capturing and manipulating moving images. Unfortunately however, the arduous production practices developed in analog filmmaking that are required to organize and edit a finalized film remain largely untouched. Thus, even though both small independent films and large studio productions can be now shot, pieced-together, and shared completely digitally, the studio is still able to create cinematic content more efficiently due to its ability to sort through and organize much more massive scales of collected content. Whereas a small documentary team may make take years to plan, shoot, and edit a 120 minute film, the sheer manpower of a large studio can produce a similar amount of content in a matter of weeks.

The tasks of funneling massive amounts of unorganized footage and continuously vacillating between capturing footage and re-structuring the project make documentaries one of the most frustrating media in which to work. The documentarian must not only organize the deluge of information, but also sculpt an engaging, concrete structure from its often changing pieces. The unique qualities that arise from its rigorous and dynamic production process, however, also help to define documentary from other types of film. 

Traditionally, they way in which the large studios dealt with the organization and structuring problems of motion pictures, was to distribute the filmmaking process into simpler tasks among dozens or hundreds of individuals. 

Unfortunately, small documentary groups lack the manpower to achieve these levels of efficiency. The digital automation and or parallelization of some of these simpler tasks of the studio's production process, could, however, replace some of these gaps in manpower. In this way, small documentary productions can begin to enjoy some of the filmmaking efficiency afforded to the big filmmakers, while maintaining the artisanal quality control of a small team. 

This project, Documatic, seeks to develop a system that combines the shooting/information-gathering process with editing/structural-synthesis to allow semi-automated production of concrete video stories following established documentary models. The end product will be more or less indistinguishable from a traditional, linear documentary film, but the new process leading to its creation will hopefully be simpler, more efficient, educational, and fun.

The Cinematic Model

To design a procedural model for filmmaking, I turned to established film theorists. In their book, Film Art, Bordwell and Thompson identify two primary types of documentary film, rhetorical form and categorical form (D. Bordwell, 2004, p. 132). Pure rhetorical form specifically strives to "persuade the audience to adopt an opinion" (Bordwell & Thompson, 2004, p. 140). The topic can be provable by scientific fact, such as a film detailing the process of mitosis, but empiricism is not necessary, such as why one should vote for a certain candidate. The and relies on emotional appeals, subject and viewer centered arguments, and arguments from seemingly reliable sources (Bordwell & Thompson, 2004, p. 142). Because the primary focus of the this type of documentary is simply to prove or illustrate a specific manner of thinking, the cinematic structure of a rhetorical film is subservient to its argumentative form. Since an argument can be structured in many different ways, the cinematic structure tends to be singular to a specific film. For example, the manner in which diagrams, interviews, narration, and other illustrative footage are arranged for a scientific film, may have no relation to the way in which these elements are arranged for a different political film.
Categorical documentary films, on the other hand, make less explicit arguments, and instead focus on simply conveying diverse information in an organized way about a particular subject matter. This format follows very simple, consistent pattern regardless topic. First, the subject is introduced, and then the viewer is presented with a series of interviews or narrations grouped into topics associated with the overall subject. Bordwell and Thompson's archetypical categorical documentary film, Gap-Toothed Women, presents this basic structure:
Title/Theme - Gap-Toothed Women
1. Introduction of a few gap toothed women
2. Genetic and Cultural Explanations for gaps
3. American Stigma
4. Careers and Creativity
5. Epilogue
6. Credits

(Bordwell & Thompson, 2004)

Thus Bordwell and Thompson's "categorical" film can be described by the following rule::
1) A CATEGORICAL DOCUMENTARY CONSISTS OF A SERIES OF TOPICS ABOUT A PARTICULAR SUBJECT

The order in which these sections are arranged, creates the overall narrative experienced by the viewer. Additionally, the temporal arrangement of these sections can be based upon external factors, such as the order that certain events happened in real life. Therefore, even films such as Morgan Spurlock's Super Size Me, can fit the categorical film model as it consists of a series of sections detailing portions of the fast food industry, while the arrangement of these sections describes the overall narrative concerning the transformation of the filmmaker's body.
Though Bordwell and Thompson warn that "because categorical form tends to develop in fairly simple ways it risks boring the spectator," they also remark on a strength of the form in that, "the categorical form can maintain interest by mixing in other kinds of form...[even] rhetorical form" (D. Bordwell, 2004, p. 134). Thus it is this simple, yet potentially powerful cinematic configuration upon which Documatic will be based.

DEEPER RULES: INTERVIEW, EXHIBIT, NARRATION
Using this categorical model to simply arrange footage into groups based is a good start for organizing the footage. One could imagine the use if, after filming several interviews, all of the sequences in which the interviewees spoke about "American Stigma" were automatically gathered together in one bin, and all the sequences about "Careers and Creativity". Automating this process alone, would already lift the burden of keeping track of many specific instances of time scattered across hours of footage. 

To supply an additional reprieve for the filmmaker, I also constructed a supplementary rule-set would be helpful to aid in actually editing the footage once it has been organized. This secondary rule-set breaks down an individual section of the categorical documentary into three fundamental footage elements: Interview, Exhibit, and Narration.

Interview footage
Interview footage serves as the bulk of the content for most categorical documentary films. It simply consists of video from a camera pointed at a person (typically a close-up head shot), who is describing or answering questions about something. A single recorded interview with a person is chopped apart into smaller interview clips, and these clips from different interviewees are grouped according to what was being said in each clip. These groupings by topic form the individual sections of the overall categorical documentary.

Exhibit footage
In order to make a documentary more engaging, it can be helpful to show the audience what the person being interviewed is talking about [insert quote from "Documentary Storytelling" (Bernard, Documentary Storytelling for Film and Videomakers, 2004) ]. This is a common feature of nearly any video interview. For instance, while an interviewee in Gap-Toothed Women delivers an anecdote about the characteristic "Gap-toothed" bit pattern that she would leave in food, a video image of that exact bite pattern is shown directly to the viewer. While the video from the interview is momentarily replaced with that of the apple close-up, the audio from the interviewee is not interrupted at all. This delivers the effect of simply visually illustrating what is being said. For Documatic, this type of footage can be optionally linked to a specific person, or piece of narration in addition to the topic with which all footage elements are associated.

Narration footage
Whether or not this final video element, narration, is incorporated into the project at all is up to the discretion of the director. Errol Morris's films, like The Fog of War, have little or no narration, and consist entirely of interviews and exhibit footage. Other films, like Hearts of Darkness: A Filmmaker's Apocalypse, utilize narration to smooth the links between sections of the documentary and establish a more solid narrative throughout.
In the context of Documatic, narration clips are left as optional elements which simply provide an Audio-only introduction to a particular section. Exhibit footage can be linked to a specific piece of narration, and automatically grouped with it during the editing process.

Thus, adding this deeper set of rules, to Bordwell's concept of a "categorical" film, results in this full rule-set that governs the output and production of a Documatic documentary:
1) A DOCUMATIC DOCUMENTARY CONSISTS OF A SERIES OF TOPICS (SECTIONS) ABOUT A PARTICULAR SUBJECT.
2) EACH SECTION CAN CONTAIN PIECES OF INTERVIEW, EXHIBIT, OR NARRATION PERTAINING TO ITS TOPIC .
3) AN INTERVIEW CLIP CAN BE OVERLAID WITH AN EXHIBIT SEGMENT OF FOOTAGE WHICH ILLUSTRATES WHAT IS BEING DESCRIBED IN THE INTERVIEW.
4) AN INTERVIEW CLIP CAN BE OVERLAID WITH TEXT OF FOOTAGE WHICH CONVEYS INFORMATION SUCH AS THE INTERVIEWEE'S NAME.
5) A NARRATION CLIP CAN INTRODUCE A SECTION AND BE OVERLAID WITH AN EXHIBITORY SEGMENT OF FOOTAGE WHICH ILLUSTRATES WHAT IS BEING DESCRIBED IN THE NARRATION.

By using these simple rules to and using them to guide the production process, and then applying them to how the Project Generator builds a video sequence, many steps of the filmmaking process can be automated, and a great deal of effort can saved by the filmmaker in terms of organization and structuring. Using a logical system has the additional benefit in that individual projects can be handed off to a third-party without additional guess-work or extra explanations.





Again, if you want to get deeper into the media theory which let to the development of this program, please check out my official design document at:
http://dwig.lcc.gatech.edu/projects/Quitmeyer/site/papers/Quitmeyer_Andrew-Thesis_Design_Document.pdf

Documatic divides the labor of traditional film-making into two simultaneous roles:

The Annotator, who uses an Android device running the Documatic app to annotate footage as it is being shot

and

The Recorder, who simply captures the pertinent audio and visual information

The Recorder's raw data is then paired with the semantic data coming from the Annotator, and this footage is automatically turned into a "pre-edited" Adobe Premiere sequence according to my program's cinematic model.

Throughout this instructable, I will detail how the first documatic documentary, "Long Dogs" was created.

To start producing your own Semi-Automatic Films, first download the Documatic App to your android device. Here is a download link:
https://market.android.com/details?id=andy.documatic&feature=search_result

Once you have the program, launch it, and it will ask you to create a new project.
In general, 
1 Project = 1 Movie

but the program allows you to merge and continue projects in a couple of different ways, so, for now, just make a title for your project.

In my example documentary, I titled my project, "Long Dogs."

In Documatic, the sections represent a chunk of time in the final documentary that refers to some particular theme. For a documentary, these sections can generally correspond to specific questions being answered

In my "Long Dogs" documentary example, some of these sections were, "Breeding," "Abilities" and "Cuteness"

Arranging
The order in which these sections are arranged (from top of the screen, to the bottom), specify the order in which their corresponding groups of pre-edited video clips will be arranged in the generated Adobe Premiere sequence.
For this reason, each project intially features an "Introduction" section at the beginning, and an "Ending" section at (you guessed it) the end!

If you create a new section and wish to change its order, simply grab the tab on the left of the section and drag it to its new desired location. In the example pictures I dragged


Deleting
If you enter a section that you do not like, simply press on the name of the section for a couple of seconds. It will turn white, and then a pop-up alert will ask you if you would like to delete it.
(CAREFUL! IF YOU DELETE A SECTION, THIS CANNOT BE UNDONE!)

The way in which the Documatic App is able to pair the annotations with the videos being recorded, is by comparing time-stamps. Thus to get your equipment in sync, you need to set the clock on your digital camcorder to match the clock on your android device.

*Note: you really want to get the camera and the Android device synced as closely as possible. TO THE EXACT SECOND is preferred. Since many android things don't really display the seconds on the clock, I use this a free analogue clock widget to view the exact second, and precisely sync up my camera:
https://market.android.com/details?id=de.dnsproject.clock_widget_main&feature=search_result

Generally the documentary production process for Documatic goes in this order:

1. Perform an Interview
2. Capture exhibit footage about that interview
3. Repeat steps 1 and 2 until you have collected enough content

4. If necessary, record narration to introduce the documentary or specific sections


You can, of course, collect footage in any order or fashion you would like!
In following the above pattern, though, the next steps will show you have to use the documatic system when filming.

When the recorder and annotator are ready for an interview, the annotator, (for the long dog documentary, Mariam), collects the subjects preliminary information, like their name and title or occupation. This is the information that will appear as a text caption over the video of the interviewee. In our sample case, the name is "Henry and Spot" and the subtitle is "Long Dog Enthusiasts."

While this is happening, the "recorder" (for the long dog documentary, Andy) frames up the subject and starts the camera recording. Then the annotator begins the interview by asking questions whenever she and the subject are ready.

At this point,  the recorder's job is very simple and relaxed. All he has to worry about is keeping the person roughly in the frame (and maybe monitoring audio levels). He does not have to worry about the interviewee's responses, and is thus able to get the highest quality footage possible.

Similarly, the annotator's job of conducting the interview is made easier through this division of labor. To provide real-time annotation of the footage being recorded by "the recorder"'s continuously recording camera, the "annotator" simply taps the tag in the list corresponding to the topic being discussed by the interviewee. If the subject, for instance, begins by discussing his relationship with the pet dog, but then immediately start talking about what factors affect how cute the dog is, the "annotator" simply taps the "Cuteness" menu item, and a the video being recording during this time is automatically categorized into the "Cuteness" section and linked to the subject, "Henry and Spot." While footage is being annotated, the theme of the user interface flashes bright red to indicate that virtual clips are being recorded.

During parts of the interview that the documentarian wishes to leave out of the final product (such as when she is asking the subject a question, or there is a lull in the conversation), the "annotator" presses the large, "Stop" button at the bottom of the interface. This returns the interface to the standard "Waiting to record..." color scheme. If something terribly important happened to occur while the "annotator"'s device was in this not-recording mode, the continuously recording camera will still capture the footage, it will just not be automatically included in the final project, and this missing segment will just have to be inserted manually. Thus no permanent damage can be done by the annotation system and further pressure is removed from the documentarians. Since the "annotator" is not faced with the worry of how the subject is being framed and captured, she is able to focus more on engaging the interviewee and getting the best overall interview. The act of tapping between different sections was minimally obtrusive, and was actually found to be helpful, as the list of sections serves as question prompts for the interviewer.

If the person being interviewed begins to discuss a topic outside of the pre-established sections, such as "Fur Color", the annotator can quickly add this new section and begin annotating.

Note that Interview footage is always tied to a specific person (in this case "Henry and Spot"). Each person interviewed on a project is stored, so that more footage can easily be collected from a person during continued interviews in the future. New people can also be added to the project's person database at any time.
 

After the interview is over, it is good practice to try to collect footage illustrating what was being described by the interviewee. In the Long Dog Documentary, for instance, this can mean getting close footage of the pet dog performing some of the abilities and attributes that were mentioned by the owner.

The collection of the interview footage
While Andy, the recorder, films close-up shots of the dog performing various activities, Mariam switches the interface to the "Exhibit" tab and starts annotating this footage as part of the "Abilities" section. Then, Mariam asks Andy to get an overhead shot of the dog, and she selects the "Dog Length" section. 

During this time, Mariam, the annotator, has the "Link Person" feature highlited in the Exhibit pane. This means that these collected exhibit clips will be grouped near the interview that was previously performed, instead of being paired with the narration for the section (as in the "Link Narration" case) or just being loaded at the end of a particular section, (as in the "No Link" case). 

Narration is an optional element where you can record audio (video is also captured), to introduce a particular section. 

Here is how it was used in the production of the example documentary, "Long Dogs" (Mariam is the annotator, and Andy is the recorder):


The duo continue to collect interviews and exhibit footage in this manner until they feel they have captured enough. Mariam did not want to have narration throughout the film, but did feel that there should be a simple narrated introduction that explained the purpose of the film. Now, while Andy records her voice, Mariam simply switches to the "Narration" tab and selects the "Introduction" section to begin recording a virtual clip. Since narration tends to be a much more rehearsed and scripted portion of a documentary, there exists the option to overwrite and restart a "Bad Take." This simply erases the previous virtual clip, and begins recording a new one for the same section. The purpose of this feature is to allow the documentarian to rehearse their narration live until they record a good take and not have to worry about sorting through lots of bad takes later. In order to collect some illustrative footage for this opening narration, Andy films establishing shots of people walking through the park with their dogs while, in the "Exhibit" tab Mariam selects the "Link Narration" option and starts recording a virtual clip in the "Introduction" section. 

Whether the team is prepared to produce finalized, distributable video, or they want to get a quick feel for how the video is coming together by viewing it edited together in a video editor, the post-production process is made simple with the Documatic system.

First, connect the android device to the editing computer, and copy the desired project folder from the Documatic directory on the device to the computer.

Documatic's file system is structured as follows:
ANDROID_DEVICE/
----- Documatic/
----- ----- - Readme.txt
----- ----- PROJECT_A/
----- ----- ----- - projectmanifest__PROJECT_A.xml
----- ----- ----- - Generate_Project.jar
----- ----- ----- video/
----- ----- ----- ----- RAWVIDEOFILE_536.MOV
----- ----- ----- ----- RAWVIDEOFILE_486.MOV


Each project folder contains all the information necessary to edit. share, and generate a single documentary. Each project folder has three primary components:
 
• a "projectmanifest__PROJECTNAME.xml" file which contains all of the virtual clip annotations arranged inside a structured xml tree
• a "Generate_Project.jar" file which, when clicked, executes the Project Generator program to automatically build a pre-edited sequence
• and a "video" directory which will hold all of the gathered footage.


Now, copy the collected footage from any of the cameras into the "video" directory located inside the project folder. This is the location where the Project Generator will search segments of raw video to find matches for each of the virtual clip annotations.

Because of the way the current project generator functions, you will need to quickly download and install this free software library:
http://www.xuggle.com/xuggler/downloads/

Then, once all the footage has been collected into the project folder's "video" directory,  all you have to do is double click the automatically included "Generate_Project.jar" file, and a new sequence is generated as a project readable by Adobe Premiere ( or, one day,  Final Cut Pro).


This file automatically appears in the main project folder and uses the naming convention, "PROJECTNAME__PreEdit.xml" (in our sample case it would be "Long_Dogs__PreEdit.xml").

Finally, drag this pre-edited sequence into Adobe Premiere to see how the project turned out, or to provide final sweep of editing or trimming to the clips.

Continuation
Sometimes you will want to continue a project by collecting more footage. With documatic this is simple. Just use the same project as before, and record and annotate more interviews, exhibits, or narration. Then, replace the "projectmanifest" file in the project directory on your desktop with the latest version from your phone, and also drop the new video footage into the project's "video" directory.

Now, click the "Generate Project.jar" and a new version of the pre-edited sequence will be created containing all the updated clips.


Merging / Collaborating
If you are unable to access the original project, or you want to combine the projects from multiple annotators, Documatic features a simple merging feature which can easily combine several projects.

Just dump all the collected video from all the projects into the same video directory inside the same project folder, and then copy all the different "projectmanifest" files into the same project folder. Then run the "Generate Project.jar" again.

Information will be taken from all the projectmanifest files (as long as each of these files still starts with the characters "projectmanifest"), and a new sequence will be generated from the merging of this information.

I have done my best to create a functional prototype for a semi-automatic film-making system for simple documentaries, but i need more help to really open up the system and make it more powerful and accessible by others!

If you would be interested in helping me develop this free, open-source project, please contact me at:
andrew.quitmeyer|at|gmail

or visit the project website at:
http://www.semiautomaticfilmmaking.org/



Here are some easy ways that the project could be expanded 

Since I left Documatic's underlying framework to be based off universal timestamps, there are several ways to extend the system to encompass a wide variety of additional features and uses.

Multi-Camera / Single Annotator
The current system places no limits or assumptions on the amount of video data files that could be referenced by a single annotation. It merely searches the "video" folder simple to turn this system into a massively multi-camera system. This is a scene from a Beastie Boy’s documentary where they gave everyone in the audience camcorders to film the show. You could have one director annotate the entire show, and the Documatic system could automatically synchronize, segment, and group this massive amount of raw video data directly into a digital editing timeline, and let you effortlessly browse between the various camera views.

Multi-Camera / Multi- Annotator
Growing even more enterprising, for large events, like a political protest you could have an indiscriminate amount of people, filming and annotating throughout the day. Then they could separately upload these video files, and time-stamped XML annotations to a central server, and interesting views of the day could be automatically generated representing individual or merged experiences from the group as a whole.

More procedural models
The most important feature of this product, is that we finally have a video production system that is actually digital all the way through. The digital documentary, is just one possibility, but by simply creating a procedural model of a different cinematic genre, you could easily beginning producing digital, semi-automated Sitcoms, Thrillers, or Dramas. In fact, we could create a procedural model creation system, where users generate their own rule-sets of custom complexity. The important thing here, is that we are treating video in the same way that a web developer treats documents of text and images. By pairing annotation with video, and forming intelligent rule sets, we can make the labor intensive video-editing process, as simple as changing Wordpress templates.

Model Creation system
In extending the project to harnessing multiple procedural models, the most helpful tool would be a user accessible Model Creation system. That is, when they begin a new project, instead of just specifying a project name, they would also choose what procedural model this project would follow, or they could have the option of designing a new one on the fly. Instead of working with the same elements of "Interview" "Exhibit" and "Narration" used by the Documatic model, the interactor could have the option of designing their own elements and specifying rule-sets for each.
Standardized, Searchable Video Depositories
If the amount of documentarians embracing the Documatic system reaches high enough levels, repositories could be established where video producers could share and search for annotated footage.