was a kid and we went to the movies my dad would say, "Let’s sit towards the
back. I want to watch the movie not be in it!” My how times have changed.
Welcome to the world of immersive entertainment and welcome to "Live Action VR 101.”
Producing for VR is a virtual hack-a-thon right now but hopefully this primer
will give you some insights into the production process, the technology and the
workflow. Imagine if you will what filmmakers did when the first motion picture
cameras came out. First, they learned the technology and then they brought
artistry to the medium. This is exactly what is happening with VR. As with any
new technology there is a "tail wagging the dog” tendency and when that happens
the result is bad VR. As any great producer knows, the technology needs to
serve the story.
The greatest strides in the art of what I
prefer to call "immersive narrative” (also called cinematic VR) are being made
by independent filmmakers who "hack” the technology - from cameras and lenses
to sound and light- in order to create the story-driven experience they know
will move the immersed viewer. Today’s hacking is tomorrow’s automation and
these filmmakers are informing the evolution of VR production tech.
As storytellers, we want
to emotionally impact the viewer and even leave a lasting impression. Virtual
Reality offers an unprecedented opportunity to do exactly that, sometimes in
extraordinary ways. In what I consider to be the fastest moving industry ever,
we are now transitioning from the wow of an immersive experience to the
compelling content and artistry of narrative.
Getting there involves experimentation, hacking and learning a new
language of cinema that borrows from traditional film, video games and believe
it or not theater. Consider this brief a snap shot of where production is
will need a lot more moving up to professional high quality VR filmmaking.
Right now it’s the Wild West. Something new seems to come out every day. Part
of my job as a producer is staying on top of all of it! Processes that took a
lot of patience, time and resources just a year ago are now taken care of with
software. Examples: A year ago, synching a 16 Camera GoPro array to shoot in
360 meant a single Wi-Fi remote, a handclap for sound and a shot sync. Timecode
Systems just released SyncBac at this year’s NAB Show. It allows multicam sync
via a time code generator that fits on the back of GoPro Silver and Black
There’s a timesaver in
post! Also at NAB Show, Deep, Inc. launched Liquid Cinema, which allows for
"forced perspective” allowing the content creator to direct the audience’s
attention inside the VR experience at any time as well as live rendering
graphics and even some visual effects. These features are big leaps in VR
production. Finally, when it comes to workflow and stitching, meaning putting
the output from multiple cameras together into a spherical image, more tools
are coming out all the time from companies competing to be the gold standard
like Video Stitch, GoPro’s Kolor, Jaunt and Ozo.
Story first. There are great differences in
approach for production in immersive journalism, live streaming, sports,
documentary and animation. For our purposes, we are going to be considering narrative
drama or comedy. Remember, you are working with an entirely different medium
that actually can fool the brain and the body physically and emotionally into
what is called "presence.” This places the viewer in the environment that the
characters are experiencing, can allow the viewer to stand in the shoes of a
specific character, or even change characters. Sight, sound and light as well
as superb acting will allow you to create a first person experience. The what of good storytelling does not change. You
still need a great story with the elements of character, time and place, canon
and a story arc. It is the how that changes and to master the currently evolving how is
like working with mercury. The best advice, really learn the medium and the
tools. A good rule of thumb is that if you can tell the story in traditional 2D
film then that’s what you should do. If your story can only be told in virtual reality using and pushing the limits of the
medium then in my humble opinion, you’re on the right track. You’re going to
learn a lot but you are also going to shape a new and exciting new storytelling
medium. It’s a green field right now with lots of room for experimentation.
But, as far as consumers go, we should only release our best in VR.
Writing for a 360
environment is revolutionary. For example instead of just setting a descriptor
for the movie scene in front of the viewer, or describing shots, you are
setting the scene and action around the viewer. You are also blocking the camera and
positioning the actors in relation to the camera and each other in the round,
which figures prominently into your narrative and character relationships even
when nothing is being said. Depending on the number of cameras being used
you’ll have multiple "stitch lines.” These are areas where the images captured
by the cameras overlap. As example, if your actors are too close to the camera
and on the stitch line you may have an uneven match that splits their face
right down the middle in a Picassoesque way. Not so easy to fix in post.
Personally, I write to a
circular blocking form. It’s literally a pie chart with each segment
representing a camera lens where scene descriptions, actions and dialogue are
written in the inside of each segment. You are also deciding where and when to
direct the viewer’s attention in a manner that serves the immersive story. The
form even allows me to create transition shots easily understood by the
director. It’s my contention however, that the writer is part writer, part
director, part cinematographer and part sound designer when scripting VR. It’s
exceedingly helpful if you have a background in gaming or theater. Years ago
while touring with "Fiddler on the Roof,” Theodore Bikel taught me how to "act
with your back” for theater in the round. Who knew that would come in so handy?
As you might guess, some
new professions are immerging including immersive sound designers and directors
of virtual photography (DVP) as well as specializations in stitching and
The Difference Between
360 and VR
On the consumer distribution side, you’ll
hear these terms used together or interchangeably but many purists in the
industry will tell you they are not the same. While most VR is 360 not all 360
is VR. Think of 360 as something at arms length and VR as something in which
the audience is immersed. Thanks to relatively inexpensive consumer cameras
that shoot in 360, and the world of advertising, 360-video is surging. You can
see it on Facebook, YouTube, Vrideo and Littlstar and thanks to Google’s "Magic
Window” technology released last month, anyone can iframe, i.e. embed 360 video
into any web or mobile site. On desktop you use your mouse to navigate to
reposition the spherical video and on mobile you use your finger to slide
around the scene or you turn your body with your phone. You’ll see a lot more
360 commercials coming. Some say 360 is the gateway to immersive experiences.
But for VR you need some sort of Head Mounted Display (HMD). Even Samsung has
acknowledged this migration path and now offers 360 MilkVR content in a mobile
app for specific Android phones compatible with their GearVR HMD. If you tap
the on screen headset icon you are prompted to place your phone in the GearVR
and voila!, you are in an immersive experience. With other 360 mobile vids you
tap the Google Cardboard icon and drop the phone into your Google Cardboard or
other stereoscopic viewer and the image splits for stereoscopic view.
You can imagine how
confusing this is getting for the consumer who thinks they are watching VR when
they are not.
The "immersion continuum”
illustrates the degrees of immersion from 360 to consumer HMD’s right on up to
the fully immersive and interactive Oculus Rift, HTC Vive and Sony PlayStation
VR and ultimately 3D/4D dark rides and VR domes. You can of course create
content across the entire continuum or even scale what you create for one to
work on all but it’s best to decide what your anchor platform will be. Here we
will be talking about Virtual Reality production for HMD’s.
You have choices as to what shooting format
you will use.
360 2D. This is exactly what it
says. An image that surrounds, well that is really spherical but in 2D. Nothing
leaps off the screen or beckons you to interact. This is not truly cinematic
VR. It is 360 video in a VR HMD. Your viewer is immersed and can see all around
and depending on camera rig can see in detail. This is the least expensive
option depending on the cameras you choose. More on that when we get to camera
rigs which by the way can contain as many as 24 cameras.
360 Stereoscopic VR. This is more how our eyes
see, is in 3D and closer to reality. For every one camera that you would have
in 2D you will now have two cameras. This creates greater presence and has more
Advanced computational photography is used here and creates superior VR and is
as close to reality as we can get right now except for touch and smell.
Now that you have your story, it’s time to
look at the best way to film it. Flexibility is in order as is research.
Choosing the right camera and the right rig is as much of an art as the filming
itself. We do a lot of camera tests to check performance in relation to what we
want to accomplish. In a recent production meeting on a new project the
director said "Wouldn’t it be great if we could do this?” He was ready to hack a camera and lenses but we also knew of
a camera in development. Days later we talked to the company and we’ll have
that camera by July. My motto however is "if you can’t find it, hack it!”
Video and image get a lot of attention in VR
but sound is just as important to get right. Most onboard camera microphones
will not allow you the flexibility that you’ll need to create object oriented
spatial audio that captures a realistic sound field. This is where Ambisonic
microphones and field set microphones come in. Ambisonic mics record a 360
sound field that can then be decoded into 7.1, 5.1 or the binaural sound used
in VR experiences. This allows the viewer to be immersed in sound as well as
visual circumstances. Some VR sound designers are placing microphones not just
on actors but also around the area of the sound filed and then editing into a
360 immersive environment. Startup "Two Big Ears” has launched 3Dception for
both cinematic VR and Games that is a well-reviewed end-to-end solution for
From Shoot to Distribution
When producing for film you know what your
delivery format will be and you leave transcoding to the distribution folks.
But as mentioned before, you’ll need to take that immersion continuum into
Depending upon what
cameras you have decided to use your tasks may change, as may aspects of your
workflow. Preparation will make all the difference and save you a lot of time
in post. For our purposes let’s just say you are using a GoPro Odyssey 16
camera rig. Briefly, to prepare, you will need to number all your cameras as
well as numbering the SD Cards that you will be using. This is so you can keep
track of the cameras and the shots when you are stitching them together in a
spherical image. With this rig you probably won’t want to run on the in-camera
batteries but rather get a large battery to power all of them. And of course,
use the new SyncBac product for synchronization. You’ll need storage in the
field, at least 2 TB(terabytes) Think of the math. You have 16 GoPro’s each
with a 64gigabyte SD card. That’s about 2 hours of high-resolution (1080/4K)
high frame rate (120fps/30fps).
Rehearse Rehearse Rehearse
This is just not rehearsal for your actors,
which is vitally important considering that they will be in a long take but
also that acting for VR is different. It’s whole body and wholly authentic. A
bit like acting in theater in the round but intensely intimate. More than a few
potentially great VR pieces have been destroyed by bad acting. You and your
crew including camera, lighting and sound must also be well rehearsed. I
suggest using a consumer 360 camera or the new Orah 4i during rehearsal so that
you can use playback to work with the crew and the actors. Where you place the
camera in relation to your actors is important especially if you want to
maintain the correct perspective. If your viewer is looking directly into the
eyes of your character you don’t want the character’s head to be twice as small
as the viewer’s.
Contrary to popular belief
you don’t just plunk a 360 cam down and act around it. A year ago no one wanted
the camera to move now there’s camera blocking, transition shots and moving
cameras. There are single POV stories, multiple POV and moving from 2D into 3D
stereoscopic. The art form is developing. Having witnessed brilliant
independent VR filmmaking, I’d say we are only limited by our imaginations, not
After the shoot, it’s time to get all your ducks i.e. SD cards in
a row. Fortunately you have mapped the cameras and numbered them as well as the
SD cards so you know the shots and where they came from and now with SyncBac
they are timecoded. And, fortunately there is some great software to help with
file management. CamMan from 360Heros has a great intuitive interface and there
is also ManyCam, which is a bit simpler. This may save you a bunch of renaming
and reorganizing files but you still
need to be vigilant and attentive to make sure you get it all right.
Now that your files are set you can move on
to stitching the shots into a spherical video. Some consumer and professional
cameras have "on-board” stitching. As a matter of fact, if you are using the
GoPro Odyssey you can use it in combination with Google Jump, which automates
the workflow. But, if you want to be more hands on you can use GoPro’s Kolor or
VideoStitch. Now it’s time to export into editing.
Once you’re all stitched
up you’re ready to edit! There are choices here too. You want to be able to
output to Spherical or Cubic Monoscopic (for the web and mobile 360) and
Stereoscopic VR for specific or all HMD’s. Your choice of course depends on
your distribution and with these software programs you can specify the output
device. There are choices here too. One of them is Mettle’s Skybox, which has a
nice suite of tools for 360/VR editing for Adobe Premier Pro. Think After
Effects for VR- you can tilt, pan and roll the sphere and edit to have an even
horizon line, direct the scene start field of vision, correct the stitch line
seams and much more, all key frame editable. Maxon’s Cinema 4D also has a great
VR Workflow that works with Adobe Premier and After Effects. You can learn more
about Maxon’s Cinema 4D workflow at cineversity.com. Finally, Dashwood’s 360VR
Toolbox allows for in HMD VR editing and monitoring as a plug-in suite for
Adobe Premier Pro, After Effects and Final Cut Pro.
Sound Editing and Mixing
We spoke earlier about Two Big Ears
end-to-end spatial audio solution 3Dception but you can also visit
Ambisonic.net to learn more about the approach to spatial audio.
Live action VR production
is both challenging and rewarding. Best practices are emerging. Courses are
being taught in universities. Hardware and software solutions are empowering
filmmakers to be more creative. My advice, get inside an HMD (GearVR, HTC Vive,
Oculus Rift and soon Sony Playstation VR) and watch as much content as you can
as any film student would. Know what can be done and then hack and push the
This is intended as a
fast overview rather than a dive into the technical weeds. But if you want to
get started in the basic A.B.C’s of VR my recommendation is 360Heros.com.
Michael Kintner, CEO and founder of 360Heros has made getting started in VR
easy with a ton of guidance and products. The VR starter kit will have you off
and running and you will be joining an open and informed community of VR
enthusiasts and professionals. 360Heros offers workflow solutions and hits
every point on the content value chain even offering storage and distribution
for your work. Working at the basic level you’ll gain a greater understanding
of filming, editing, and producing in VR. If you’re in Los Angeles you can even
take their VR classes at Village Workspaces in West LA. More cities are coming.
Your choice of rig depends on many factors
including the format as described above, the story you want to tell, how you
want to tell it and your budget. Some vendors offer end-to-end solutions
including image stitching and editing. The cost range can go from say $2,000 up
to $60,000+. There are also of course consumer cameras such as the Ricoh Theta
and forthcoming LG 360 Cam that shoot in 2D with onboard stitching for under
$500. Some basic things you need to know if you’re up to shooting higher
quality VR. Define your criteria for camera selection.
the camera(s) that best serves your story and the environment in which you are
shooting. Decide how many cameras you want to use based on desired coverage,
final resolution and of course budget.
be shooting "oners” or long takes. While there is "framing” of shots in VR it
is not shot framing in the traditional sense. Length of scene, lighting,
environmental conditions- all matter.
will have to use natural light, sometimes low light or light from the outside
in in a studio setting. You cannot light like you would a traditional movie.
(Nor, can you have crew standing about- everyone has to hide!)
need to shoot in at least 4K and at least 60fps for top quality VR.
don’t always need to shoot in 360 sometimes 180 will serve your story.
lenses you choose will determine your field of view and coverage (as will the
number of cameras).
must be synched with each other and all settings must be the same. You cannot
mix and match cameras in a rig.
power and cooling can be an issue.
higher the resolution the more data which means bigger files.
more cameras the more data.
- Monitoring your shot isn’t easy. Some
cameras like the OZO have real-time monitoring. Many filmmakers are hacking
solutions together. Recently the Orah4i camera launched for live 360 streaming
and filmmakers are actually using it in addition to their camera rigs for
previz and monitoring. Incidentally, previz is highly recommended
----Brian Seth Hurst is Co-Founder and Chief
Storyteller of StoryTech where he works with networks, studios and storytellers
in all aspects of VR from strategy and packaging to production and
distribution. He is also creator and executive producer of the forthcoming
"determiNATION” from Bunim-Murray productions