Bringing 6DoF to Live Recorded Audio and Video

By Piotr Szczechowiak (COO, Zylia) and Tomasz Żernicki (CTO, Zylia)

Imagine a popular music festival you’ve always wanted to experience in person. For one reason or another, you can’t make it to the festival. But what if you could plug into an immersive experience that allows you to move freely through the entire audio soundscape and visual landscape? This is the promise of Six Degrees of Freedom (6DoF).
  Until very recently, immersive audio and video were limited to just Three Degrees of Freedom (3DoF). Examples include the 360° videos now available on platforms including YouTube, Facebook, and Vimeo. Sometimes also referred to as immersive videos or spherical videos, these 360°-video offerings incorporate video recordings shot simultaneously using either an omnidirectional camera or a collection of cameras.

Images are stitched together to build a panoramic scene, which can be viewed on a laptop or mobile device, on a head-mounted display, or within a purpose-built space equipped with projectors or displays in a spherical configuration. While people viewing or listening to a 360° or 3DoF scene can rotate to look in different directions or to hear audio from different direction, they themselves remain static—they cannot freely translate their position in space.

The 360° audio is similarly recorded. Algorithms within associated software convert the signals from the recording device into the Ambisonics domain. Ambisonics is the full-sphere surround sound format that has become the standard for 360° filmmakers and video game developers, and it covers not only the horizontal plane, but also sources “above” and “below” the listener (see Figure 2).

Unlike other multichannel surround formats, the Ambisonics format carries speaker-independent representations of a sound field called B-format over its transmission channels. The sound field then can be decoded for playback on the target system or device. This approach gives creatives and engineers the flexibility to design sound in terms of source directions rather than loudspeaker positions. Photo above shows Zylia, applying third-order Ambisonics ZM-1 microphone arrays with Infineon’s digital MEMS microphones, has an innovative and unique approach to Six Degrees of Freedom (6DoF) technology in virtual and augmented reality (VR/AR).

This article was originally published in audioXpress, October 2019.

Read the complete article now available here.

About the Authors
Piotr Szczechowiak is the Co-founder and COO of Zylia. He manages the day-today operations. Piotr has a PhD in embedded systems and makes sure that ZYLIA hardware and software fits together. 
Tomasz Zernicki is the Co-founder and CTO. He manages the technical operations. Tomasz received a PhD in digital audio processing and is an audio expert in the ISO/IEC MPEG standardization group.