360 degree video is an immersive media format that allows for an interactive virtual reality experience.  The term "360 cam" is a term used to describe any camera capable of capturing an approximate 360° panoramic or spherical view.  A web search using the term "360 cam" returns a number of interesting results, revealing a wide array of 360 hardware devices and software applications.

 

360 cameras continue to trend strongly on social media.  A quick search of social media for "360 cam" will give you a wide variety 360 degree panoramic videos from all over the world.  Few technologies have come along that allow people to share their experiences in such an immersive, interactive way.  This technology sector is growing rapidly now and should experience significant growth over the next five years in conjunction VR headset sales growth and the introduction of Augmented Reality (AR)  products.

360 cam technology overview:

The 360 Omnidirectional camera:

 

The word “omni” is a combining form meaning all or everywhere, from the Latin omnis for “all”. The word’s meaning in relation to photography or video is a camera capable of an approximate 360° field of view which results in a panoramic, wide angle image of the entire visual sphere.

 

Normal cameras capture at most a 180° semi-sphere of light from the focal point.  360 cams on the other hand usually use a dual lens design, with two ultra-wide-angle fisheye lenses, concentrating light via two 90° prisms to divide the light between left and right image sensors.  Some 360 cams use 16 different cameras mounted in a circular array.  However dual lens systems greatly minimize parallax error by shortening the distance between the lenses.

 

In order to produce a seamless spherical image from images shot simultaneously via two or more lenses the images need to be “stitched” or dynamically processed in real time. The processing engine is responsible for this operation, calculating the amount of shift among the images using pattern matching techniques.  The image processor powering the processing engine needs to be robust enough to handle large volumes of data in real time.  360 video files can be 4K and higher, at bit rates that can be over 50 Mb per second or about 22 GB per hour of footage. 3D 360 stereo videos are twice that at about 44 GB for an hour of footage.

 

 So you need a lot of processing power and a compression algorithm that can compress well without giving up much in terms of image quality.  One thing that helps image processing enormously is to start with two images that have high peripheral image quality and very little parallax error.  Video output is typically H.264/MPEG-4 and photo output is usually JPEG.

 

The ability of current digital image processors to process large volumes of data on board the camera also allows for virtual pan, tilt and zoom over the entire 360 degree panoramic image space, something that in the past required multiple cameras and often resulted in dropped, ghost or double images.  This is a feature often seen in modern IP surveillance cameras.

 

Many smartphones contain magnetometers and it stands to reason that 360 cams and VR headsets will contain an Inertial Measurement Unit (IMU), that is a 3-axis gyroscope, 3-axis accelerometer and 3-axis magnetometer. The IMU allows for moving or swiping in any direction and an overall more immersive experience. Note that some cameras now feature six-axis gyro stabilization.

 

The two most popular types of Fisheye lenses are Circular and Full frame.  Circular Fisheye lenses will capture an approximate 180° circular view angle, but not the whole area of the frame.  Focal length varies between 5.4 - 9mm.  Full frame lenses will capture an image which corresponds to a 180° view angle within the diagonals of the whole area of the frame.  Focal length will normally be between 14 -17mm.

 

360 spherical image formats:

 

There are a number of full and partial spherical formats which are often categorized by the type of projection used to map 3D content onto a 2D screen.

 

The Equirectangular and Cubic full spherical formats are most common.  These formats can display the surrounding environment about 360° along the horizon and 90° vertically. The Equirectangular format uses a single image frame with a 2:1 aspect ratio.

 

The Cubic format consists of six cube faces which will translate into one  whole spherical image.  The cube faces are either compressed into a single JPEG or are expressed as a single row or column with a 6:1 aspect ratio, which are then combined to create the sphere.

 

One odd format known as “Little Planet” projects an entire sphere in such a way that it looks like a tiny little planet.  The result is a 360° Fisheye image where the nadir is in the center and the zenith is on the outside border.

 

There are partial panoramic formats which do not fill the entire sphere.   The two well known partial formats are Cylindrical and Rectilinear.  Partial formats typically have a short side of 120°.  

 

The Cylindrical format is often used to display landscape panoramas due to the format’s ability to display ~360° along the horizon.

 

Rectilinear panoramas have horizontal and vertical fields of view which are ~120°  which produces something of a tunnel vision but is often optimal for the interior of buildings such as long hallways.   Other partial panoramas will limit the Zenith and Nadir to present a wider look.

 

When filming 360° wide landscape panoramas it is important to maintain focus, including both background and foreground, from Zenith to Nadir. This is optimally done in manual focus mode, if possible, with optimal Hyperfocal distance, which is the distance between a camera lens and the closest object that is in focus when the lens is focused at infinity.

 

Lately there has been some confusion among people online between 360 cameras and 3D cameras. 3D cameras capture image data from a single angle as opposed to capturing image data from an approximate 360 degree spectrum as with 360 cameras. As a result, 360 camera output feels more intimate and immersive.

 

3D cameras have at least two lens each with their own image sensor. These lenses capture images and combine them in such a way that it simulates depth, the elusive 3rd dimension. This is similar to how human eyes and the brain work together to create depth perception.

 

Both 3D and 360 image data can be used as source material for VR (virtual reality). However 3D will require more work on the software side to stitch things up than 360 images.

 

Recently some innovative capabilities have appeared in 360 cameras, largely driven by software or algorithmic upgrades. Perhaps the most exciting is the idea that you’ll never have to worry about “getting the shot” again. In other words, after seamlessly capturing everything around you, with the output being a fully stitched, stabilized 360 sphere, new software enhancements enable reframe, pan, zoom, speed up, clip, crop and object tracking via an app, all after the shot.

Essentially these new capabilities mean that you don’t have to think as much about shot composition, rule of thirds, golden ratios and such, beforehand. It’s all captured in the 360 sphere. You frame your perfect shot in post, moving around inside the sphere, with the option to save the output to full HD 1080p. It’s like being able to “direct” a shot after it has been taken, allowing users to shoot away and make creative decisions later. Could this be the killer app of 360?

To expand on object tracking, this is a feature lets you follow an object or subject, keeping it in frame even if the camera or the subject moves. Tracking makes it easy to create a “drive by” effect where the focus remains on the subject as the camera moves by.

Picture-in-picture or side-by-side displays is another feature working its way into 360 cameras, making content such as tutorials and the two sides of a conversation easier than ever to produce.

Smooth panning can be achieved after the shot has been rendered into a 360 sphere by choosing specific spots or points and linking them, similar to how key frames are used in animation. The pan then flows from point to point. How cool is that?

It’s about time for “bullet time”: a visual effect where time and space appear to slow down dramatically, similar to the slow motion flying bullets in the Matrix movies. When the Matrix was filmed hundreds of cameras circling Neo were fired off in sequence and then every still image was interpolated to create the effect. The resulting shot was amazing but also very complex and expensive.

Now the bullet time effect is starting to make its way in the 360 camera world. Although not nearly as professional and polished as in the film industry, it is probably good enough for retail consumers while being orders of magnitude less expensive.

How does bullet time work with 360 cameras? Currently by using an attached selfie stick or cord and rotating the camera around at 120 fps. Software can “erase” the accessory used to rotate the camera giving the impression that the camera is flying around the subject. Interpolation algorithms can then be used to increase the effective frame rate to 240 fps, which slows down the action even more while boosting bullet time effect.

Six-axis gyro stabilization, a feature in some of the new cameras, helps to smooth out the vibrations from the centrifugal force being exerted on the camera.

Live streaming auto-stitched, 4K, virtual reality 360° footage is something that we should see much more of in the near future. Such bandwidth intensive streaming should be helped greatly by the increasing adoption of HEVC (H.265)-enabled imaging chips, which compress video much more efficiently than the AVC (H.264) codec.

360 Vehicle Cams!

360 vehicle cams are a growing product category. Vehicle cams in general are small video cameras that are usually attached to the dash or windshield of a vehicle. A 360-degree vehicle cam usually has the ability to see everything all around the vehicle, both on the inside and outside.

Most vehicle cams are “set and forget” devices meaning that they can continuously record data, overwriting old video data with new video data so the camera effectively never runs out of space. Of course if you need to download or upload video due to an accident or some other reason, you can always do so as long as you save the data before it is overwritten. The video is often timestamped via a tamper-proof method called “trusted timestamping”.

360 vehicle cams often have the ability to perform 24/7 surveillance. Drawing power from an internal battery or an external second car battery, surveillance vehicle cams use a motion detector to record only when motion is detected, saving power and storage media space.

Some additional features of recent 360 vehicle cameras are night or low light vision with IR LEDs , HD rendering engine, some combination of radar, 3-axis accelerometer and GPS, super smart power management, an LCD touchscreen and a G-force sensor which will stabilize or lock the video during a collision. With all these functionalities 360 vehicle cams will likely see continued growth as a sub-category of the 360 camera market.

360 camera clip-ons?

One relatively new product is the 360 clip-on. What the clip on does is attach to a regular smartphone camera to produce output similar to an actual 360 camera. It does this by adding two wide-angle lenses to the cell phone’s front and rear cameras. The clip-on itself will usually do any video stitching required. There are some clip-on cameras that rely on mirrors to capture a near 360 field of view.

A couple of advantages of 360 clip-ons are that they don’t require charging and are less expensive. That said, the quality is not the same as a true 360 camera, designed from the ground up to capture and process 360 images and video.

 

360 field of view home surveillance cameras:

Home surveillance cameras used to be primarily fixed cameras with a limited field of view. Even if the camera panned it was still limited to the field of view spec of the camera. Now consumers have a much more robust home security option, 360 field of view in home surveillance cameras. These cameras feature 360 degree rotation and a vertical 45 degree tilt. They can be either wall mounted or placed on a flat, stable surface such as a table.

Home surveillance cameras are normally controlled via a phone app or the web allowing for pan, zoom and real time monitoring of video and audio. Camera activation on motion detection anywhere within the 360 field of view followed by a push notification is a typical feature. In other words, your cat can keep the camera active.

Night vision mode is a popular feature found in many 360 degree camera in home surveillance products. IR LEDs allow for motion detection at night and cameras will have multiple of them. Another interesting feature of these 360 surveillance cameras is that they can act as an intercom, where you can connect remotely and broadcast via the camera’s speaker.

Some manufacturers have a subscription option for cloud storage of home surveillance camera video and audio which can be quite useful if you want to archive your 360 surveillance video.

Streaming Protocols

Any discussion of near 360 degree field of view cameras necessarily involves streaming video as virtually all such cameras have the ability to capture video. Once captured video can be streamed countless ways but underlying all this streaming on web sites, phones, tablets and laptops are protocols that negotiate and define the rules of transmission.

A streaming protocol specifies a method for transmitting packets of data to various hosts or devices. These packets of data are not sent in a serial or linear fashion which means that they need to be reassembled at the destination. Streaming protocols provide for this reassembly.

It’s important to note that both the transmitting and receiving devices both need to support the protocol being used. The need for both the transmitting and receiving devices to support the same streaming protocol has lead to some consolidation where a handful of protocols are dominant. We’ll cover a few of them here.

HTTP Live Streaming (HLS) protocol

At the moment one of the most widely supported protocols is the HTTP Live Streaming (HLS) protocol. A large percentage of devices in use today natively support HLS. The protocol is robust and especially adept at live streaming to large audiences. Additionally, HLS has a relatively lower cost of implementation, due to the fact that the protocol runs over HTTP.

HLS features adaptive bitrate streaming, which allows for adjustment based on the available bandwidth to deliver the best quality stream at that bandwidth level. Additionally the advanced H.265 codec is supported by HLS.

MPEG-DASH

Another widely supported and popular protocol is MPEG-DASH, an HTTP based implementation of a streaming protocol. Like HLS, MPEG-DASH has a lower cost of implementation due to the use of HTTP servers and smart, adaptive bit-rate capabilities.

An open standard protocol, MPEG-DASH is video and audio codec agnostic. On the video side you can run codes like HEVC/H.265, H.264, WebM or whatever you want. Likewise on the audio side you can run MP3, AAC, FLAC, etc.

Android devices have native support for MPEG-DASH as well.

Real-Time Messaging Protocol (RTMP)

RTMP has been around for a long time. The protocol at one time was widely used end to end with viewer facing video delivered by the flash plug-in. However, flash is being rapidly phased out in favor of HTML 5 which has native support for embedded video via the video element.

Presently RTMP is still widely used as a sort of middleware protocol used to transfer video between encoders and CDNs (Content Delivery Networks). CDNs will ingest live streams with RTMP and typically present that video to end users via the HTTP Live Streaming (HLS) protocol.

Currently much of the world is in some form of quarantine in connection with the SARS-CoV-2 virus and likely looking for creative and fun ways to pass the time. One great way to pass the time is to watch 360 virtual tours. There are jaw dropping virtual tours of some of the most beautiful places on the planet. Just pull up your favorite video platform and search for “360 virtual tour.” Before you know it you’ll have spent hours being entertained by these amazing panoramic tours!

The day is coming when many of these 360 virtual tours will be broadcast or streamed live from a variety of locations. Current wireless technology struggles to keep up with the enormous bandwidth requirements of these live streams but 5G could help solve that problem in a hurry.

5G Technology

What are 5G networks? 5G networks are a generational upgrade to existing cellular network technology. Download speeds should reach 1 to 3 gigabits relatively soon. A major carrier recently demonstrated a 5G network with measured speeds at three gigabits. Theoretically in optimal conditions, 5G speeds can reach 10 Gbit/s. The main carriers in the U.S. do offer 5G services, speed and coverage will vary.

5G impact and use cases:

It is safe to say that when true 5G bandwidth becomes widely available it will become a tool that will have revolutionary scale impact on business and technological growth. A recent study showed that approximately 80% of businesses worldwide believe that 5G technology will have significant or revolutionary impact.

The potential use cases are endless, but among those on the visible horizon are the IOT (Internet of Things), fully functional Virtual Reality (VR) and Augmented Reality (AR) applications, free, untethered and independent industrial robots roaming workspaces, remote healthcare, wearable computers, artificially intelligent vehicles and drones performing “last mile” delivery.

5G frequency bands:

There are two sets of frequency bands for 5G networks, Frequency range 1(FR1) and Frequency range 2 (FR2). FR1 is from 450 MHz to 6 Ghz, often referred to as “sub-6 GHz”. FR2 is from 24.25 GHz to 52.6 GHz, often referred to as “millimeter wave.”

Low and middle tier bands can combine two 100 MHz channels and stack three 20 MHz 4G channels on top of that. Low band operates in frequencies below 2GHz, which like TV frequencies, allow transmission to travel long distances. Sub 2Ghz channels are not wide and 4G uses these frequencies as well, so overall 5G speeds in the low band will be relatively slow compared to the higher bands.

Low-band 5G, has the advantage of the longest range of the three bands, however it also is the slowest and does not present much, if any advantage over 4G. You’ll find that speeds in this range work out to 30 to 250 megabits per second (Mbps).

High-band occupies the 20 GHz to 100 GHz range. This is the short, millimeter-wave sweet spot where 5G can really shine. This high intensity, short wave band has a functional range of only about 800 feet from a tower. The upside of this band however is the 1 to 3 Gbps speed.

All of these bands will see performance improvements as technology improves and more towers come online.

U.S. spectrum auction

On July 23rd, 2020 the Federal Communications Commission (FCC) held its first auction of licenses for 5G mid-band spectrum. The auction will offer county-based Priority Access Licenses in the 3550-3650 MHz band. This is a very exciting development!

Is 5G safe?

To date there is a broad scientific consensus that 5G technology is safe. Studies on the subject are ongoing and as 5G gets rolled out more data will be available.