VR explorations at Hotstar

In case you didn’t hear about this last year, in October 2016, we launched the first live streaming virtual reality (VR) experience in India for Kabaddi, as part of the 2016 Kabaddi World Cup experience.

Let’s first get something out of the way, yes, it was a World Cup, and unlike MLB World Series, there were 10 other countries that participated, such as, USA, Australia, England, Argentina etc. So, that’s settled then.

Now, back to the VR experience.

Our VR offering for Kabaddi, comprised of the following unique experiences:

1. A 2-dimensional 180 degree experience, so one could pan around using the mobile phone without a VR headset, and soak in more of the in-stadium experience than traditional broadcast frame.
2. A fully immersive stereoscopic 3D VR experience, that could be enjoyed inside a Google Cardboard headset or a Samsung Gear VR headset.

In this post we will share some of our key learnings from this experience. We’ll start by exploring the experience we built, the key decision points for us, current challenges with VR and talk briefly about what’s next in VR explorations at Hotstar.

Our experience

Our VR experience comprised of 3 key features (annotated in the image):

1. Switch cameras: We had two 180-degree cameras setup on either side of the court, so you didn’t miss any court-side action from any angle. You could pick whichever camera provided a better vantage point, based on the sequence of play on the court. (Besides, one of the cameras was strategically placed to afford a sneak peek into the VIP box, so you could, if you so chose to, stare into the VIP box all through the game)
2. Cardboard: If you had a cardboard device, you could switch to this viewing mode, drop you phone into the cardboard headset and get into 3D VR experience.

3. Switch out of 360: Besides this, we also had our regular curated broadcast stream available. So, one could turn off 360 and get back into the broadcast experience.

Some technical details of our experience:

• For the stereoscopic capture, there were a total of 12 cameras per pod, 6 per eye. We had 2 streams per pod. One for 2D and the other one for 3D. So, overall there are 4 streams (URLs) being generated for this experience from the stadium.
• The streams were uploaded to Amazon S3 storage in Mumbai, from where they were pulled down into our Akamai cache and from there distributed to the apps.
• Both Pods had a side-to-side 180 degree coverage and 60 degrees (top-bottom)

Key decision points

Some of the key decision points when building for VR:

#1: 2D monoscopic or 3D stereoscopic?

Monoscopic videos are typically captured using one camera per field-of-view (FOV) and these frames are stitched together to form an equirectangular video that’s then projected inside the headset around your face. Naturally, since there’s one camera per FOV monoscopic videos lack depth. Stereoscopic videos, on the other hand, are captured using two cameras per field of view (one for each eye) to provide depth perception. Typically stereoscopic footage is much more challenging to get right, as the videos for each eye need to be stitched separately, and a side-by-side 3D video needs to be created, mapping the left and right video to each eye. The other tradeoff when evaluating between monoscopic and stereoscopic is the output resolution. With monoscopic videos, you get higher resolution per eye, because you don’t have to stack the left and right channels within the phone’s limited resolution. So, monoscopic videos, broadly speaking, are twice the resolution of their stereoscopic counterpart. In our case, we decided to offer the best of both worlds, by shooting in stereoscopic 3D, then using one of the cameras outputs to stitch a 2D 180 viewing experience, that could be viewed without having a headset, while also, offering a fully immersive stereoscopic 3D experience, that could be enjoyed inside a VR headset.

#2: What platform to build for?

There’s no obvious choice here, understandably so, it’s early days for VR, so many companies are pursuing independent paths when it comes to owning the VR platform. So, till last year (before Google announced Daydream) we had the option to build for Google Cardboard, which made the experience available on both Android and iOS devices, and potentially make the experience widely accessible, since cardboard headsets are the cheapest VR headsets one can find in the market. The other option was to build for Samsung’s Gear VR platform, which meant shipping a separate native VR app in the Oculus store, built from the ground up for VR. This has limited reach, since the experience is available only on Samsung Gear VR headset, which limits access to users with top-of-the-line Galaxy phones with Gear VR headsets. The next option was to build for Oculus Rift (or HTC Vive, which had not come out yet), which significantly limits the reach, as these devices are in the $1500+ range (headset + computer). So, to strike a good balance between reach and experience, we chose to build for Cardboard and Gear VR: cardboard gave us reach, while Gear VR was our flagship VR experience.

Some background on VR devices:

In terms of economic reach, Cardboard by virtue of its low cost, makes VR broadly accessible. Cardboard is a specification, rather than, as is sometimes misconstrued, the material that needs to be used to manufacture this device. Given the low cost of the device this is considered to be the “starter” VR experience. Cardboard is a particularly challenging platform to develop for, as we realised. The proliferation of both high and low end devices, not all of which are equally capable of decoding a VR stream, means a lot of those constraints need to be discovered and handled by applications. Incidentally, developing for Cardboard on iOS was somewhat easier than developing for Cardboard on Android, far fewer variables on the hardware to account for, with iOS. True to its name, it’s a very basic experience, that said, it does give you a glimpse of the immersive experience in VR. It does come with several limitations such as high latency, discomfort, lack of head-tracking among others, which makes extended viewing inside these headsets, sort of, impractical. (Google, though, appears to be moving upstream in terms of the experience, as they recently announced at their I/O event last summer, a new VR platform called “Daydream”. In terms of experience daydream is expected to be in the realm of Samsung Gear VR.)

The next higher end experience comes from Samsung, in the form of Gear VR headset (which works with high end Samsung Galaxy line of phones). So by virtue of this being a relatively closed ecosystem, this yields a rather limited reach. These headsets come with head-tracking built-in, so you can turn your head around and the visuals respond to your movement, albeit with some perceptible latency. (See note on latency below that explains how this affects your experience inside a VR headset) That said, overall, the experience inside a Gear VR headset is much better than a cardboard experience, the headset itself being a lot more comfortable to wear, and due to its lower latency (compared to cardboard) affords a more responsive experience. However, to build for Gear VR, you have to build a separate app for the Oculus store on Gear VR. This is a native VR app, so, you’ll have to build the surround environment for your content to live inside the Gear VR app.

The next step-up in experience is offered by devices such as Oculus Rift, HTC Vive, Playstation VR. These devices need a computer (or a playstation) in addition to the headset, so not quite portable or cheap. They have a really low latency thereby reducing the dizziness that can sometimes be observed in the low end devices, and thereby lend themselves to longer viewing sessions. These devices also come with (outside in) positional tracking, so you can move about in the virtual environment by moving around in the physical environment. But, that does translate into a step function, when it comes to the experience. In layman terms, it’s nothing short of “mind blowing”. Positional tracking, or room-scale VR, which essentially means, you can move about in a room, and you’ll move inside the virtual environment.

Latency matters:

This is one of the biggest hardware challenge for VR devices today. It is the time it takes for an image on the headset to match to the users head movement. This is pretty much what differentiates a great experience from one that makes you feel dizzy. As Michael Abrash explains: “When it comes to VR and AR, latency is fundamental — if you don’t have low enough latency, it’s impossible to deliver good experiences, by which I mean virtual objects that your eyes and brain accept as real. […] The key to this is that virtual objects have to stay in very nearly the same perceived real-world locations as you move; that is, they have to register as being in almost exactly the right position all the time”

#3: 180º or 360º viewing?

We chose to scope the field-of view to 180 primarily to accommodate having an experience for majority of the devices in India. A 360 pano is almost twice the size, so the bitrates are going to be way beyond what even 3G networks would have been able to support. So, to strike a balance between making the experience broadly accessible and at an acceptable quality we chose scope the field of view to 180. Besides, much of the action, happens within the 180º courtside view.

So, what’s next?

VR is an exciting, fast evolving space, that will change the way we consume content. The question is not, if, but rather, how soon. With Oculus developer conference around the corner we can expect some exciting new announcements on the device front. Facebook talked about social VR, yet again, at its F8 conference earlier this year. We can expect, that devices in the mid to high end spectrum of experience, will continue to evolve rapidly, with the higher end devices possibly going wireless, getting inside-out tracking (so you could get room-scale VR with portability) and mid-range getting more comfortable, lighter, with lower latency.

With Kabaddi, we have taken some initial steps in exploring VR. And we picked Kabaddi, primarily because it’s a sport that lets us, from the production side, get the cameras positioned really up-close to the field of action. But, it’s natural, given the cricket frenzy country that we live in, we would explore VR experiences for cricket, which being an outdoor sport, brings a whole new set of production challenges. Our goal at Hotstar is to continue to push the envelope, when it comes to content experiences with state-of-the-art technology. So, we’ll continue to explore this medium to bring more in-stadium like experiences to the sports audiences in India.

Hope you have found our journey on VR as exciting to read as we have found building it. As always, feel free to drop us a comment below and if you are excited about building VR experiences, or even producing VR content, reach out to us at careers@hotstar.com.