Wednesday, July 2, 2014

Bringing VR to Chrome

The Good Stuff

Let's get all the links out of the way upfront, because that's what you're really here for, right?

WebVR-enabled Nightly Chromium Builds
Go here for builds

Source
Blink WebVR experimental branch
Chromium WebVR experimental branch

Demos (Should work with Chrome and Firefox's WebVR implementation)
Quake 3 level renderer (Source) - Click the "VR" button in the lower right
WebVR test page (Source) - Simple one-page Three.js app

Related Links
blink-dev "Intent to Implement WebVR" thread
Vladimir Vukićević blog post about Mozilla's WebVR implementation
web-vr-discuss mailing list - If you have feedback about WebVR in general, this is the place for it
Chrome WebVR bug tracker - If you have issues with Chrome's WebVR implementation, log them here.


WebVR: What it is and what it isn't.

To start out, I'm going to highly recommend that you go read Vlad's blog post about Mozilla's WebVR plans first. Chrome's current implementation is very close to Firefox's, so it doesn't make sense to re-state what's already been written. (I'll discuss the differences in implementations in a moment.)


One thing that's worth talking about, based on the reaction I've seen to Vlad's post, is what a "VR-enabled browser" actually entails. There were several people that seemed to expect that this meant you would start up your browser, put on your headset, and immediately be immersed in a fully VR browsing experience. Sorry, but no. That day is a long ways off, if it ever gets here at all. (If you are interested in something like that, check out Janus VR.)

Instead, just like adding WebGL to the browser doesn't magically make everything 3D, adding WebVR doesn't turn everything into a Virtual Reality experience. Instead, it provides an API that allows developers to create VR content in the context of a web page. Picture this: You are browsing Amazon and find a jacket/TV/bike/whatever that you're interested in. If Amazon's developers took advantage of the WebVR API they could add a button that says "View in VR" which let you view the item through a VR headset in 3D at 1:1 scale. In the case of a piece of clothing you could see it on a virtual mannequin, walk around it, lean in and examine the stitching, and so on as if it were actually sitting right in front of you. You could also imagine similar experiences with educational tools, data visualization, mapping, and so on. WebVR gives developers the tools needed to make it happen.

And of course there will be games. That's such a given it's not even worth mentioning.

The State of WebVR in Chrome

The above builds are created from an experimental branch of Chrome. This means that WebVR features won't be added to Chrome proper until the API has matured a bit, and then only if there's sufficient developer and user interest. It's still an open question as to whether or not VR will take off this time around or flop again like the last several attempts. It doesn't make sense to add significant new features to Chrome if only a tiny fraction of enthusiast users will ever be able to make use of them. In addition, there's very few VR devices that are openly accessible to developers at this point (really just the Oculus Rift and, um... Google Cardboard?) so it's not clear what the common functionality will end up being. If we commit to an API too early we risk excluding devices that don't conform to assumptions made from a very small sample set.

If we wait until VR becomes ubiquitous before considering making it part of the platform, however, we'll already be hopelessly behind the curve. Maintaining an experimental branch of the browser for VR development gives us a way to work out the kinks in the API implementation without interfering with the core code base prematurely.

Currently the builds support the Oculus Rift, and I'm going to be looking at adding support for Cardboard's SDK as well, and I'm happy to looks at other VR solutions as well assuming that they have an open source SDK. I should also mention that this is a side project for me. My normal responsibilities of making WebGL even more awesome take precedence, and so I can't promise timely integration of every cool VR gadget that hits the market.

I will periodically update the builds linked on this page to ensure they stay reasonably up-to-date with future Chrome development, but I should also stress that these builds are intended for development and testing only! They have not been vetted for stability or security, so please don't use them as your day-to-day browser.

The "L" word

Ah, yes. Latency. The first and last thing anybody brings up when talking about VR development. By now you've probably all heard that 20ms is the magic number for achieving that mystical "presence" we all want. The problem is that browsers on the whole are, to put it mildly, not known for their spectacularly low latency. So how does Chrome do?


64ms from headset motion to LCD change. For reference on the same machine the Tuscany sample gives me 48ms. This actually isn't quite as bad as it may seem at first. That means that Tuscany takes 3 frames on a 60Hz display (like the DK1) for your head motion to affect the image on screen. Chrome, then, is taking 4 frames. 1 frame of difference vs a native app is definitely not insurmountable. Of course, both of those are a long way from the holy grail of 20ms, but that's a goal that everyone is still struggling to hit.

So how do we reduce the latency in the browser? That's a far more complicated question than it may seem, and there's a lot of people on the Chrome team who are working on that problem in various ways. There may be some shortcuts that VR content can take to get ahead, though. Chrome has a pretty hefty rendering pipeline that's geared around rendering traditional web content (rectangles, images, and lots of text) as smoothly as possible. This involves a lot of layout, rasterization, and compositing, which is great for that type of content but doesn't really do anything for you when your content is a single, full screen WebGL context. As such, there may be some gains to be had by short-circuiting the compositing process in that special case. I haven't had the opportunity to actually try it yet, but it's an example of the types of optimizations that can be attempted in pursuit of that elusive "presence".

Implementation Differences

Chrome's experimental implementation of WebVR is intentionally very similar to Mozilla's prototype, but there's currently some minor differences due to the parallel nature of the ongoing development. It's perfectly possible to build web apps that work with both right now, and I expect the differences will get ironed out as developers give us a better idea of what's important and what's not.

(Keep in mind that these interfaces absolutely, without question, WILL change. This is just a description of the initial versions.)

First off, since it's Blink's policy to no longer add prefixed APIs, none of the functions begin with "webkit" or "blink" or anything like that. Instead, in order to access them you will need to launch Chrome with the --enable-vr flag. (That's not required for the builds linked above, since their sole purpose is to test out WebVR)

getVRDevices in Chrome returns a promise, rather than accepting a callback. This is because:
  • Many new Web APIs are trending towards Promise use for asynchronous behavior.
  • Promises have explicit "success" and "failure" paths, which fits the use case well.
  • Promises are specced to succeed or fail exactly once, which resolves ambiguity about wether or not a passed callback may fire on an ongoing basis (like when new devices are connected).
To write code that works with both browsers is trivial:

function EnumerateVRDevices(devices) {
  // Do stuff with the devices
}

if (navigator.getVRDevices) {
  navigator.getVRDevices().then(EnumerateVRDevices);
} else if (navigator.mozGetVRDevices) {
  navigator.mozGetVRDevices(EnumerateVRDevices);
}

Chrome's PositionSensorVRDevice is almost identical to Mozilla's but includes a resetSensor function. Calling this function resets the HMD sensors to consider the user's current yaw "forward".

Chrome's HMDVRDevice has a few attributes Mozilla's lacks, and is missing a couple that are present in Mozilla's code. Chrome has displaySize and renderTargetSize vectors to indicate the resolution of the HMD display and the recommended render target size respectively. The render target size is expected to be larger than the display size, and is calculated so that the rendered output has a 1:1 pixel ratio with the display at it's most distorted point (near the center of the eye) once the resulting image has been run through the distortion post-process. It's worth noting that the devicePixelRatio should not be taken into account when setting the size of a canvas for VR rendering.

[UPDATE: Chrome no longer support displaySize, since it doesn't have any effect on rendering. renderTargetSize was changed to getRecommendedRenderTargetSize() to account for the fact that it's value changes when you change the Field of View.]

Chrome currently lacks Mozilla's finer control over field of view. You can call getRecommendedFieldOfView, but there's no getCurrentEyeFieldOfViewgetMaximumEyeFieldOfView, or setFieldOfView. All rendering in Chrome currently assumes that you are using the recommended FOV.

[UPDATE: Chrome now support setFieldOfViewgetMaximumEyeFieldOfView, and getCurrentEyeFieldOfView.]

Finally, more for my debugging purposes than anything else, with Chrome you can pass vrDistortion: false into webkitRequestFullscreen along with vrDisplay: hmd to prevent the distortion post-process from running. This is useful for checking rendering results, but can also be used if you prefer to implement your own distortion pass. The API doesn't currently provide any information about the desired distortion formula, however, so doing so will be a "best guess" affair, possibly based off of the HMD name. It's highly recommended that you let the browser do the distortion for you.

[UPDATE] I did a terrible job of explaining this and it's already causing confusion, so let me explain the Fullscreen API better. There are two functions in Chrome that you can use to go fullscreen: webkitRequestFullScreen and webkitRequestFullscreen (Note the lowercase "s" in screen). The former (capital "S") is deprecated while the latter (lowercase "s") conforms to the W3C spec. I have only set up the conformant version to work with the VR API. If you use the other one you'll go fullscreen but not get the distortion effect. So in short, use the following for cross-browser VR fullscreen:

if (canvas.webkitRequestFullscreen) {
  canvas.webkitRequestFullscreen({ vrDisplay: hmdDevice });
} else if (canvas.mozRequestFullScreen) {
  canvas.mozRequestFullScreen({ vrDisplay: hmdDevice });
}
[/UPDATE]

I should also point out that Chrome supports the Oculus DK1 latency tester with no effort on the developers part. That may go away in the future, since it's not very generic behavior, but it's useful for the time being.

Let me reiterate that all of this is subject to change based on developers needs.

Known Issues

The Chrome builds linked above definitely have a couple of issues right now that will be fixed in the near future. First and foremost is that when you call webkitRequestFullscreen({ vrDisplay: hmd }) the canvas does go fullscreen but it doesn't automatically show the content on the HMD. You either have to move the window to the HMD display prior to going fullscreen, which is a major pain, or use screen mirroring. Neither option is great, but hopefully that situation is short-lived. The goal is that when you go fullscreen with a VR device the content automatically moves over to the HMD display and then moves back to the main display when you hit escape.

Oh, and passing a HMD into the fullscreen API only works with canvas elements that have a WebGL context right now. This restriction is likely to stay in place for a while while I watch Mozilla's work with using 3D CSS content in VR. To be blunt: I have no idea how that will work out from a UX perspective, and since Mozilla has more man power on WebVR I'm happy to let them feel it out first.

Also, the final output image after the distortion is done is the same size as your render target, not the size of the HMD display. I think this might screw with chromatic aberration correction, but I'm not sure. It's not optimal from a memory usage standpoint, certainly, and should be corrected in the near future.

Finally, since this is built against a nightly build of Chromium and not a stable branch you should expect all the quirks that come with using a nightly build, most of which will not be related to the WebVR implementation.

How can you help?

If you're excited about this technology and want to help push it forward, that's awesome! Probably the single most influential thing you could do to help the cause is build something awesome that uses WebVR! Whether it's hacking support into an existing app or creating something entirely new, the more use cases there are the more compelling the API becomes.

If you want to try your hand at hacking on the Chrome implementation directly, you can do so by following the normal instructions for your platform to checkout and build Chrome (Warning! Long build times ahead!) Then check out the WebVR experimental branch over the top of it by doing the following:

cd <chrome_src>
git remote add -f wip https://chromium.googlesource.com/experimental/chromium/src
git fetch wip "+refs/wip/bajones/webvr:refs/heads/webvr"
git checkout webvr 

cd <chrome_src>/third_party/WebKit
git remote add -f wip https://chromium.googlesource.com/experimental/chromium/blink
git fetch wip "+refs/wip/bajones/webvr:refs/heads/webvr"
git checkout webvr 

You should now have a webvr branch in both your chrome and blink repositories that you can checkout and develop against. See Chromium Experimental Branches for (slightly) more details.
Beyond that, if you find issues in the implementation please be sure to report them to the appropriate places (Chrome, Firefox). If you have questions or suggestions about the API, please join the discussion on the web-vr-discuss mailing list. If you want to voice support for seeing the API land in Chrome's trunk you can comment on the Intent to Implement email, but really this is a case where actions speak louder than words. If there's sufficient developer activity around the API it will find it's way into the official channels naturally.

I'm really excited to see what the development community does with VR on the web. I've been consistently blown away at the creative genius displayed by the WebGL community and can only hope that we see even a fraction of the same ingenuity applied to VR content. There's still so much to discover in terms of VR best practices and conventions, it seems only natural that the Web become a test bed for this up-and-coming tech!

15 comments:

  1. Awesome. Greatly appreciated. We got/are getting the tools now it's time to do the awesome stuff that show the power of the web when combined with VR. So many opportunities wide open for innovation.

    It's already starting: http://blog.sketchfab.com/post/90350872539/browse-sketchfab-with-your-oculus

    ReplyDelete
  2. any plan to use Dartium instead of Chromium ?

    ReplyDelete
  3. Thanks a lot for your awesome work!

    Keep going, especially on the reducing of the latency.
    I guess direct mode support will improve this critical topic.

    FYI, i'm working on a big French 3D Company, and we are really interested about webVR.

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
    Replies
    1. I suspect you removed your comment when you realized that I *am* actually a Chrome engineer, but the sentiment was still a good one! It's still an open question, as far as the Chrome team at large is concerned, whether or not it's worthwhile to make this functionality part of the core browser. Feedback from the community, people like yourself with real-world use cases, helps to sway that decision. Feel free to bring it up!

      Delete
  5. Hi!

    How do you see, what chance it has, to have VR support available in the stable version of Chrome and Firefox in the near future?
    Do the developers plan to make available not just the VR glasses, but the movement capture devices, such as Cyberith Virtualizer also? Or are the devices like nunchuk still supported trough Gamepad API?

    ReplyDelete
    Replies
    1. I'm can't really say what the chances are for this getting into the stable versions of the browser. I'll do everything I can to push it, but in the end it probably depends more on developer interest and how well the current crop of VR devices catch on.

      As for more generic motion capture devices, the WebVR API is designed so that it could potentially be expanded to expose more functionality like that, especially if they provide precise position and orientation data, which is something the gamepad API doesn't handle well. If any devices in that category really start to see widespread adoption we'll look at adding interfaces to them.

      Delete
  6. When I tried running the dev build, AVG Antivirus complained about it. Might it be corrupted?

    ReplyDelete
    Replies
    1. Do you get the same warnings if you download an earlier build? https://drive.google.com/folderview?id=0BzudLt22BqGRa29TN1loVDBSSE0&usp=sharing&tid=0BzudLt22BqGRbW9WTHMtOWMzNjQ#list

      Delete
    2. No, that one didn't get any complaints. I did see a warning about 'no keys' or something in both.

      Delete
    3. Thanks, I'll try rebuilding soon, see if that was causing the problem.

      The "No Keys" message is expected, it shouldn't be detrimental unless you're trying to sync that browser with various built in Google services.

      Delete
    4. Got latest again yesterday, same problem. AVG is a fairly common anti-virus program. Maybe it's a false-positive, but I'm very uncomfortable with it. It seems to be some 'identity risk' and I'm not sure what it means by that.

      Delete
  7. Hi brandon. Great job on this. Is the video playing disable on oct 13 version on mac? I cant play a video on the experimental browser but I can play it on the latest version of chrome.

    ReplyDelete
    Replies
    1. Ok solved. chromium does not play mp4. http://www.chromium.org/audio-video

      Delete
  8. > If you use the other one you'll go fullscreen but not get the distortion effect.

    Ok, might this be the reason the demos on http://mozvr.com/ look skewed?

    ReplyDelete