My guest today is Neil Trevett, Vice President at NVidia and President of the Khronos Group. This episode was recorded on June 22nd, 2017.
Neil and I talk about VR/AR standards, overlapping standards organizations and what’s the best way for companies to get involved.
Neil Trevett: Hey Yuval, good to speak to you.
VRguy: Thanks for joining. So, who are you and, what do you do?
Neil: So, my day job is NVidia. At NVidia, I work to help developers get access to high performance hardware and GPUs. But by night, I’m also elected President of the Khronos Group. And we do open standards for cool stuff like, graphics and parallel computation, and vision purchasing and now, virtual reality.
VRguy: Good. So, there seems to be a lot of interest in virtual reality standards. Why do you think that came up? Why do people want standards and, what kind of standards are they looking for?
Neil: Yeah, I think you’re right. I think there’s a lot of buzz, perhaps even hype in the industry around both virtual reality and augmented reality. I think we’re in the early stages. I think virtual reality is becoming commercially interesting because, there’s … The systems you can buy, they’re not perfect yet, they’re still evolving very quickly.
But you can certainly do some really interesting and useful stuff today. And so, people are very interested … The developer community particularly is very interested to figure out how to bring new immersive experiences onto the current generation of VR hardware. And to … it’s kind of the early days of the Gold Rush, right? You want to stake your claim early and ride the wave as The VR market expands and kind of learns how to deliver more and more compelling experiences. And, if you look at the VR industry today, note there’s a lot of fragmentation. A developer who wants to ship across the maximum number of VR systems, has to spend quite a little time, just kind of doing busy work. Porting to all of the different runtimes. Like, the Oculus runtime or, the Daydream runtime, OSVR, SteamVR.
All those runtimes currently, because the things are evolving in the early days. They’ve all chosen very, no, perfectly workable but different ways of accessing the devices in the VR functionality in those runtimes. And, those differences are not really creating any value, they’re all pretty much doing the same thing. They’re just creating friction in the industry because, developers have to spend time porting from one to the other, to the other, to the other.
Very often, that doesn’t happen. Developers end up having the time and the resources to support just one or, one or two of the runtimes, which hurts everybody. It hurts the developer, because they top ship everywhere, and it hurts the different runtimes because, they lose out on getting good content running on their platform.
VRguy: And I would say that it also hurts to other constituents. One is the hardware vendors that are trying to get support. They’re trying to get high quality content for their hardware, whether it’s an HMD, or a tracking system, or a haptics glove or, what have you.
And then, the lack of standards in my opinion, impacts the end users, the consumers because, they are unsure whether hardware, and softer investments that they make today will work on next year’s stuff.
VRguy: Would you agree?
Neil: I totally agree with both of those. And, now for the hardware vendors, it is tough because, if I create an innovative new tracking device, perhaps a hand tracker using a new camera or something, you’re exactly right. It’s really hard for that device to get rapid exposure into multiple runtimes and through the runtimes, into multiple VR experiences.
And, they you’re right. The consumer … we’re very much in a kind of VHS versus Betamax kind of situation. Where there’s computing standards out there for no good reason, and do I keep investing in content for one platform or the other whilst there’s still this market uncertainty.
And, no. Consumers hate uncertainty and so, yeah it’s holding back the whole industry, which is why I think standards are going to be an important part of the whole ecosystem.
VRguy: So let’s start from the beginning. And, as I was preparing for this conversation, I couldn’t make up my mind whether us discussing standards is going to be the most boring Podcast I ever did or, the most interesting one. I think it’s one of the more important ones. So, we’ll see if we can make it interesting.
Neil: It’s going to be exciting.
VRguy: So, if you’re think about the Kronos Group, and just for the record, Sensics is also a member of the Khronos Group and, of the OpenXR Working group. How does Khronos decide, which standards to pursue? And also, which areas in VR or AR to focus on?
Neil: That’s a great question. So, the key thing to realize about the Khronos Group, if people haven’t come across Khronos before, is that it’s an open organization. So, any company is welcome to join. And, we have a very democratic process as the working are being proposed, and detailed work is ongoing, it’s one company, one vote.
So, Khronos is not like a secret society, secret counsel deciding what’s going to happen. It’s very much the industry coming together, and figuring out together at Khronos’ safe space to come and cooperate even for competitors to know in the marketplace.
We can come to Khronos and actually decide, what we should do that is commercially relevant to the members and the wider industry. So, it’s a free Darwinian approach. If a number of the members join or, are already members and they see a new need arise then, any member can propose a working group at any time. And, if there’s enough interest from the membership as a whole, then that working group is created and, work will begin.
But, at any stage, Darwinian mechanics do apply. If interest fades off, and participation decreases, it’s a good sign that might have been a valiant effort and a good thought but, perhaps it’s not quite the right thing to do and, perhaps not quite the right time.
But if an effort does begin to gain momentum and, a lot of energy is going into the design, and more companies are joining the working group to participate in a design of a standard to make sure it meets their own market needs, then that’s a good sign that there’s a need in the marketplace and what we’re doing you know it’s going to be relevant.
And, you mentioned OpenXR. OpenXR is the Khronos working group that’s currently working to create a standard that’s relevant to both VR and AR, hence the X. And, all the signs are looking positive. There is a lot of energy. A lot of cooperation, and a lot of companies participating to get OpenXR out the door.
So, I think OpenXR is going to be commercially welcomed, and I hope to see it actually release sooner rather than later because, there is urgent need for it, I think.
VRguy: So, the uninitiated can look at these standards landscape and say well, I see that the ITU is working on a standard, and IEEE is working on one, and there’s SID and then there’s GVRA I think, an industry association. How say, someone on the outside. How would one decide where to join, should I join all of them, should I join several? And, related to that, how much is OpenXR aware of what other organizations are trying to do?
Neil: Right. Yeah, that’s an interesting question. And, you’re right. There is more VR initiatives appearing every day. But, it’s not as bad as you might think because, most of them are complimentary and not competing. So, you mentioned things like, the IEEE. They have a list of potential VR standards that they’re working on. But it’s things like, safety, and privacy, or things that IEEE are great at.
And I think that, you soon come to a realization that both AR and VR, there isn’t going to be a single standard that will be enormously huge, and complicated, and unwieldy for VR system and an AR system soon after that. It’s going to take hundreds of standards. Everything from, how do I wirelessly connect my headset, how do I configure my display, how do I connect my cameras into my processor, how do I connect my displays into my hardware.
There’s going to be hundreds of standards that we need. And so, it’s good that there’s a lot of activity from the different standards organizations in the industry, each of which have a different expertise. We need them all to be successful. And the good news is, from Khronos’ point of view, OpenXR falls firmly in Khronos’ expertise area, which is defining runtime APIs that interface between hardware and software.
We work in that area with rendering APIs like, Open GL, and Vulkan. And, OpenXR definitely falls into that category of … We’re figuring out at a very low level, how the hardware and software stacks cooperate and communicate together, which is a vital piece, but it is just one piece of this larger constellation of standards that we need.
And the good news is that we do keep an eye on the other standards that are being out about to see whether there is overlap. And so far, I don’t think it’s because it is Khronos’ traditional area, none of the other standards initiatives are defining a runtime API. So, that’s good. So, we’re not duplicating work within the industry.
If you find duplication, then we reach out to an organization and see if we can work together. Life’s too short to fight over this stuff. All we ever just want to see the industry move forward. But right now, there are multiple VR standards in flight but, Khronos is the one that’s working on the runtime APIs.
So to answer your other question. So, if I’m outside and looking in at all these different standards, should I get involved? I think the answer is, what part of the VR stack is most relevant and critical to your business? And, if you see that you really do care about different parts of the stack, which standards organizations are addressing that part.
And, if you want to really make sure that those emerging standards meet your business needs, then that’s where you should get involved. It wouldn’t be possible for any one company probably, to get involved in every single standards underway. But no, you should pick the areas that are really most relevant to your own business and your bottom line.
VRguy: So, there’s been some press about DirectReality, which I guess is a Microsoft trademark. And it sounds like Direct Reality is going to be to OpenXR, what DirectX is to OpenGL.
Neil: Mm-hmm (affirmative).
VRguy: Do I get that right? What do you know about DirectReality that you can share?
Neil: Firstly, only what I’ve seen in the press, so I have no inside knowledge. I don’t think they’ve made anything public in any detail. But yes, it certainly appears like that. As people are creating hardware, software stacks, they can use open standards. Or, if you’re fortunate enough to own a platform, like Microsoft owns Windows, of course you have … The alternative option is to build a stack of proprietary APIs.
And I think it is interesting that there is DirectReality, if it turns out to be a thing. Then, yes you can begin to see that there the stacks on the rendering side, and the stacks on the VR hardware side, the devices and the displays, are beginning to look similar.
So, you have hardware vendors and, you have the low level API for GPUs that’s an OpenGL or Vulkan, and on open standards for virtual reality devices, it’s OpenXR. But it’s directly comparable to the proprietary APIs on Windows, which would be a DirectX, Direct3D for 3D rendering. DX12 is the latest version, and yeah, Direct Reality sounds like it could be doing similar kind of thing to OpenXR.
So, is that good, or bad? Well, depends on who you are. And, overall I think it’s okay because, … For a couple of reasons. I think it’s unavoidable. Platform vendors, and there are perhaps three platforms that really matter. There’s Android, Windows, and Apple, the big three right now. Linux of course, is there too. But Linux tends to be more open standards oriented and, Android is more open standards oriented. They’ve adopted Vulkan as their primary API, rather than do their own API like, DX12.
But, I think it’s inevitable that some of the bigger platforms will try to create their own app stacks for reasons of control, because they can. For developers, it’s a double-edged sword. It’s good that people are investing in VR on these different platforms. It’s good that developers can access VR wherever they go. It increases developer costs, if they have to go to different APIs every time they hop between platforms, like we were saying before.
So, it can make the world difficult for developers. But, you need to zoom out to the big picture. I think it’s not just inevitable that there are going to be proprietary and open standards, it’s the way of the universe. It’s like the law of physics. But there is a positive dynamic, everything needs a competitor to push them to advance quickly. Open standards are no exception.
An open standard for anything like VR or, rendering moves much faster and, it keeps an eye on the ball much more effectively if, there’s a proprietary competitor pushing. And, the open standards will push the other way too. The proprietary APIs definitely move forward if the open standard is making good forward progress.
So overall, competition is good and, it’s up to the open standards to make sure that they are a viable competitor that’s moving forward to really make sure they’re a player in this competition that goes forward.
VRguy: And is there anything you know about Apple, and where they stand? Do you expect them to have their own standard? Do you expect them to join or hope that they join OpenXR?
Neil: I can’t speak for Apple obviously. I wouldn’t ever speak for Apple, I can just look at what they’ve publicly announced at WWDC. But they have started talking about VR, and they are supporting SteamVR on the Mac platform. So, I think it will be interesting to see Steam VR and Valve as you know, those will be both participating in the OpenXR working group.
Valve has been a very positive and proactive member of the working group, helping to push things forward. And, Valve publicly stated that, once OpenXR is done they’ll support it. So I think Apple has an interesting decision. At the moment, if they have SteamVR on Mac, it means that applications that want to come on to Mac will report to the SteamVR APIs.
When the Steam VR runtime supports OpenXR, perhaps Apple would get more content if they encourage and enable the runtimes like Steam VR that are on the Mac platform to expose the OpenXR application facing interfaces. Perhaps they’ll get more content that way or, perhaps they’ll define their own API like they have for the proprietary rendering API, Metal, which is the competition to Vulkan and DX12. It’ll be interesting to see which way they go.
Again, I think it comes down to the OpenXR community and the OpenXR working group. A, to deliver a specification as quickly as we can, because of course whilst we’re not shipping it publicly for the people outside the working group, it’s just theoretical. So we need to get something out there as quickly as we can. And then, they’ll push forward the OpenXR ecosystems, so there’s a lot of contents, a lot of devices, that will be accessible to a platform if they choose to expose the OpenXR interfaces.
That will be the thing in the end that persuades a platform like Apple to consider, should we expose the OpenXR interfaces, if you get more device content why not, one might argue. So, it’s up to the OpenXR community to make OpenXR vibrant so there is that incentive to use the standard, and it’s not just a theoretical … It doesn’t make any positive impact.
VRguy: Let’s switch for a second for your other hat as a VP at Nvidia. I think you told me once that you spend about 70% of your time on Nvidia, and about 70% of your time on Khronos.
Neil: Yeah, that’s right. 70/70. You remember that, huh?
VRguy: Unless it has changed. Or maybe you say that to everyone.
Neil: No, no. That’s about right.
VRguy: So, when we did, and we continue to do OSVR of course, we have to sort of support time warping on NVidia, and time warping on AMD, and then, similar capabilities. So, how come you guys aren’t able to agree on a common API to access really, the same kind of functionality?
Neil: That’s a great question. I think it’s just a matter of timing. It’s all fairly new, I think is probably the best answer. I don’t think we want to fragment it, but actually OpenXR, … OpenXR of course, is not a rendering interface. But, I think OpenXR might be enabling, and at least a kind of a catalyst to get people to think about this, on how we might begin to strategize some of the interfaces down into the rendering pipeline.
Because for OpenXR, this is all still being discussed, of course. But if OpenXR begins to define some kind of parameters the rendering pipeline can pick up and understand how best to support a runtime, then, I think that’s a force for good. And, people will begin to coalesce around agreed ways of efficient low latency, multi view type rendering.
Yeah. So I think OpenXR can help. At NVidia, we have delivered a lot of innovative functionality. And we’re still innovating on the best way that we can reduce latency, and provide multi view type support of other innovations. And, you have to be careful. Whilst people are still innovating, and truly figuring it out, rather than just doing something that we all know how to do. It can be too early and, if you try to standardize something too early, it just falls apart because no one can agree yet because no one knows of what’s the best thing to do.
And I think, at least partially on that rendering side, we’re still in that kind of situation. And actually, it’s very interesting … Again, you were there too and, I’m interested if you agree. I thought that the major barrier to setting up OpenXR in general was going to be, … And there’s a lot of innovation in VR, and there is a lot of innovation coming. Is it going to be too soon? I was very interested to your views on that, and Oculus, and Valve.
It was good. I thought it was a good outcome. There was a realize … We haven’t stuck our head in the sand, we realize that we are at just the beginning of the innovation curve. But, it was good that everyone agreed that we know enough to start the process. There’s a kernel of things that are well understood, that are common across different platforms. And that makes sense to standardize that.
And then, as we understand more about the overall platform, including how to drive a rendering pipeline efficiently, we can begin to fold that into subsequent versions of OpenXR. But it’s good to start the standardization process. Once there is this inner kernel of functionality that we can begin to agree on quickly. Because I think that will help the industry move forward.
VRguy: I think VR is the youngest 50 year old industry. Because, it’s been in existence forever. There have been HMD’s, … Even Sensics has been making HMD’s for 15 years now and, there are already frameworks that supports dozens or hundreds of motion trackers and so on.
Neil: Right, right.
VRguy: So, it has certainly picked up a lot of energy. I know Steam is a reserved word, but it’s picked up a lot of steam or energy. But, now I think that we can apply a lot of the lessons learned in dealing with all these devices and, seeing what’s sometimes referred to as the suffering of users and trying to get everything together.
So, from my perspective, it’s a good time.
As we get closer to the end of the conversation, if there was one or two things that you want the OpenXR group to do faster or, better or, more of, what would that be?
Neil: Well, I think time is of the essence. Particularly if they’re going to be competing proprietary GI’s, then to my earlier point that’s good, it pushes us forward. And I think we shouldn’t waste time. I think the conversations in the working group are good. And, to your point, there’s a lot of people there, yourself included, with a lot of experience and it’s kind of awesome to watch the discussions in the working group.
It’s not just theoreticals, it’s people with real hard run experience coming and sharing that experience as a group to enable the standard to be more reality based. And that’s an awesome thing to watch. And so, it’s good work going on. I think the most important thing is to get it out there as quickly as we can, not to rush it. These things do take time, but not … to be honest, there is an urgency and, the sooner we can get things shipping, the sooner we can help the industry move forward. I think that really is the most important thing.
The other comment I would make is, a lot of people do ask well, why is it X? And I think in general, the industry’s really making a mess of this naming debate. But, we did deliberate, and we kind of mentioned this earlier, but we did choose X because, we didn’t want the standard to be V, as that’s virtual reality. And of course, Valve already has Steam VR but, even if they didn’t we wouldn’t have called it OpenVR because, we want the standard to be relevant to augmented reality down the line too.
There’s a lot of commonality between AR and VR and, there’s no reason to reinvent all of this again when we AR becomes a little bit more mature. I think, the group has made the wise decision to focus on VR first because, the need is more present. I think it’s … I would urge the working group to keep AR in the back of their minds. Let’s not preclude anything to do with AR, as we’re designing OpenXR 1.0.
VRguy: And there’s certainly also, products that are … They’ll use AR and VR whether it’s because of camera based AR or, just put a cover on the AR side and you’ve got a VR headset. So, I agree with you, there are many reasons to keep it combined. So-
Neil: Yeah, yep. Actually, the CTO at Meta had a great quote the other day. He was presenting a GTC and he said, “VR is just AR with a bucket over your head.” So I thought that was cool.
VRguy: All right. So Neil, what’s the best way to get in touch with you or, to learn more about the things that you’re working on?
Neil: So, anyone’s welcome to contact me directly at, Nvidiaintrepid@nvidia.com, or on Twitter at, neilt3d. But, all the information about the standards, kernel standards that we’ve been discussing, everything is available on the Khronos website, which is Khronos.org, not Khronos.com, Khronos.org and, there’s lots of links there too, reaching out. And again, as I mentioned right at the beginning, this is an open initiative, this is an open organization. And, if people are out there interested to help us move this forward, they would be very welcome to join and have a voice in the vote in how this falls.
VRguy: Excellent. Well, thank you again for joining me today.
Neil: Okay. No, thank you.
The post VRguy podcast Episode 23: Neil Trevett, President of the Khronos Group appeared first on Sensics.