Meta CTO Andrew Bosworth gave his thoughts on the idea of giving Quest developers access to the passthrough cameras.
While mixed reality headsets like Meta Quest 3 and Apple Vision Pro use cameras to let you see the real world, only the system and certain first-party apps actually get raw access to the cameras. The third-party developer can set up a camera background for their app, but they don’t see the actual passthrough. Instead, they get high-level information, like hand and body coordinates, 3D meshes of the environment, bounding boxes around furniture and object tracking. That means they can’t run their own computer vision models, which severely limits the augmentation capabilities of these headsets.
With visionOS 2, Apple is now giving enterprise companies raw access to Vision Pro’s passthrough cameras for non-public internal apps. While this requires a special licence from Apple and is restricted to “in a business setting only”, the move has renewed questions to Meta as to whether it plans the same.
In a wide-ranging interview with Matthew Ball, the author of the book The Metaverse: And How It Will Revolutionize Everything, Meta CTO Andrew Bosworth was asked about the issue and gave his full extended thoughts:
“The serious answer is we can all imagine phenomenally useful use cases if a developer has direct access to the camera. Meta will never build this, nor can the developer upload every image configuration that they might have to use to create a classification system. That’s also not credible. That’s a use case that goes underserved if you don’t build out the capability.
At the same time, when this technology is so new in the world, we also do want to make sure that bystanders feel comfortable if someone chooses to wear the headset, for example, on an airplane, that they understand what the implications for them are. We’re now finding new ways to deal with this. Bystander signals can be done in many different ways. Let me reverse my joke and say that I believe the Apple Vision Pro will be a great asset to our industry. It’s a fact that these devices are well understood. The more people know about them, the less they fear that someone will react negatively or be shocked by the device. It’s unfair. The context in which technology exists is the society. The more comfortable a society is with technology, then the more freedom you have to implement it. And the less comfortable a society is the more careful you must be when bringing that technology to the market. Everyone understands the conservative approach we took – i stand by it as being the right decision at that time. We are excited about the possibilities of mixed reality. If a developer with full consent from the consumer and an understanding audience contextually around the person using is
…
if we can unlock that functionality, I think we’re all excited about what that represents.
So yeah, we’ll keep looking at it: how we see consumer comfort, let’s call it, with this technology evolving, and what kind of power that unlocks. We recommend that you listen to the entire interview. It is available on Bosworth’s podcast Boz To The Future, on platforms such as
or ..