A developer figured out a clever way to access Quest 3’s passthrough cameras to run custom computer vision models.
Meta does not allow third-party developers to access the cameras of Quest headsets, but Michael Gschwandtner, the co-founder and lead developer of the hand tracking fitness app XRWorkout, posted a video to social media platforms showing the results of an interesting workaround to this restriction.
The demo video shows the open-source MobileNet V2 object detection model running natively on Quest 3 at 0.75 Hz.
Gschwandtner sent UploadVR a test build and we independently verified that this is real and works. He also exclusively revealed to UploadVR how exactly he pulled it off.
How It Works
Gschwandtner’s app build creates a webview, a Chromium web browser instance embedded within the app, which automatically loads the oculus.com/casting URL which Quest headsets can cast their view to.
You then need to go through the full process of logging into your Meta account in this webview. You can then use the Quest menu to start webcasting. Uniquely, you’re casting from the headset to an app inside the same headset, not to a different device.
Gschwandtner’s app then samples the pixel content, the rendertexture, of the webview receiving the casted view, and from there can do with it whatever he likes. He’s using Google MediaPipe’s on-device software SDK in the example. Gschwandtner says he’s got a plan to work around this. The company may decide to make the technique unusable in future versions of Horizon OS. Gschwandtner’s social media posts include a public request to “hop on a call” with Meta CEO Mark Zuckerberg or CTO Andrew Bosworth to “show them” the technique.
Last month Bosworth gave his thoughts on the idea of intentionally giving Quest developers access to the passthrough cameras
, saying Meta will “keep looking at” the issue.
In comparison, for its Vision Pro headset Apple only gives raw access
, and only for non-public internal apps.during an interview