A VR developer found a lower friction way to access Quest 3’s passthrough cameras.
Meta does not officially allow third-party developers to access the cameras of Quest headsets, but earlier this week we reported on a clever workaround from Michael Gschwandtner the co-founder and lead developer of the hand tracking fitness app XRWorkout.
A Developer Figured Out How To Access Quest 3’s Cameras
A developer figured out how to access Quest 3’s passthrough cameras to run custom computer vision models.
Gschwandtner’s technique involved having you use Quest’s casting capability to cast to a WebView inside his app which loaded the oculus.com/casting URL. But this requires painstakingly logging into your Meta account in this WebView, and requires the user to take action by casting.
Another VR developer, Julian Triveri, has now figured out a technique that doesn’t involve a WebView or casting, and thus doesn’t require logging in or initiating a system action.
Triveri realized that the Android media projection API, designed to enable screen capturing for custom recording or casting setups, works on Quest headsets. On Quest it returns 1024×1024 frames of the central 82 degrees of the field of view, slightly lower than you’d get with casting but still usable to run custom computer vision models.
The API requires a user-granted permission, but after that the app now has full access to what the user is seeing.
Triveri released an open-source demo of this technique on GitHub, and UploadVR has confirmed it works as described. Triveri even created a branch that shows the AprilTag tracking technique (a fiducial marker) being applied. The other restriction is the same as Gschwandtner’s older technique: the captured view includes virtual interfaces and objects, in addition to the camera view. This could cause computer vision tasks to be impacted. Or could the company crack down by patching it in a future release of its Horizon OS? Could the company patch it into a new release of Horizon OS to crackdown on this issue?