Meta’s Quest SDKs now enable placing virtual objects on surfaces without a scene scan, showing a passthrough cutout of any keyboard, and discovering nearby headsets for colocation via Bluetooth.
Instant Placement and Keyboard Tracking are available in v71 of Meta’s Mixed Reality Utility Kit (MRUK), and Colocation Discovery is available in v71 of the Meta XR Core SDK.
Instant Placement
Many mixed reality apps place virtual objects on real surfaces, such as your table or wall, by leveraging the Scene Mesh that Quest 3 and Quest 3S can generate via the optional mixed reality room setup process.
If an app tries to leverage the Scene Mesh in a room the user hasn’t scanned one in, the user will be forced to go through that setup process, which can take up to a minute of panning their head around the room.
That’s a lot of friction for an app that may only need to know where a table or wall is and that’s what MRUK’s new Instant Placement feature aims to solve.
Footage of Instant Placement from VTuber CafeSingularity.
Instead of using the Scene Mesh, Instant Placement leverages the Depth API to provide a raycast from a controller or hand to an occluded position in 3D space, allowing spawning a virtual object or interface on a surface.
Keep in mind, however, that this is only appropriate for spawning simple stationary objects and interfaces. If they need to move around the table or interact with any of the rest of the room, the Scene Mesh would still be needed.
Keyboard Cutout
Meta’s SDKs have included a feature that allows rendering a virtual version of select supported tracked keyboards for years now.
But attempting to model every possible Bluetooth keyboard would be a losing battle, and in v71 of Quest’s Horizon OS Meta added the ability to track any keyboard and cut it out via passthrough, instead of rendering a model of it.
Footage of keyboard cutout from VTuber CafeSingularity.
Now, that capability is available to app developers too, as part of MRUK. That means any VR app that wants to can have your keyboard show via a passthrough cutout, particularly useful for productivity software.
Virtual Desktop has been doing this manually for months now by leveraging the tracking of the select supported keyboards. Now, in theory, its developer could expand this support to any keyboard.
Bluetooth Colocation Discovery
For over a year now, Quest headsets have supported same-space local multiplayer, known as colocation, via the Shared Spatial Anchors API.
But a major friction point in enabling colocated experiences has been the need to use the same kind of UX as online multiplayer, a contact invite or room code flow, often involving a sequence of menus.
Quest 2 Finally Supports Shared-Space Local Multiplayer
Quest 2 finally supports shared-space colocation. “Colocation” is the term for multiple headsets sharing the same physical play space with the same virtual coordinate space. Facebook showed off an “arena scale” prototype of this feature in late 2018, but didn’t ship it to a headset until late last year
With v71 of the Meta XR Core SDK, Meta now has a Colocation Discovery API. Meta XR Core SDK v71 now has a Colocation Discovery API.