visionOS 2 brings major improvements to hand tracking on Apple Vision Pro and expands the headset’s understanding of the environment around you.
visionOS 2 Officially Announced For Apple Vision Pro
visionOS 2 introduces a number of new features and improvements across the operating system.
These improvements weren’t announced in the main WWDC24 keynote, but Apple engineers detailed them in developer sessions available online.
Hand Tracking Improvements
In our review of Apple Vision Pro we criticized the fact that developers received a very low update-rate for hand tracking, making games like Super Fruit Ninja and Synth Riders feel sluggish and unresponsive.
With visionOS 2 Apple is bringing two significant improvements to hand tracking.
The first improvement is increased update rate. In visionOS 1, the system offered developers new hand-tracking poses. This was done at 30Hz. In visionOS 2, hand tracking poses will be provided at display refresh rates, usually 90Hz. If you enable prediction, it will show extrapolated poses of the hands based on where they should be at the time the frame is shown. This results in noticeably lower effective latency, but with the tradeoff of some accuracy.
Apple recommends developers use prediction to attach virtual objects to your hands, but not when fluidity is important such as 3D drawing where you want the lines to be smooth.
Scene Understanding Improvements
Since launch Apple Vision Pro has provided developers with a continuously updated 3D mesh of the environment, generated by the LiDAR sensor. To build useful apps, developers must understand what is contained within the mesh. With visionOS 2, headsets now have Room Tracking. Apple says a developer could use this to have a virtual pet greet you when entering your bedroom, for example. Apple says a developer could use this to have a virtual pet greet you when you enter your bedroom, for example.
Beyond just walls and floors, visionOS 2 also brings improvements to plane detection.
While visionOS 1 could only detect horizontal and vertical planes, visionOS 2 can detect slanted planes too.
These improvements should allow visionOS developers to build apps more aware of your physical environment that adapt to the space you’re in and anchor to real world surfaces.
visionOS 2 also adds the ability
, an entirely new tracking capability.
The
is available to install already, and the stable release is set for fall. The stable release of visionOS 2 is set for fall. It includes the ability to track 3D objects.