Researchers from Meta and ETH Zurich have developed a software system called TouchInsight, which they say solves the problem of turning any surface into a virtual keyboard.
Text input in VR and AR today is cumbersome, and significantly slower than on PCs and smartphones. The floating virtual keyboard requires you to hold your hands awkwardly up while slowly pressing each key, which does not provide any haptic feedback. It also prevents you from relaxing your wrist. Quest headsets allow for pairing with a Bluetooth keyboard to enable full-speed typing. However, this requires you to bring a larger device than your headset. The ability to build virtual surfaces that are locked to a surface is available to developers today. This can be done by using hand-tracking and having the user tap on the surface in order to calibrate the position. In practice, however, the slightest difference in the height of the virtual surface from that of the actual surface can result in unacceptable errors. Last year’s system required a set of fiducial marks on the table. This may have acted as a dynamic calibration system. In
the researchers said their solution achieved a maximum speed of 70 WPM and an average speed of 37 WPM. TouchInsight works on any surface, using only the standard Quest 3 head-set, without any external hardware. The researchers claim that TouchInsight has an average error rate of 2.9%. Meta did not disclose this error rate last year. TouchInsight uses a bivariate Gaussian to represent locations to take into account uncertainties caused by sensing errors. We then resolve these through context priors in order to accurately determine intended user input. “
TouchInsight could also be used for general detection of finger presses on physical surfaces beyond just typing. For example, the researchers say, it could enable a mixed reality Whac-A-Mole game where you use your fingers to squish tiny moles.last yearHow TouchInsight works.
The researchers say they’ll demo TouchInsight at the ACM Symposium on User Interface Software and Technology the TouchInsight paper, in Pittsburgh.
It’s unclear whether Meta plans to integrate TouchInsight into Horizon OS any time soon, and if not, what the specific barriers to doing so are.