Apple’s Patent Suggests Stylus Pencil as a Controller for Playing VR Games Through its Mixed Reality Headset

The US Patent and Trademark Office recently released an Apple patent application to enhance Mixed Reality headset (Reality Pro), iPad, and Mac user interfaces. The patent envisions a future version of the Apple Pencil, resembling Stylus, that could serve as a controller for VR gaming and apps, allowing users to manage interactions with virtual agents or physical objects.

In the patent example, a user holds an Apple Pencil as a gaming controller to move objects or a character in a game called “Virtual Agent.” The Pencil can be gestured in many different ways, tapped, rolled, twirled, flicked, and swiped on its touch surface to control a VR game. The control device has sensors that detect user inputs through touch and feel on the touch-sensitive surface.

Apple explains how to use eye or gaze tracking to control the movement of objects within a game or extended reality (XR) environment. Patent FIG. 5A depicts the first part of a content delivery system, while FIG. 5C shows the other part. FIG. 5B displays sample data structures. Patently Apple summarized this in one picture.

Should the company choose to implement the idea, users who already own a Stylus might be able to avoid purchasing additional controllers for the Reality Pro headset (if the pencil in the patent is Stylus). Besides, the tech giant previously patented other methods for interfacing with virtual reality. As shown in one of its documents, the company proposes using a set of Apple Watches as controllers within the metaverse.

According to the image below and a patent from last summer, Apple Watches can track hand gestures and contact points on the palms using electrodes. This allows users to control their Apple AR/VR headset display.