3D Pen: Augmented Reality Writing
In 2011, while at HP Palm, I developed a prototype exploring unconventional uses of tablets and pens. This system enables users to write or draw not only on surfaces but also in mid-air, with creations rendered in 3D space using augmented reality. Leveraging an OptiTrack motion capture system, the pen’s movements and the tablet’s position are tracked to create persistent 3D drawings, viewable through a tablet or smartphone acting as a Mobile AR device (MAR).
The prototype also featured a color-picking capability, allowing users to select colors from the environment before drawing. Although the system relied on external sensors, the goal of a production device was to integrate on-device tracking. The code for this innovative proof-of-concept was developed in C, and the project resulted in a granted patent after 6.5 years of review.
Augmented Reality Writing Mock-Up
These photos showcase a mock-up system demonstrating the user's perspective and experience before building the first prototype. The user experience intention was for the display to render the written word "Hello" precisely where the user wrote it, despite no real ink being used. This system allows writing and drawing not only on visible surfaces but also extends into 3D space, enabling creations to go above and around the physical surface the user is interacting with.
Company: HP/Palm R&D Center
Team: Future Concepts and Prototyping team
Location: Sunnyvale, CA
Year: 2011
This video demonstrates the fully functional 3D pen prototype, enabling users to write and draw anywhere, including in mid-air (highlighted at around 1:30). Using a video see-through tablet and a tracked pen and display, the system showcases its capabilities in real-time, including the infrastructure required for its operation.
Developed by my Future Concepts and Prototyping team at HP/Palm’s R&D Center in Sunnyvale, California, this proof-of-concept was completed in December 2011 and illustrates the potential of augmented reality writing in 3D space.
Augmented Reality Writing Mock-Up
Looking from the side instead of through the display, it becomes clear how the writing is not is visible on the table
Images from the Granted Patent that protects this system and interaction are taken directly from the video and images above