bare-hand Gesture Interactive Sound Panorama
Project developed at: Harman International
This project explores intuitive interaction concepts for controlling sound placement using hand gestures. The system allows users to grab, move, and release sound sources such as music, phone calls, and alerts, enabling dynamic and intuitive control of the surround stereo panorama without relying on a display.
In an automotive setting, this interaction complements HARMAN's Individual Sound Zones (ISZ), letting drivers seamlessly transfer sounds to other passengers with simple gestures. In the demo video it’s possible to see portable speakers that light up when music is played through them - this provides a useful visual feedback to indicate the sound’s location, enhancing the user experience of the PoC.
As the project lead, I contributed to ideation and concept refinement. I then developed the first prototype and oversaw subsequent iterations. The concept of input-output coincidence—where the gesture directly correlates with sound manipulation—was central to the design, resulting in a highly engaging and intuitive user experience.
This interaction method extends beyond automotive applications, with prototypes for headphones, AR/VR devices, and wearables. For example, headphones equipped with gesture sensors enable users to adjust volume, change tracks, and spatially control soundscapes directly with their hands, making the system versatile across various platforms.
A preliminary automotive product was called GESS, “Gesture Enabled Sound Space” and this is an early product sheet
Granted Patents:
This image showcases the first prototype system I developed for gesture-based sound interaction. The setup includes six portable speakers equipped with lighting effects to visually indicate the position of sound sources. An Xbox Kinect sensor tracks hand movements in real-time, allowing the user to intuitively grab, move, and release sound sources within the audio panorama. The system demonstrates an early proof of concept for innovative, hands-free sound manipulation.
Company: Harman International
Team: Future Experience Team
Location: Palo Alto, CA
Year: 2013-2015
This video showcases the first proof-of-concept prototype I built for gesture-based sound interaction within a mock car cabin. The system features six JBL Pulse speakers, which light up to visually represent sound sources, creating an intuitive way to demonstrate spatial audio interactions. Using an Xbox Kinect sensor, the prototype tracks hand movements in 3D space, as well as grab and release gestures, enabling users to manipulate audio sources dynamically.
In this demo, a sound source, such as music or a phone call, can be "grabbed" by the user, moved to a different location within the cabin, and "dropped" or "thrown" towards another seat or passenger. This allows for seamless, hands-free control over the soundscape without needing to physically reach the target location. The prototype highlights the potential of gesture-based sound manipulation in enhancing in-cabin user experiences.
This image illustrates a concept video showcasing the integration of bare-hand gesture interaction with a conceptual 3D holographic display in a car. On the left, the system highlights the 3D location of an audio point source, such as a ringing phone, using a holographic icon. On the right, the driver uses a simple hand gesture to "pull" the sound (the ringing phone) closer to their ear, demonstrating an intuitive, hands-free interaction.
This concept combines advanced gesture recognition with spatial visualization, offering an innovative and user-friendly way to interact with in-cabin audio systems and holographic displays.
This prototype extends the gesture-controlled sound interaction concept to headphones, featuring a Leap Motion controller mounted on the headband to track the user's hands in 3D space. Users can grab and move individual sounds within their soundscape, adjusting their position left-right, forward-backward, and even up-down, though vertical sound rendering is limited.
The system leverages off-the-shelf 3D audio tools like RealSpace 3D Audio for spatial sound rendering, as advanced 3D sound capabilities were outside the project’s scope. While modern AR devices now include limited gesture sensing, this system was an industry-first and developed years earlier, earning granted patents. Its design operates without a display, making it ideal for headphones. HARMAN later acquired OSSIC, a leader in 3D audio rendering, to enhance this technology in gaming headphones.