Gesture Synth

What is Gesture Synth?

Gesture Synth is an interactive browser instrument created by Eric Wei. It uses a webcam to track both hands and turns visible finger positions, hand tilt, thumb position, and hand height into chords and real-time sound controls. Players can perform without a physical MIDI keyboard while still choosing a musical key, chord mode, voicing, octave, filter level, and volume.

Getting started with Gesture Synth

Enable the camera and audio

Open the instrument in a compatible browser, allow camera access, and click the start overlay to enable audio. The webcam feed stays behind the visual hand-tracking overlay so the player can see whether both hands are being detected.

Choose a key and synth tone

Select one of the 12 chromatic keys before playing. The tone selector provides Warm Synth, Bright Synth, and Retro Synth options, allowing the same hand movements to drive different oscillator sounds.

Gesture Synth hand controls

Left hand harmony controls

The left hand chooses the scale degree and major or minor mode. Tilting inward selects major and tilting outward selects minor. Showing one through five fingers maps to scale degrees I through V, while the index-and-pinky combination selects VI and adding the thumb selects VII.

Right hand chord controls

The right hand changes the chord quality and voicing. One finger plays root position, two fingers select first inversion, three fingers add a major or minor seventh, and four fingers produce a dominant or diminished seventh according to the active mode. Moving the thumb inward or outward shifts the octave.

Right hand expression controls

Right-hand movement also shapes the sound while a chord is active. Tilting the hand changes the filter, and raising or lowering the hand controls volume. This allows a player to articulate and reshape a sustained harmony without reaching for an on-screen knob.

Gesture Synth visual feedback

The interface displays the detected chord, current chord quality, filter percentage, and a vertical volume meter while the instrument is running. An optional gesture guide summarizes the left- and right-hand mappings on screen, making the controls available without leaving the performance view.