Maestro: Conducting Music You’ve Never Learned to Play

Maestro lets someone with no musical training conduct an ensemble like an orchestra. A glove selects and shapes four instruments, a baton sets the tempo by beat, and a projected interface of colored waves pulses in time. It is a build and a study: empty-handed instruments give the player no physical feedback, and we wanted to know whether adding it back matters.

How you conduct it

The instrument is played with two hands and no buttons. Four instruments, three intensity levels each, all controlled by pointing and gesture.

The research question

Gesture instruments have no keys and no strings, so there is nothing to feel. We added haptics (vibration in the glove palm and baton) and ran a between-subjects study with 18 participants, haptic feedback being the single variable. The honest result: a 100 ms vibration on selection significantly improved glove usability (SUS, p ≈ 0.05). Playing experience and tempo accuracy trended better but were not significant.

The takeaway is not that haptics are magic. It is that even a minimal cue moves usability, and that usability scores did not track what players said they preferred. So you have to measure both: the metric and the felt experience.

How it works

I owned the technical spine end to end: the circuit and Teensy firmware (IMU pointer, three flex-sensor gestures, the two haptic motors under test, and baton BPM via z-score peak detection), and the entire projected interface in Processing and GLSL.

The interface is not decoration, it is coupled to the interaction. The baton's detected tempo drives a shader uniform, so one wave cycle equals four beats. Gesture to tempo to light is one real signal chain, built at every link. That coupling is what makes the instrument feel alive.

The team

A four-person build. I owned the firmware, circuit, projected interface, and data analysis. Pietro Cau (gestural design, 3D printing, music pipeline, video), Andrea De Carlo (concept, BPM algorithm), and Lucrezia Di Bari (glove build, experimental design, recruitment) rounded out the system and the study.