media collage creative
Screen Shot 2018-11-28 at 1.43.32 PM.png

Climber Synth

As an artist, I love how sonic feedback changes my flow.

Playing music that has already notated is valuable and rewarding, but for me it can never compare to the satisfaction of exploring new possibilities on an instrument.

The concept for Climber Synth came out of a conversation with Sid Fels about how a rock climber wearing an EEG headset, might perform a piece for an audience below.

While I’m interested in working with EEG data, I currently don’t have access to the gear required for this approach. For now I’m approaching the concept using body articulation of a climber on a tread-wall to design a synthesis engine for immediate sonic feedback.

 My capture system consists of a Microsoft Kinect v2 and Max patch to record OSC data streaming from the Kinect.  For development purposes, climber skeleton articulation data was captured to a text file. This replay-able articulation data functions as a mock climber input source to facilitate synth engine development without a climber present.  Once the synth engine is developed, Climber Synth will be able to offer real-time sonic feedback to climbers performing on the tread wall or on  any  climbing route.   Next steps:   1. Develop synth engine based on skeleton articulation data   Pertinent questions for research:   1. How could sonic feedback facilitate performance on the climbing wall? What kind of performative movement will be inspired?  2. How could sonic feedback provide real-time “coaching” for a climber’s technical form?

My capture system consists of a Microsoft Kinect v2 and Max patch to record OSC data streaming from the Kinect.

For development purposes, climber skeleton articulation data was captured to a text file. This replay-able articulation data functions as a mock climber input source to facilitate synth engine development without a climber present.

Once the synth engine is developed, Climber Synth will be able to offer real-time sonic feedback to climbers performing on the tread wall or on any climbing route.

Next steps:

1. Develop synth engine based on skeleton articulation data

Pertinent questions for research:

1. How could sonic feedback facilitate performance on the climbing wall? What kind of performative movement will be inspired?

2. How could sonic feedback provide real-time “coaching” for a climber’s technical form?

Climber Synth Wekinator Neural Network Jan 2019

In this video I demonstrate the results of integrating Wekinator to my climber synth patch to manage the many-to-many mapping of climber skeleton input data to sonic parameter output data.

For the input data, I am calculating the difference between the each limb endpoint and the torso: [hand/foot position] - [torso position]. This difference is visualized as a teal sphere. These four pairs of relative offsets are my input features derived from the Kinect Skeleton data.

I used Wekinator to train a neural net with 700 training examples associating various input sets with various output sets. After training, this model can be enabled to react in realtime to incoming input data.

For the sonic output in this example I am using [blotar~] from the Percolate package originally by Dan Trueman and Luke Dubois. My 5 output parameters are mapped to note, vibrato frequency, filter ratio, pluck amplitude, and distortion gain.