media collage creative

Vox Augmento

Vox Augmento represents an attempt to leverage VR technology for creative musical expression. What are the advantages of interacting with holograms musically?  This experience allows you to easily create your musical interface in real time with a three-dimensional sonic paintbrush and the ultimate synthesizer-- your voice. As you paint in space, audio is recorded from the headset mic and mapped to that brushstroke.

This mapping of vocal recordings facilitates non-linear access to the playback location of the recording. That allows the user to interact with the vocal sample by colliding with the brushstroke to “scrub” through the audio buffer. Furthermore, certain characteristics of the user’s brushstroke can modulate the sound in real time.

Paint a line plummeting towards your feet, your voice will be pitched down as the line falls. This type of modulation control allows users of all skill levels to play with vibrato and tremolo effects by painting lines that zig zag and wiggle.

The proof of concept for Vox Augmento was developed at the Seattle VR Hackathon September 15-17, 2017, and won an award for “Best Technical Achievement.” The playable demo is an exciting first iteration of a modern musical interface that is expressive for all musicians while being intuitive and accessible for beginners.

On the development side, we plan a few more improvements:

1. Rendering the brushstrokes as 3D waveforms

2. Brushstrokes that loop become looping samples

3. Amplitude modulation (tremolo) via outward/inward brushstroke movement

 Our large team of nine people allowed the team to put together a stunning VR world with custom designed environment and virtual painting from painter Nathan DiPietro. We got custom sonic bone wand controller design from Mike Chokran, and professional Unity development expertise from Cory Robertson.  While the VR world was being built, the rest of the team discussed different options for mapping bodily expression to sound. We settled on the idea of painting lines in space to record the voice, with expressive playback being controlled by interacting with the lines using the sonic bone wand.

Our large team of nine people allowed the team to put together a stunning VR world with custom designed environment and virtual painting from painter Nathan DiPietro. We got custom sonic bone wand controller design from Mike Chokran, and professional Unity development expertise from Cory Robertson.

While the VR world was being built, the rest of the team discussed different options for mapping bodily expression to sound. We settled on the idea of painting lines in space to record the voice, with expressive playback being controlled by interacting with the lines using the sonic bone wand.

 The Vox Augmento VR experience relies on two software systems working in coordination.  1. Unity powers the visuals and interactions such as painting lines or colliding with lines.  2. Max powers the audio engine, recording vocal samples, playing them back, and applying effects based on controller data sent from Unity.

The Vox Augmento VR experience relies on two software systems working in coordination.

1. Unity powers the visuals and interactions such as painting lines or colliding with lines.

2. Max powers the audio engine, recording vocal samples, playing them back, and applying effects based on controller data sent from Unity.

 Awarded  “   Best Technical Achievement   ”,  Seattle VR Hackathon, Fall 2017   The "Megateam"  Philip Kobernik (initial idea, audio programming)    Andrew Luck  (audio design + implementation)  Cory Robertson  (Unity programming)  Nathan DiPietro  (set design, implementation, 3d painting)  Mike Chokran  (3d modeling)  Arunahb Satpathy  (ux, graphic design)  Daniel Nelson  (ux/pm)  Michael Wolf  (ux/pm)  Christopher Nguyen  (ux)

Awarded Best Technical Achievement”, Seattle VR Hackathon, Fall 2017

The "Megateam"
Philip Kobernik (initial idea, audio programming)
Andrew Luck (audio design + implementation)
Cory Robertson (Unity programming)
Nathan DiPietro (set design, implementation, 3d painting)
Mike Chokran (3d modeling)
Arunahb Satpathy (ux, graphic design)
Daniel Nelson (ux/pm)
Michael Wolf (ux/pm)
Christopher Nguyen (ux)

 Following the Hackathon, I continued working on the audio engine to improve sample playback. Andrew Luck and I submitted Vox Augmento to NIME 2018 and were invited to demo at the conference.

Following the Hackathon, I continued working on the audio engine to improve sample playback. Andrew Luck and I submitted Vox Augmento to NIME 2018 and were invited to demo at the conference.

 Vox Augmento travelled to Virginia Tech for NIME 2018

Vox Augmento travelled to Virginia Tech for NIME 2018

Vox Augmento Demo w/ Scrubbing

This demo showcases the kind of sample exploration and performance that is possible with Vox Augmento.

Features demonstrated:
- quickly creating sample banks in VR space
- non-linear sample access
- granular-synthesis-enabled “scrubbing” playback
- squiggly line vibrato pitch modulation
- dramatic pitch modulation
- creating and performing pitched “notes.”