Mark Matamoros ITP Blog

ITP Courses

Fall 2019

Spring 2020

Fall 2020

The Neural Aesthetic

Final Project. December 7, 2020.

POSENET FOR SOUND CONTROL: REVISITED

For my final, I chose to revisit my Processing midterm project where PoseNet was being utilized as a controller for sound manipulation. Within this iteration, however, the Minim library implementation was removed and replaced with TheMidiBus library. Accordingly, derived PoseNet values are converted to MIDI data; thereafter, the MIDI is virtually sent to and captured by musical software instruments. Regarding the video documentation, the virtual instrument Massive (Native Instruments) is recieving and utilizing the MIDI connection. Furthermore, the "macros" that control multiple facets of the software are being directly manipulated by the recieved MIDI CC values.

In addition to the MIDI implementation, a smoothing function was utilized to handle numerical "jumps" pulled from PoseNet. Unfortunately, as my laptop was unable to effectively handle Runway, Logic, Massive, and Processing at the same time, the smoothing function was only slightly effective. This problem is evident in the video documentation.

Processing Code:

Individual one playing the instrument sitting down Individual one playing the instrument sitting down Individual one playing the instrument sitting down Individual one playing the instrument sitting down Individual one playing the instrument sitting down

Midterm Project. November 3, 2020.

POSENET FOR SOUND CONTROL

My midterm project is an excercise in utilizing PoseNet as a controller for sound parameters within a Processing program. Specifically, a waveshaped signal is generated and thereafter manipulated by a variety of effects. It must be noted that all sound-related components are provided by the Minim library.

Regarding the control parameters, the following PoseNet-derived body part coordinates were utilized for sound manipulation:

Unfortunately, occasional sound "hiccups" pertaining to the derived PoseNet coordinates creates a small amount of random parameter control. Additionally, my computer appeared to have a difficult time handling my Processing program in conjunction with the Runway application. Due to these issues, I decided to implement two delay effects within the signal processing chain to "smooth" the final output. Furthermore, these delays provided a personal sonically gratifying effect.