ICM Media Entry# 4: Sound Synthesis with Ami

For our project this week we were told to make something using sound synthesis in P5 using any manner of expression. During our brainstorm session, Ami proposed this idea of this instrument that is an amalgamation of different body parts, a body horror instrument of terror. With each limb of this monstrosity acting as a separate instrument that plays music.

We were inspired by Pan's Labyrinth as you'll see below. We wanted the user to interact with this instrument using their own body and for that we chose ML5's Posenet function. With this bizarre idea we went to work trying to learn ML5 and Posenet. We created this skeleton code for the body following Dan Shiffman's videos on the topic. This is the code link for our Posenet skeleton. With Posenet out of the way we had to find a way to construct this in P5 using images and buttons.

We created the skeleton for Posenet through a separate function rather than it just being in draw. In this way we could re-use the skeleton function in future sketch. (Yay for refactoring!).

This is the code for Skeleton function:

function drawSkeleton () {
  
  if (pose) {
    let eyeR = pose.rightEye;
    let eyeL = pose.leftEye;
    // let d = dist(eyeR.x, eyeR.y, eyeL.x, eyeL.y);
    fill('red');
    ellipse(pose.nose.x, pose.nose.y, 30);
    fill('blue');
    ellipse(pose.rightWrist.x, pose.rightWrist.y, 32);
    ellipse(pose.leftWrist.x, pose.leftWrist.y, 32);

    for (let i = 0; i < pose.keypoints.length; i++) {
      let x = pose.keypoints[i].position.x;
      let y = pose.keypoints[i].position.y;
      fill('yellow');
      ellipse(x, y, 16, 16);
    }

    for (let i = 0; i < skeleton.length; i++) {
      let a = skeleton[i][0];
      let b = skeleton[i][1];
      strokeWeight(3);
      stroke(255);
      line(a.position.x, a.position.y, b.position.x, b.position.y);
    }
  }
}

Here it is working:

The tricky part of Posenet was remembering that each pose has different key-points for all the different body parts. And those key-points have a threshold of success they have to hit before they show up in the skeleton. For example there is no key-point for hands, instead we use wrists and at points my wrist would not show up because it wouldn't satisfy the threshold to show up in the skeleton.

We decided that the sketch should just focus on the users skeleton and not have an overlay of the webcam capture. So we removed the background entirely. To make sure the user could still see their different limb points we colored the nodes yellow. The wrists were made blue and the nose red, to get rid of the color monotony.

This is what our body horror instrument was supposed to look like. (OwO)

Body-Horror Instrument

Yes, its horrific that's exactly the vibe we were going for. This chimera of a beast has the arms of a giant spider, the tentacles of a sea monster and the face of the Pale-Man from Pan's Labyrinth. But it also has the heart of a human ;). So as you can see making something similar to this was very daunting. We decided on having each limb (image) be a button that plays some happy sound effects. when the user touches with their Posenet skeleton.

Unfortunately this part of the two sketches failed. The size and pixel quality of the images coupled with the number of sound effects kept causing P5 to crash. The sketch could only run a single image with sound before breaking on us. To make sure each limb worked I made a button class that would fit every image, but as you can see it only worked with the one image and sound file. Even then it crashes.

This is the code for the Button Class:

class Button {

  constructor(inX, inY, inImg) {
    this.x = inX;
    this.y = inY;
    this.img = inImg;
  }

  display() {
    stroke(0);
    if (this.over()) {
      tint(204, 0, 128);
    } else {
      noTint();
    }

    image(this.img, this.x, this.y);
  }
  over() {
    if (mouseX > 250 && mouseX < 320) {
      sound1.play();
    } else {
      sound1.stop();
    }
  }
}

Having spent most of the week on this and seeing it not work we decided to scrap this idea and make something else. The backup idea was to create a simpler instrument, I decided we could create a xylophone. I had the idea while going through different YouTube videos on Posenet. We constructed the xylophone using tutorials from Mister Bomb. The sketch itself uses oscillators and an envelope with ADSR to create the sound rather than uploading and using pre-recorded sound effects.

The tutorial we changed the frequency from hertz to Midi notes.

osc.freq(midiToFreq(67)); setting the 

Set up for the oscillator and envelope:

function setup() {
  createCanvas(500, 400);
  colorMode(HSB);
  
  env = new p5.Envelope(0.01, 0.1, 1, 0.5); // set values for the ADSR 
  
  osc = new p5.Oscillator('triangle'); // we used a triangle wave 
  osc.start();
  osc.amp(env); // amp is affected by the envelope 
}

function draw() {
  background('black');
  for (let i = 0; i < 8; i++) { // for loop that affects the color and midi notes 
    fill(0, 0, 100);
    if (keyIsPressed && key == keys[i]) { //  the color changes as long as key is held down and what key is pressed 
      fill(c[i], 100, 100); // refers to the index of the color array and changes the color accordingly as long as the key is pressed
      osc.freq(midiToFreq(notes[i])) //  key presses are linked to the notes through the loop and the index
    }
    rect(25 + (x * i), 100, 40, 200); // makes 8 rectangles by multiplying x by the count and creates 25 pixels of space between each rectangle 
  }
}

The function for the key presses:

function keyPressed() {
  env.play(); // plays the midi notes when the keys are pressed
}

Here it is in action: