CanHaptics Team Crescendo Project Iteration 2: Multimodal Music Notation
My team and I are creating a music notation viewer with haptic, visual and auditory feedback. This is my take on what happened in our second development iteration
In this iteration, we continued building our multimodal music notation viewer. To ease coordination of synchronous sessions, our team worked in pairs, divided by timezones. Based on the results of our previous project iteration, Sabrina K. and Juliette R. tackled how the transition from a note to another should feel like and how to convey different pitches. Hannah E. and I explored how to convey different note durations using haptics.
Summary
Goal
In this iteration, we wanted to tackle how to communicate different note durations using haptics.
Materials
Haply device, sketching tools (for me, pen and paper), base code.
Tools
Processing Development Environment.
Process
Hannah and I worked together, via Zoom, on different approaches. My direct contribution was brainstorming and implementing haptic feedback for different types of notes.
We categorized our main ideas based on the feeling we were expecting to convey:
- Bumpy (previous iteration)
- Sandy
- Dip
- Sandy (modified)
Synchronous work division
Since my computer isn’t capable of holding a video conference and running Processing at the same time, most of the work was accomplished with me dictating or annotating code changes via Zoom, and Hannah running the code and describing her experience with the Haply. In other words, my task was to essentially translate our ideas into code, while both of us contributed to brainstorming ideas.
Final product
Our code can be found here.
Bumpy
From our previous iteration, we achieved the feeling of a “bumpy” note, similar to what would happen when running a probe through a textured material like the one below:
One easy way to create a stable and reliable feeling through the notes is to have a “static” force applied to the end effector. In iteration 1, I chose to make the forces point towards the top left corner of the screen, since the movement in our notation viewer will mostly occur in the opposite direction — or at least from left to right.
In iteration 1, we based on the fisica package to control force and damping in our sketches. However, to simplify how we manage all elements in our system (e.g., measure bars, staff lines, notes), we decided to not use fisica in our final implementation.
Implementation
The excerpt below was implemented by Juliette based on Hannah and I’s iteration 1 results.
PVector force(PVector posEE, PVector velEE) {
PVector posDiff = (posEE.copy().sub(getPhysicsPosition()));
final float threshold = 0.005;
if (posDiff.mag() > threshold) {
return new PVector(0, 0);
} else if (this.state == NoteState.NOT_PLAYING) {
this.state = NoteState.START_PLAYING;
}
return new PVector(-1.1, -1.1);
}Result
Discussion
Ideally, we want the user to be able to understand: (a) when they are hitting a note and, (b) what kind of note they are hitting (in terms of duration). Particularly for (a), it’s important that we are also able to convey if the note is at a higher, lower or equal height in relation to the other adjacent notes. Since the user can’t place their end effector on top of a note without being “bumped” out of it, causing upwards or downwards movement, this approach might not be suitable for our project.
Sandy
In Lab 3, Hannah and I were able to recreate the feeling of “sandpaper” using small random forces that are not strong enough to move the end effector. We were inspired by what static noise should feel like if it was a tangible texture:
Implementation
The implementation for this attempt was fairly quick to achieve, since we only had to modify one line of code:
PVector force(PVector posEE, PVector velEE) {
PVector posDiff = (posEE.copy().sub(getPhysicsPosition()));
final float threshold = 0.005;
if (posDiff.mag() > threshold) {
return new PVector(0, 0);
} else if (this.state == NoteState.NOT_PLAYING) {
this.state = NoteState.START_PLAYING;
}
return new PVector(random(-1.5, 1.5), random(-1.5, 1.5));
}Result
Discussion
Overall, this texture seemed to be a promising, non-intrusive way to convey hitting a note. From the video, it can be observed that the end effector still suffers some disturbances in trajectory. This can be solved by calibrating the magnitude of the forces applied to the end effector, which might vary from user to user. Through previous labs and iterations, Hannah and I have been observing that our Haplys respond differently to the same code (or, perhaps, we have different perceptions of what is being conveyed? it’s hard to pinpoint working remotely). My Haply generally needs more “intense” forces so I can start feeling the same texture, while hers is more sensitive, capturing the effect at lower intensities.
Dip
The previous texture seemed promising for “filled” notes, such as quarters and eighths. For “hollow” notes (halves and wholes), Hannah and I had a common idea of creating something that felt like a dip, or as if the end effector had fallen inside of a note.
First attempt
For our first attempt, I thought about using the intensity of the note’s pixels as a way to control the forces being applied to the end effector. In this case, we would need to use Processing’sloadPixels() and pixels[]. From my previous experience in Lab 3, both of these functions are very costly, and make my simulation crash when called. Hannah also attempted running these functions on her computer, but it also caused her simulation to crash. Thus, we decided to pivot to another approach.
Second attempt
For the second attempt, I suggested that we implement a radial vector field. My idea was that hollow notes would pull the end effector towards the center, as when dropping a ball in a circular bowl.
Implementation
PVector force(PVector posEE, PVector velEE) {
PVector posDiff = (posEE.copy().sub(getPhysicsPosition()));
final float threshold = 0.005;
if (posDiff.mag() > threshold) {
return new PVector(0, 0);
} else if (this.state == NoteState.NOT_PLAYING) {
this.state = NoteState.START_PLAYING;
} float m2 = 1;
float fx = m2*(posEE.x/pow(posDiff.mag(),3));
float fy = m2*(posEE.y/pow(posDiff.mag(),3)); fx = constrain(fx, -1.5, 1.5);
fy = constrain(fy, -1.5, 1.5); return new PVector(fx, fy);
}
Discussion
After trying a range of values for a the constant m2 above, varying from -1 to 1, we noticed that the behavior of our implementation approached more to what would be expected for the following field:
We were not able to pinpoint the reason why our implementation failed, and decided to pivot to another approach.
Third attempt
In our third attempt, Hannah suggested using sine waves. I translated this into the code snippet below. The idea is to have forces in each axis that always add to a magnitude of 1, and that pulls the end effector towards the note in the direction of reading the score (left to right), but away from it when coming from right to left.
Implementation
PVector force(PVector posEE, PVector velEE) {
PVector posDiff = (posEE.copy().sub(getPhysicsPosition()));
final float threshold = 0.005;
if (posDiff.mag() > threshold) {
return new PVector(0, 0);
} else if (this.state == NoteState.NOT_PLAYING) {
this.state = NoteState.START_PLAYING;
}
float theta = 0;
float fx = cos(atan(posDiff.x/posDiff.y));
float fy = sin(atan(posDiff.x/posDiff.y));
fx = constrain(fx, -1.5, 1.5);
fy = constrain(fy, -1.5, 1.5);
return new PVector(fx, fy);
}Discussion
While writing this blog, I realized that I inverted the terms when calculating Fx and Fy, and the correct implementation should have been:
float fx = cos(atan(posDiff.y/posDiff.x));
float fy = sin(atan(posDiff.y/posDiff.x));
Overall, this approach seemed to be a promising way to convey hitting a hollow note.
Combining approaches: sandy + dip
For testing, we decided to combine both textures to represent two different note durations.
Implementation
PVector force(PVector posEE, PVector velEE) {
PVector posDiff = (posEE.copy().sub(getPhysicsPosition()));
final float threshold = 0.005;
if (posDiff.mag() > threshold) {
return new PVector(0, 0);
} else if (this.state == NoteState.NOT_PLAYING) {
this.state = NoteState.START_PLAYING;
}
float fx = 0;
float fy = 0; switch (getText()) {
case "\ue1d2": // whole
fx = 1.15*cos(atan(posDiff.x/posDiff.y));
fy = 1.15*sin(atan(posDiff.x/posDiff.y));
break;
case "\ue1d3": // half (stem up)
fx = 1.15*cos(atan(posDiff.x/posDiff.y));
fy = 1.15*sin(atan(posDiff.x/posDiff.y));
break;
case "\ue1d4": // half (stem down)
fx = 1.15*cos(atan(posDiff.x/posDiff.y));
fy = 1.15*sin(atan(posDiff.x/posDiff.y));
break;
case "\ue1d5": // quarter (stem up)
fx = random(-1,1);
fy = random(-1,1);
break;
case "\ue1d6": // quarter (stem down)
fx = random(-1,1);
fy = random(-1,1);
break;
case "\ue1d7": // eighth (stem up)
fx = random(-1,1);
fy = random(-1,1);
break;
case "\ue1d8": // eighth (stem down)
fx = random(-1,1);
fy = random(-1,1);
break;
}
fx = constrain(fx, -1.5, 1.5);
fy = constrain(fy, -1.5, 1.5); return new PVector(fx, fy);
}
Result
Result: Combing sandy texture+ dip + auditory feedback
Discussion
Hannah was able to test this approach with a friend who is not part of the course. Overall, the feedback we received for this approach is that while they understand that the end effector is hitting a note, it feels confusing to have two different feelings for the same type of notation element.
Sandy (modified)
Taking into account the feedback for our combined version, I decided to create a modified version of our “sandy” texture. The modifications I introduced are:
- Apply the “sandpaper” texture to all note types;
- Create an empty space inside whole and half notes, without texture;
- Increase or decrease the intensity of the forces based on note duration (wholes feel more “intense” than eighths);
Implementation
In this approach, I added a condition to only apply forces to the end effector when it is moving, to avoid oscillation when the end effector is resting on top of a note. I also replaced the use of random(-1,1) to abs(randomGaussian()), so we are able to better control how the texture is generated, since the values will follow a normal distribution.
Moreover, for larger notes, I noticed that we are able to generate a more consistent “grainy” feeling when all the forces are in the opposite direction of the movement, which also gives the impression of a “drag” through the note. For eighths, the texture is very subtle, and there wasn’t a noticeable difference when applying forces in the direction of or in the opposite direction of movement.
Last, the gradation of intensities was achieved by reducing the magnitude of the constant that multiplies the random values.
PVector force(PVector posEE, PVector velEE) {
PVector posDiff = (posEE.copy().sub(getPhysicsPosition()));
final float threshold = 0.005;
float fx = 0;
float fy = 0;
if (posDiff.mag() > threshold) {
return new PVector(0, 0);
} else if (this.state == NoteState.NOT_PLAYING) {
this.state = NoteState.START_PLAYING;
}
if (velEE.mag() > 0.00) {
switch (getText()) {
case "\ue1d2":
if (posDiff.mag() > 0.0025) { // "hollow" center
fx = -velEE.x/abs(velEE.x + 0.001) * 2 * abs(randomGaussian()); // added 0.001 to ensure not dividing by 0
fy = -velEE.y/abs(velEE.y + 0.001) * 2 * abs(randomGaussian());
}
break;
case "\ue1d3":
if (posDiff.mag() > 0.0015) { // smaller hollow center
fx = -velEE.x/abs(velEE.x + 0.001) * 1.5 * abs(randomGaussian());
fy = -velEE.y/abs(velEE.y + 0.001) * 1.5 * abs(randomGaussian());
}
break;
case "\ue1d4":
if (posDiff.mag() > 0.0015) {
fx = -velEE.x/abs(velEE.x + 0.001) * 1.5 * abs(randomGaussian());
fy = -velEE.y/abs(velEE.y + 0.001) * 1.5 * abs(randomGaussian());
}
break;
case "\ue1d5":
fx = -velEE.x/abs(velEE.x + 0.001) * abs(randomGaussian());
fy = -velEE.y/abs(velEE.y + 0.001) * abs(randomGaussian());
break;
case "\ue1d6":
fx = -velEE.x/abs(velEE.x + 0.001) * abs(randomGaussian());
fy = -velEE.y/abs(velEE.y + 0.001) * abs(randomGaussian());
break;
case "\ue1d7":
fx = 0.75 * randomGaussian();
fy = 0.75 * randomGaussian();
break;
case "\ue1d8":
fx = 0.75 * randomGaussian();
fy = 0.75 * randomGaussian();
break;
}
} fx = constrain(fx, -1.25, 1.25);
fy = constrain(fy, -1.25, 1.25); return new PVector(fx, fy);
}
*(results and discussion after the next section)
Staff
When testing our combined approaches, Hannah and I noticed that the original implementation of a “bumpy” staff feeling interfered with the forces applied to notes, causing the end effector to overshoot when close to a staff line. Also, moving the end effector in opposite directions made the staff feel differently:
private PVector staffForce(PVector posEE, PVector velEE, PShape line) {
PVector linePos = getPhysicsPosition(line);
if (abs(posEE.y - linePos.y) < 0.0005) {
return new PVector(1, 1);
}
return new PVector(0, 0);
}To mitigate this effect, I reduced the magnitude of the force and introduced a component to guarantee a force in the direction of movement:
private PVector staffForce(PVector posEE, PVector velEE, PShape line) {
PVector linePos = getPhysicsPosition(line);
if (abs(posEE.y - linePos.y) < 0.0005 && velEE.mag() > 0.01) {
fx = velEE.x/abs(velEE.x + 0.001) * 0.7;
fy = velEE.y/abs(velEE.y + 0.001) * 0.7;
}
return new PVector(fx, fy);
}When testing Sabrina’s and Juliette’s work for this iteration, I realized that Juliette’s approach delivers a similar feeling to what I aimed to achieve, in a more elegant form:
private PVector staffForce(PVector posEE, PVector velEE, PShape line) {
PVector linePos = getPhysicsPosition(line);
PVector force = new PVector(0, 0);
// Include dead band to prevent oscillations if you're staying on the line
if (abs(posEE.y - linePos.y) < 0.0005 && velEE.mag() > 0.01) {
force.set(velEE.copy().normalize().rotate(PI));
force.x = 0;
force.setMag(2);
}
return force;
}Results
Reducing force.setMag(2) to force.setMag(1.2) creates staff lines that are very subtle, but still perceptible:
Even with the conditions introduced for the notes and the staff, we are still facing oscillatory behavior around a note:
Discussion
At this point, Juliette and Sabrina tested our implementation. Although the consensus seems to be that using a sandpaper texture on notes generate a pleasant feeling, we have two items to refine in the next iteration:
- Decouple the interference of notes and lines, to reduce oscillations and overshoots when moving in notes close to the staff;
- Based on the feedback below, it could also be interesting to refine the gradation of intensity for notes with different durations;
Reflection
Stepping away from the fisica package was an interesting challenge. In our previous iteration, we were able to move quickly around different ideas by playing with fisica’s parameters, and did not have to think in depth how our force fields behaved. In this iteration, I reconnected with physics and calculus concepts that I haven’t had contact with in years, and it was very interesting to apply them in a different context than what I was taught.
In relation to our project goals, I wonder if the experience we are creating goes beyond pleasant, and is actually able to help novice musicians with dyslexia in reading sheet music faster. For our next iteration, aside from the items that need to be refined in our implementation, I hope to implement more songs, so we are able to explore how different note transitions and durations feel like. Perhaps, if time permits, we will also be able to implement a mechanism to automatically import music scores from other formats*, giving users an easier alternative to testing their own scores.
*such as PDF or svg, currently, songs are hardcoded, written note by note
