Since I’ve been learning Puredata, I thought it would be a fun exercise to attempt to recreate the software Ablinger wrote to translate speech to midi notes. The secondary purpose was to turn my oft-idle digital piano into an interactive sound piece, translating sound from another part of the house into music downstairs. The result isn’t perfect, but I think it still achieves the same ambiguous result where you are able to hear the voice once you see the transcript. The biggest difference is obviously that I’m using a digital piano, not a mechanically-actuated analog piano. However, the Roland has a fairly sophisticated physical model with things like dampening and string resonance, so it’s better than nothing.

Below are the two components to the software. Clicking the image will link to the pd file if you’d like to experiment yourself. You can either load in a pre-recorded wave file and play it back, or set the gain to the adc~ to 1 and use a microphone to drive it in real time (although I set up a delay of about 3 seconds so I could evaluate the results without hearing my own voice). delread~ passes the data into fiddle~ which does all the hard work of Fourier analysis. A metronome set to 15 ms samples the outputs of the individual sine components and creates midi notes. The blocks that create the actual notes are partial_key.  The highest key on the piano is midi 108, which corresponds to 4186 Hz, so I added a low pass filter to remove frequencies that couldn’t be reproduced.

partial_key.pd creates the midi notes which are sent to the piano. The signal makenote_b is received from the metronome, which causes the note to be made. Additionally, no note is sent if the midi key number is higher than 108, the limit of my piano, or if the amplitude is too small (< 0.01).

Here is a sample of the result, speaking the following: “these are very profound words, which is why they are being spoken by a piano. I hope you are forever moved by these profound words.”

I’d love any feedback from Puredata or DSP gurus on how the software could be improved. I’m not quite sure what sorts of additional analysis and synthesis steps are being taken in the Quadraturen software as it is not available.

This is how I’d like to imagine the first Christmas–even the wind crying out. And doing so subtly, not unlike the whispering symbolic language of babies and cattle stalls.

The idea came about from the desire to manipulate various parts of the built environment to create music as the wind is blowing. Maybe change the orientation of a sheet of metal, the size of some opening, etc. With a feedback loop, the system could progress through a melody once it detected that the desired notes of the sequence had been played. Although a composer could control the content, the weather would determine the tempo. A listener may have to wait weeks or months for the piece to complete without sufficient wind.

In the meantime, I created the above synthetic version of what I imagine it could sound like (though the song above would probably have to be recorded somewhere in Antarctica). The synthesized version uses a custom patch written in Pure Data to shape the frequency of white noise to follow a midi file of choice.

Clarinet Jam Recorded over the 2007 Christmas break shortly after impulsively buying an old French-made Buffet clarinet to sate my desire to have a woodwind. I like to imagine some old Jewish rabbis in the town square having a break-dance competition.

Mourning Bells (Dueling Marches) A slightly morose piece of musical humor. Chopin and Wagner never sounded so good together.

Last Transmission An atmospheric piano piece intended to invoke the feelings of detachment. Also used as the demo in Falling Notes.

Clarinet and Piano I had the opportunity to pull my clarinet out of storage after a year and play around in an incredible space with a world-class musician. His piano playing more than compensates for my squawking.

In this interactive visualization, a musician watches the screen while performing on a keyboard. As notes are played, ‘raindrops’ begin to fall. Louder notes fall faster than softer ones and continue to accelerate as the note is sustained. Eventually, notes will blacken if they have been falling for an extended period, and will slowly dissipate unless cleared away by fresh rain from above.

The real-time feedback causes the musician to alter what is being played in order to generate particular audio and visual outcomes simultaneously.

Music in this demo is Final Transmission by Luke Loeffler