My work has been developing visualizations that utilize the new physical display in different ways. The first visualizes air traffic over four major airports in the US, drawing moving lines to indicate aircraft taking off and landing. The second, entitled Order/Disorder, displayed below aggregates a number of sources for natural disaster information and draws the disasters as “tears in the fabric” of the world. Lastly, I am working to create software that reacts to the environment and responds by flashing a series of animations developed by Marshall Reese.

A program in Processing communicates with the hardware and serves as a hub accepting drawing commands from other software via OSC. A number of scripts in Python perform data aggregation, scraping, parsing and animation for the flight tracker. A patch in Pd performs sound analysis and drives the canvases through OSC.

“Order/Disorder is a software visualization of destructive and restorative forces in the world. The software runs on an electronically-controlled tapestry of woven fiberoptic threads created by New York-based artists Ligorano/Reese. Two computer programs, named order and disorder, modify the tapestry throughout the day in response to natural and man-caused events such as earthquakes, biohazards, and aircraft accidents. Order seeks to weave a rainbow-gradated pattern representing peace and wholeness while Disorder seeks to destroy and unravel the tapestry by “tearing” the fabric and weaving in aberrant threads.”

Since I’ve been learning Puredata, I thought it would be a fun exercise to attempt to recreate the software Ablinger wrote to translate speech to midi notes. The secondary purpose was to turn my oft-idle digital piano into an interactive sound piece, translating sound from another part of the house into music downstairs. The result isn’t perfect, but I think it still achieves the same ambiguous result where you are able to hear the voice once you see the transcript. The biggest difference is obviously that I’m using a digital piano, not a mechanically-actuated analog piano. However, the Roland has a fairly sophisticated physical model with things like dampening and string resonance, so it’s better than nothing.

Below are the two components to the software. Clicking the image will link to the pd file if you’d like to experiment yourself. You can either load in a pre-recorded wave file and play it back, or set the gain to the adc~ to 1 and use a microphone to drive it in real time (although I set up a delay of about 3 seconds so I could evaluate the results without hearing my own voice). delread~ passes the data into fiddle~ which does all the hard work of Fourier analysis. A metronome set to 15 ms samples the outputs of the individual sine components and creates midi notes. The blocks that create the actual notes are partial_key.  The highest key on the piano is midi 108, which corresponds to 4186 Hz, so I added a low pass filter to remove frequencies that couldn’t be reproduced.

partial_key.pd creates the midi notes which are sent to the piano. The signal makenote_b is received from the metronome, which causes the note to be made. Additionally, no note is sent if the midi key number is higher than 108, the limit of my piano, or if the amplitude is too small (< 0.01).

Here is a sample of the result, speaking the following: “these are very profound words, which is why they are being spoken by a piano. I hope you are forever moved by these profound words.”

I’d love any feedback from Puredata or DSP gurus on how the software could be improved. I’m not quite sure what sorts of additional analysis and synthesis steps are being taken in the Quadraturen software as it is not available.

This is how I’d like to imagine the first Christmas–even the wind crying out. And doing so subtly, not unlike the whispering symbolic language of babies and cattle stalls.

The idea came about from the desire to manipulate various parts of the built environment to create music as the wind is blowing. Maybe change the orientation of a sheet of metal, the size of some opening, etc. With a feedback loop, the system could progress through a melody once it detected that the desired notes of the sequence had been played. Although a composer could control the content, the weather would determine the tempo. A listener may have to wait weeks or months for the piece to complete without sufficient wind.

In the meantime, I created the above synthetic version of what I imagine it could sound like (though the song above would probably have to be recorded somewhere in Antarctica). The synthesized version uses a custom patch written in Pure Data to shape the frequency of white noise to follow a midi file of choice.

Over the years I’ve played with a number of tools for audio processing: Matlab, jMusic, a Java library for algorithmic composition, Nyquist, a Lisp-based synthesis/analysis environment, Beads, another Java library for synthesis and analysis, and Supercollider, another synth/analysis environment with smalltalk-like syntax. All of these are powerful tools, but aren’t as engaging in terms of interactivity. Having been forced to use LabVIEW in the past, another dataflow language, I was initially reluctant to pick up another, but for audio work, it’s been great. It is so easy to try new ideas without any need to recompile. It’s a lot like playing with a running circuit.

So far I’ve used to to analyze sound and control some lighting panels to create a reactive environment, synthesize tones for my invisible chimes project, and do some other synth experiments. This brief subtractive synth test uses filters to shape pink noise into hazy tones forming a chord. synth2 tinkers with sample playback and ring modulation. Next up, granular synthesis to build some instrumental Christmas music?

Other Useful Audio Software

Jack is a great tool for routing inputs and outputs on your system. It has made it really simple, for example, to send the output of iTunes to Pure Data, which allows me to sample chunks, process them, and mix it back into songs playing. I made a small program that samples chunks of the last song and then injects them into the new song when it detects beats. It also supports plugins, so you could use Pure Data as a signal processor for other programs like Logic.

Wiretap Studio is really useful for capturing any sound source on your system, doing basic waveform editing, fades, effects, and exporting to any other sound file format.

RjDj for the iPhone is a program that lets you download (and create, using Pd) “scenes” that generate music or process environmental sound and play it through the headphones. For example, one might identify that fan humming along at 300 Hz and re-inject overtones to change its timbre.

The first piece I have made to explore this concept features invisible wind chimes. When a person walks through the space occupied by the virtual chimes, they begin to ring. Should the actor stop to explore the source of the sound, he will uncover a fine-grained, predictable, knowable system. Using computer vision techniques to detect movement in a space, real, physical objects are able to interact with virtual, “physical” ones. Although not concrete, they are still physical because a physics simulation ensures the individual chimes still hang behave in ways we know and expect. They hang from the ceiling by strings, collide with each other (causing them to ring), swing and respond to gravity by eventually settling back to rest, and respond to touch by a soft body (muting any ringing).

Invisible Chimes is an intangible, interactive system that prompts us to consider how we acquire knowledge of complex systems from limited experiential data. The chimes, though installed in a space, cannot be seen, and while they can be touched, they cannot be felt. The only indication of their presence is the sound they make. Nevertheless, the system is still very much a “physical” one, in that the chimes are governed by the laws of physics. Although they can be appreciated by simply walking through them, further interaction reveals that the individual chimes can be separated, muted, and lifted, and that they behave in a predictable, understandable way. The inset video reveals the physical model the custom software uses to create the interactive system.