The first piece I have made to explore this concept features invisible wind chimes. When a person walks through the space occupied by the virtual chimes, they begin to ring. Should the actor stop to explore the source of the sound, he will uncover a fine-grained, predictable, knowable system. Using computer vision techniques to detect movement in a space, real, physical objects are able to interact with virtual, “physical” ones. Although not concrete, they are still physical because a physics simulation ensures the individual chimes still hang behave in ways we know and expect. They hang from the ceiling by strings, collide with each other (causing them to ring), swing and respond to gravity by eventually settling back to rest, and respond to touch by a soft body (muting any ringing).

Invisible Chimes is an intangible, interactive system that prompts us to consider how we acquire knowledge of complex systems from limited experiential data. The chimes, though installed in a space, cannot be seen, and while they can be touched, they cannot be felt. The only indication of their presence is the sound they make. Nevertheless, the system is still very much a “physical” one, in that the chimes are governed by the laws of physics. Although they can be appreciated by simply walking through them, further interaction reveals that the individual chimes can be separated, muted, and lifted, and that they behave in a predictable, understandable way. The inset video reveals the physical model the custom software uses to create the interactive system.

The trackmate system examines an image (in this case coming from a cheap webcam) and looks for special coded labels. Each label contains a unique ID and also encodes its rotational direction. When trackmate identifies a label, it sends the id, position, and rotation to my software in processing which in turn draws lines.

The screenshot was taking during an interactive session where one person was controlling the color, the other was controlling the opacity, and the third was moving the brush.

The lines in the drawing are broken up due to the low resolution of the camera (this could have possibly been solved with interpolation). I need to calibrate the camera again (and really, buy a cam with a res higher than 640×480), and find a better surface that will filter some of the ‘noise’ from the scene. Still, trackmate behaved remarkably well given the circumstances.

The trackmate website has good tutorials on how to get a system up and running. My source code is provided to help get you started if you want to play around with the system, although I accidentally lost the final revision, so the code provided may not function as expected.

In a TED talk by Golan Levin, he presented an interactive project that highlighted the negative (or interstitial) spaces of cast shadows. This idea seemed like a natural way to segment a generic image, so I decided to use blobs (uninterrupted regions of an image) as the basis for generating tones. In this processing sketch, the system finds three areas of a particular size and assigns each a tone according to its size associating larger pieces with lower tones.  The colors indicate the three sound-generating regions.

The applet has been posted along with the source, although you may have to run it locally in processing to work properly.  You will need a video camera attached to your computer.  Listen to a sound clip from the synth.

An experimental, interactive audio synthesizer in which independent generators combine to create an ambient soundscape as they try to reach a state of equilibrium.

Click to listen to a sample clip, play with the applet, or view the source.

In this interactive visualization, a musician watches the screen while performing on a keyboard. As notes are played, ‘raindrops’ begin to fall. Louder notes fall faster than softer ones and continue to accelerate as the note is sustained. Eventually, notes will blacken if they have been falling for an extended period, and will slowly dissipate unless cleared away by fresh rain from above.

The real-time feedback causes the musician to alter what is being played in order to generate particular audio and visual outcomes simultaneously.

Music in this demo is Final Transmission by Luke Loeffler