The first piece I have made to explore this concept features invisible wind chimes. When a person walks through the space occupied by the virtual chimes, they begin to ring. Should the actor stop to explore the source of the sound, he will uncover a fine-grained, predictable, knowable system. Using computer vision techniques to detect movement in a space, real, physical objects are able to interact with virtual, “physical” ones. Although not concrete, they are still physical because a physics simulation ensures the individual chimes still hang behave in ways we know and expect. They hang from the ceiling by strings, collide with each other (causing them to ring), swing and respond to gravity by eventually settling back to rest, and respond to touch by a soft body (muting any ringing).

Invisible Chimes is an intangible, interactive system that prompts us to consider how we acquire knowledge of complex systems from limited experiential data. The chimes, though installed in a space, cannot be seen, and while they can be touched, they cannot be felt. The only indication of their presence is the sound they make. Nevertheless, the system is still very much a “physical” one, in that the chimes are governed by the laws of physics. Although they can be appreciated by simply walking through them, further interaction reveals that the individual chimes can be separated, muted, and lifted, and that they behave in a predictable, understandable way. The inset video reveals the physical model the custom software uses to create the interactive system.

The persistence of the lines could be affected by altering the opacity and the radius of the contour lines could be changed by setting the thresholding point, which is the cutoff point or “line in the sand” by which we determine what white and black are. Imagine flooding a mountainous area with water: everything above the water line is considered “dry” and everything below, “wet.”  If you walk along the shoreline, or contour, add more water and then walk the shore line again, you will find a much smaller ring the second time.  If you were to plot the path you took for each of these rings, you would have a topographic map.

Continuing with this idea, I automated the process and drew all the contours on each image, leading to a series of images like this:

Next I added all the points to 3D space and set up some basic camera controls to allow exploring the mesh and ended up with a video like this, which felt a lot like those old-school flight simulators from the 80’s.  There’s something very amusing about flying over your face…

In a TED talk by Golan Levin, he presented an interactive project that highlighted the negative (or interstitial) spaces of cast shadows. This idea seemed like a natural way to segment a generic image, so I decided to use blobs (uninterrupted regions of an image) as the basis for generating tones. In this processing sketch, the system finds three areas of a particular size and assigns each a tone according to its size associating larger pieces with lower tones.  The colors indicate the three sound-generating regions.

The applet has been posted along with the source, although you may have to run it locally in processing to work properly.  You will need a video camera attached to your computer.  Listen to a sound clip from the synth.

An experimental, interactive audio synthesizer in which independent generators combine to create an ambient soundscape as they try to reach a state of equilibrium.

Click to listen to a sample clip, play with the applet, or view the source.

Once the data was collected, a number of scripts were written in Python to extract, convert, clean, and export the data in addition to performing some basic analysis. Finally, I used the Prefuse Flare library to create an animation at 40x real time of the interaction between people and key topics. Below is a still frame from the animation.  Click to load the animation.

Special thanks to David Friedlander, Chris Kennedy, Stephanie Nikolopoulos, and Reid Rogers for collecting the data (taking notes) during the event!

If you’re interested in a technical description of the code, read on. The main hurdle to getting this completed was implementing a fast “nearest neighbor” system. From the chroma-key footage (”green screen” as it’s often called), I threshold it (which means turn it to black and white). Next, I perform edge detection which leaves just the outline to the dancer. By altering the edge detector’s “kernel,” a mathematical term for a table of numbers, I can degrade the outline so it is a series of dots rather than one solid line. Next, I go through the image and make note of the locations of all the points. All that’s left to do is to connect them with a continuous line. However, if you simply connect the dots in the order they were found (top to bottom, left to right), you will get a zig-zag line that fills the area of the dancer. The next logical step is to take a starting point and ask for it’s nearest neighbor and connect the two. Then continue the line to the nearest neighbor after that, and so on. Unfortunately, this naive approach does not scale well and takes an enormous amount of time to compute. Enter the KD Tree, a method of storing the points for fast retrieval. I had started to write my own implementation last year and got distracted. Thankfully, I found a simple implementation for Java which worked perfectly. It’s beautiful to see a series of techniques converge to produce something entirely new.

The source code is provided in two files: dancers_outlined.pde and Tree.java for reference. Please be warned–it is very rough and not intended for public consumption, although I tried to comment as much as possible.

This image represents my breathing pattern for seven minutes. By reading the motion sensor in my laptop, I was able to plot the rise and fall of my chest. Each circle represents five seconds of data. This was an experiment to see whether the sensor was sensitive enough to detect respiration.

If you would like to experiment with using the sudden motion sensor (SMS) in an Apple MacBook (or MacBook Pro), my Processing source code is provided here. Please note that you must download the sms library, available from the Processing website. Commented out is code to turn the laptop into an etch-a-sketch, where you draw by tilting the laptop in various directions.

The complete series features images from early versions of the code as well as images illustrating the drawing algorithms.

In this interactive visualization, a musician watches the screen while performing on a keyboard. As notes are played, ‘raindrops’ begin to fall. Louder notes fall faster than softer ones and continue to accelerate as the note is sustained. Eventually, notes will blacken if they have been falling for an extended period, and will slowly dissipate unless cleared away by fresh rain from above.

The real-time feedback causes the musician to alter what is being played in order to generate particular audio and visual outcomes simultaneously.

Music in this demo is Final Transmission by Luke Loeffler

Additional images in this series may be found in my flickr stream.

The processing source code.