My work has been developing visualizations that utilize the new physical display in different ways. The first visualizes air traffic over four major airports in the US, drawing moving lines to indicate aircraft taking off and landing. The second, entitled Order/Disorder, displayed below aggregates a number of sources for natural disaster information and draws the disasters as “tears in the fabric” of the world. Lastly, I am working to create software that reacts to the environment and responds by flashing a series of animations developed by Marshall Reese.

A program in Processing communicates with the hardware and serves as a hub accepting drawing commands from other software via OSC. A number of scripts in Python perform data aggregation, scraping, parsing and animation for the flight tracker. A patch in Pd performs sound analysis and drives the canvases through OSC.

“Order/Disorder is a software visualization of destructive and restorative forces in the world. The software runs on an electronically-controlled tapestry of woven fiberoptic threads created by New York-based artists Ligorano/Reese. Two computer programs, named order and disorder, modify the tapestry throughout the day in response to natural and man-caused events such as earthquakes, biohazards, and aircraft accidents. Order seeks to weave a rainbow-gradated pattern representing peace and wholeness while Disorder seeks to destroy and unravel the tapestry by “tearing” the fabric and weaving in aberrant threads.”

The first piece I have made to explore this concept features invisible wind chimes. When a person walks through the space occupied by the virtual chimes, they begin to ring. Should the actor stop to explore the source of the sound, he will uncover a fine-grained, predictable, knowable system. Using computer vision techniques to detect movement in a space, real, physical objects are able to interact with virtual, “physical” ones. Although not concrete, they are still physical because a physics simulation ensures the individual chimes still hang behave in ways we know and expect. They hang from the ceiling by strings, collide with each other (causing them to ring), swing and respond to gravity by eventually settling back to rest, and respond to touch by a soft body (muting any ringing).

Invisible Chimes is an intangible, interactive system that prompts us to consider how we acquire knowledge of complex systems from limited experiential data. The chimes, though installed in a space, cannot be seen, and while they can be touched, they cannot be felt. The only indication of their presence is the sound they make. Nevertheless, the system is still very much a “physical” one, in that the chimes are governed by the laws of physics. Although they can be appreciated by simply walking through them, further interaction reveals that the individual chimes can be separated, muted, and lifted, and that they behave in a predictable, understandable way. The inset video reveals the physical model the custom software uses to create the interactive system.

The persistence of the lines could be affected by altering the opacity and the radius of the contour lines could be changed by setting the thresholding point, which is the cutoff point or “line in the sand” by which we determine what white and black are. Imagine flooding a mountainous area with water: everything above the water line is considered “dry” and everything below, “wet.”  If you walk along the shoreline, or contour, add more water and then walk the shore line again, you will find a much smaller ring the second time.  If you were to plot the path you took for each of these rings, you would have a topographic map.

Continuing with this idea, I automated the process and drew all the contours on each image, leading to a series of images like this:

Next I added all the points to 3D space and set up some basic camera controls to allow exploring the mesh and ended up with a video like this, which felt a lot like those old-school flight simulators from the 80’s.  There’s something very amusing about flying over your face…

The trackmate system examines an image (in this case coming from a cheap webcam) and looks for special coded labels. Each label contains a unique ID and also encodes its rotational direction. When trackmate identifies a label, it sends the id, position, and rotation to my software in processing which in turn draws lines.

The screenshot was taking during an interactive session where one person was controlling the color, the other was controlling the opacity, and the third was moving the brush.

The lines in the drawing are broken up due to the low resolution of the camera (this could have possibly been solved with interpolation). I need to calibrate the camera again (and really, buy a cam with a res higher than 640×480), and find a better surface that will filter some of the ‘noise’ from the scene. Still, trackmate behaved remarkably well given the circumstances.

The trackmate website has good tutorials on how to get a system up and running. My source code is provided to help get you started if you want to play around with the system, although I accidentally lost the final revision, so the code provided may not function as expected.

To use the code you’ll need a copy of Processing. Shine a bright flashlight against your thumbnail (I used a bright LED flashlight) and place your thumb over the webcam lens so it completely blocks it. Try to hold as still as possible.  The brightness of the red channel is measured each frame and a running average is kept. If the brightness drops below the recent average, a small circle is drawn representing a contracting heart.

If it is unreliable, you may need to tweak the navg parameter and/or find a way to stabilize the light source, finger, and camera, as small shifts will cause a change in the amount of light detected by the camera.

Download the Code

In a TED talk by Golan Levin, he presented an interactive project that highlighted the negative (or interstitial) spaces of cast shadows. This idea seemed like a natural way to segment a generic image, so I decided to use blobs (uninterrupted regions of an image) as the basis for generating tones. In this processing sketch, the system finds three areas of a particular size and assigns each a tone according to its size associating larger pieces with lower tones.  The colors indicate the three sound-generating regions.

The applet has been posted along with the source, although you may have to run it locally in processing to work properly.  You will need a video camera attached to your computer.  Listen to a sound clip from the synth.

An experimental, interactive audio synthesizer in which independent generators combine to create an ambient soundscape as they try to reach a state of equilibrium.

Click to listen to a sample clip, play with the applet, or view the source.

You can find a demo and the source here. Click four times to set the curve points and handles. Move the mouse left and right to alter the curve tightness.

In a nutshell, the algorithm computes the approximate length of the line segment by chopping it into smaller segments and computing the linear distance between points.  Once the length of the segment is known in pixels, textWidth() is called to obtain the width in pixels of each character. This width divided by the length of the curve determines the percentage of the curve the character occupies. Once this is know, the amount to increment the variable t in the parametric curvePoint(…) function is known. From here, we can find the starting point, determine the tangent, and draw the letter.

All straightforward, right? For whatever reason, though, the kerning is severely off under certain conditions. When there is a slight curve in the line segment, it seems to be balanced. When the curve tightness is altered and becomes very flat, or very curved, the letters become overly spaced out at one end, and overly compressed at the other.  Does anyone have any suggestions as to why this is happening?

If you’re interested in a technical description of the code, read on. The main hurdle to getting this completed was implementing a fast “nearest neighbor” system. From the chroma-key footage (”green screen” as it’s often called), I threshold it (which means turn it to black and white). Next, I perform edge detection which leaves just the outline to the dancer. By altering the edge detector’s “kernel,” a mathematical term for a table of numbers, I can degrade the outline so it is a series of dots rather than one solid line. Next, I go through the image and make note of the locations of all the points. All that’s left to do is to connect them with a continuous line. However, if you simply connect the dots in the order they were found (top to bottom, left to right), you will get a zig-zag line that fills the area of the dancer. The next logical step is to take a starting point and ask for it’s nearest neighbor and connect the two. Then continue the line to the nearest neighbor after that, and so on. Unfortunately, this naive approach does not scale well and takes an enormous amount of time to compute. Enter the KD Tree, a method of storing the points for fast retrieval. I had started to write my own implementation last year and got distracted. Thankfully, I found a simple implementation for Java which worked perfectly. It’s beautiful to see a series of techniques converge to produce something entirely new.

The source code is provided in two files: dancers_outlined.pde and Tree.java for reference. Please be warned–it is very rough and not intended for public consumption, although I tried to comment as much as possible.

This image represents my breathing pattern for seven minutes. By reading the motion sensor in my laptop, I was able to plot the rise and fall of my chest. Each circle represents five seconds of data. This was an experiment to see whether the sensor was sensitive enough to detect respiration.

If you would like to experiment with using the sudden motion sensor (SMS) in an Apple MacBook (or MacBook Pro), my Processing source code is provided here. Please note that you must download the sms library, available from the Processing website. Commented out is code to turn the laptop into an etch-a-sketch, where you draw by tilting the laptop in various directions.

The complete series features images from early versions of the code as well as images illustrating the drawing algorithms.

In this interactive visualization, a musician watches the screen while performing on a keyboard. As notes are played, ‘raindrops’ begin to fall. Louder notes fall faster than softer ones and continue to accelerate as the note is sustained. Eventually, notes will blacken if they have been falling for an extended period, and will slowly dissipate unless cleared away by fresh rain from above.

The real-time feedback causes the musician to alter what is being played in order to generate particular audio and visual outcomes simultaneously.

Music in this demo is Final Transmission by Luke Loeffler

Additional images in this series may be found in my flickr stream.

The processing source code.