My work has been developing visualizations that utilize the new physical display in different ways. The first visualizes air traffic over four major airports in the US, drawing moving lines to indicate aircraft taking off and landing. The second, entitled Order/Disorder, displayed below aggregates a number of sources for natural disaster information and draws the disasters as “tears in the fabric” of the world. Lastly, I am working to create software that reacts to the environment and responds by flashing a series of animations developed by Marshall Reese.

A program in Processing communicates with the hardware and serves as a hub accepting drawing commands from other software via OSC. A number of scripts in Python perform data aggregation, scraping, parsing and animation for the flight tracker. A patch in Pd performs sound analysis and drives the canvases through OSC.

“Order/Disorder is a software visualization of destructive and restorative forces in the world. The software runs on an electronically-controlled tapestry of woven fiberoptic threads created by New York-based artists Ligorano/Reese. Two computer programs, named order and disorder, modify the tapestry throughout the day in response to natural and man-caused events such as earthquakes, biohazards, and aircraft accidents. Order seeks to weave a rainbow-gradated pattern representing peace and wholeness while Disorder seeks to destroy and unravel the tapestry by “tearing” the fabric and weaving in aberrant threads.”

Once the data was collected, a number of scripts were written in Python to extract, convert, clean, and export the data in addition to performing some basic analysis. Finally, I used the Prefuse Flare library to create an animation at 40x real time of the interaction between people and key topics. Below is a still frame from the animation.  Click to load the animation.

Special thanks to David Friedlander, Chris Kennedy, Stephanie Nikolopoulos, and Reid Rogers for collecting the data (taking notes) during the event!

I am using the Carnegie Mellon Pronouncing Dictionary to encode each word. The CMUPD provides a list of 39 phonemes, the unique sounds that comprise spoken English. These are as follows:

Phoneme Example Translation
AA	odd     AA D
AE	at	AE T
AH	hut	HH AH T
AO	ought	AO T
AW	cow	K AW
AY	hide	HH AY D
B 	be	B IY
CH	cheese	CH IY Z
D 	dee	D IY
DH	thee	DH IY
EH	Ed	EH D
ER	hurt	HH ER T
EY	ate	EY T
F 	fee	F IY
G 	green	G R IY N
HH	he	HH IY
IH	it	IH T
IY	eat	IY T
JH	gee	JH IY
K 	key	K IY
L 	lee	L IY
M 	me	M IY
N 	knee	N IY
NG	ping	P IH NG
OW	oat	OW T
OY	toy	T OY
P 	pee	P IY
R 	read	R IY D
S 	sea	S IY
SH	she	SH IY
T 	tea	T IY
TH	theta	TH EY T AH
UH	hood	HH UH D
UW	two	T UW
V 	vee	V IY
W 	we	W IY
Y 	yield	Y IY L D
Z 	zee	Z IY
ZH	seizure	S IY ZH ER

Initially I gave each phoneme a different hue (from 1-360 degrees) by spreading each phoneme out on the color wheel equidistantly. Using this mapping, the phrase “Who knew sniffing glue could give you the flu?” translates to the following image: .  You can clearly see the phoneme UW in magenta appearing at the end of each word.  Unfortunately, there is a lot of visual noise introduced by displaying all the other phonemes.

Realizing not all phonemes are going to be equally represented in the corpus, I decided to find the distribution of each. The sound AH (the “uhh” in the word “hut”) is the  most commonly occurring, accounting for nearly 10% of phonemes (this may be different if the most common 10% of words are analyzed, haven’t looked). This also happens to answer the question I asked my linguistics studies cousin at Christmas dinner some years back after doing imitations of other languages: “So, what does a non-English speaker hear when an American speaks?” It sounds like the answer may be “uhh… duh.. buh… fuh.”

Phon.   Count   Prob.
AH	70564	0.0938934151927
N	53577	0.0712902826622
L	44148	0.0587439274124
S	43349	0.0576807671786
T	41698	0.0554839241923
R	40794	0.0542810495348
K	38174	0.0507948420096
IH	33779	0.0449467954168
IY	30957	0.0411918039527
D	28491	0.0379105109157
M	26330	0.0350350550142
ER	25871	0.0344243033905
EH	24564	0.0326851914686
Z	23955	0.0318748478111
AA	22175	0.0295063556757
AE	19151	0.0254825802726
B	18943	0.0252058126523
P	17305	0.0230262676423
OW	17147	0.0228160306999
G	12248	0.0162973548733
F	12147	0.0161629629038
EY	11851	0.0157691012903
AO	10059	0.0133846417922
AY	9838	0.0130905761956
V	9349	0.0124399061651
NG	8692	0.0115656930567
UW	8579	0.0114153337245
HH	8439	0.0112290478262
W	7737	0.0102949571077
SH	7730	0.0102856428128
JH	5461	0.00726648064689
Y	4392	0.00584405475209
CH	4378	0.00582542616226
AW	2932	0.00390135895563
TH	2597	0.00345560341329
UH	2021	0.00268917000318
OY	1124	0.00149560964056
DH	504	0.000670629233846
ZH	482	0.000641355735543

The next step is to take the frequency distribution into account… Before diving any farther into palette selection, I created a simple test application which you may play with (though be warned, it may be broken and may be an older version).

In this screen shot, syllable stress was taken into account as I experiment with the pre-attentive characteristic of height to aid visualization.

In the following version the previous phrase was repeated with a palette restricted to vowels only and colors assigned according to probability. Phonemes with a higher rate of occurrence received hues that were more distinct from others.

This change appears to have improved the legibility somewhat. More experimentation will be necessary. I suspect this linear format will be a failed experiment. It seems there is little improvement in recognition of rhyme, as the viewer must use working memory to hold what amounts to merely a new representation of sound rather than use pre-attentive factors to quickly match sounds together.

This image represents my breathing pattern for seven minutes. By reading the motion sensor in my laptop, I was able to plot the rise and fall of my chest. Each circle represents five seconds of data. This was an experiment to see whether the sensor was sensitive enough to detect respiration.

If you would like to experiment with using the sudden motion sensor (SMS) in an Apple MacBook (or MacBook Pro), my Processing source code is provided here. Please note that you must download the sms library, available from the Processing website. Commented out is code to turn the laptop into an etch-a-sketch, where you draw by tilting the laptop in various directions.

In this interactive visualization, a musician watches the screen while performing on a keyboard. As notes are played, ‘raindrops’ begin to fall. Louder notes fall faster than softer ones and continue to accelerate as the note is sustained. Eventually, notes will blacken if they have been falling for an extended period, and will slowly dissipate unless cleared away by fresh rain from above.

The real-time feedback causes the musician to alter what is being played in order to generate particular audio and visual outcomes simultaneously.

Music in this demo is Final Transmission by Luke Loeffler