Mark making can be broken into three categories: addition of material, removal of material, and the modification of a material (although there is a fair amount of overlap between the categories).  Examples of addition were to clamp a bead onto a passing string, to add a blob of glue, or to add a drop of water. Removal of material could mean cutting away, notching, or burning a substrate as it passes. Modification could be knotting the material as it passes, staining it, raising the temperature, etc.

Most media strive to be precise and permanent. By soaking portions of a passing rope with saline, these regions can be detected as they pass “listeners” further down the line. Due to the capillarity of the rope, however, the saline spreads out in several directions which causes slight conductivity and more gently ramping (as opposed to digital) signal. Due to evaporation, the signals get weaker and weaker as they progress through the system yielding unexpected results. Looping everything back around will allow one continuous rope to be used.

Additional possible methods could use a heating element to apply varying amounts of heat to a passing wire. As the heat spreads, the crispness of the signal is diminished and bleeds into adjoining signals. As the heat is transferred to the environment, the overall signal diminishes and returns the wire to its original state.

My work has been developing visualizations that utilize the new physical display in different ways. The first visualizes air traffic over four major airports in the US, drawing moving lines to indicate aircraft taking off and landing. The second aggregates a number of sources for natural disaster information and draws the disasters as “tears in the fabric” of the world, represented by a rainbow gradient. Two forces “compete” for the tapestry, one creative and one destructive. Lastly, I am working to create software that reacts to the environment and responds by flashing a series of animations developed by Marshall Reese.

A program in Processing communicates with the hardware and serves as a hub accepting drawing commands from other software via OSC. A number of scripts in Python perform data aggregation, scraping, parsing and animation for the flight tracker. A patch in Pd performs sound analysis and drives the canvases through OSC.

Since I’ve been learning Puredata, I thought it would be a fun exercise to attempt to recreate the software Ablinger wrote to translate speech to midi notes. The secondary purpose was to turn my oft-idle digital piano into an interactive sound piece, translating sound from another part of the house into music downstairs. The result isn’t perfect, but I think it still achieves the same ambiguous result where you are able to hear the voice once you see the transcript. The biggest difference is obviously that I’m using a digital piano, not a mechanically-actuated analog piano. However, the Roland has a fairly sophisticated physical model with things like dampening and string resonance, so it’s better than nothing.

Below are the two components to the software. Clicking the image will link to the pd file if you’d like to experiment yourself. You can either load in a pre-recorded wave file and play it back, or set the gain to the adc~ to 1 and use a microphone to drive it in real time (although I set up a delay of about 3 seconds so I could evaluate the results without hearing my own voice). delread~ passes the data into fiddle~ which does all the hard work of Fourier analysis. A metronome set to 15 ms samples the outputs of the individual sine components and creates midi notes. The blocks that create the actual notes are partial_key.  The highest key on the piano is midi 108, which corresponds to 4186 Hz, so I added a low pass filter to remove frequencies that couldn’t be reproduced.

partial_key.pd creates the midi notes which are sent to the piano. The signal makenote_b is received from the metronome, which causes the note to be made. Additionally, no note is sent if the midi key number is higher than 108, the limit of my piano, or if the amplitude is too small (< 0.01).

Here is a sample of the result, speaking the following: “these are very profound words, which is why they are being spoken by a piano. I hope you are forever moved by these profound words.”

I’d love any feedback from Puredata or DSP gurus on how the software could be improved. I’m not quite sure what sorts of additional analysis and synthesis steps are being taken in the Quadraturen software as it is not available.

windsong.mp3

This is how I’d like to imagine the first Christmas–even the wind crying out. And doing so subtly, not unlike the whispering symbolic language of babies and cattle stalls.

The idea came about from the desire to manipulate various parts of the built environment to create music as the wind is blowing. Maybe change the orientation of a sheet of metal, the size of some opening, etc. With a feedback loop, the system could progress through a melody once it detected that the desired notes of the sequence had been played. Although a composer could control the content, the weather would determine the tempo. A listener may have to wait weeks or months for the piece to complete without sufficient wind.

In the meantime, I created the above synthetic version of what I imagine it could sound like (though the song above would probably have to be recorded somewhere in Antarctica).

Over the years I’ve played with a number of tools for audio processing: Matlab, jMusic, a Java library for algorithmic composition, Nyquist, a Lisp-based synthesis/analysis environment, Beads, another Java library for synthesis and analysis, and Supercollider, another synth/analysis environment with smalltalk-like syntax. All of these are powerful tools, but aren’t as engaging in terms of interactivity. Having been forced to use LabVIEW in the past, another dataflow language, I was initially reluctant to pick up another, but for audio work, it’s been great. It is so easy to try new ideas without any need to recompile. It’s a lot like playing with a running circuit.

So far I’ve used to to analyze sound and control some lighting panels to create a reactive environment, synthesize tones for my invisible chimes project, and do some other synth experiments. This brief subtractive synth test uses filters to shape pink noise into hazy tones forming a chord. synth2 tinkers with sample playback and ring modulation. Next up, granular synthesis to build some instrumental Christmas music?

Other Useful Audio Software

Jack is a great tool for routing inputs and outputs on your system. It has made it really simple, for example, to send the output of iTunes to Pure Data, which allows me to sample chunks, process them, and mix it back into songs playing. I made a small program that samples chunks of the last song and then injects them into the new song when it detects beats. It also supports plugins, so you could use Pure Data as a signal processor for other programs like Logic.

Wiretap Studio is really useful for capturing any sound source on your system, doing basic waveform editing, fades, effects, and exporting to any other sound file format.

RjDj for the iPhone is a program that lets you download (and create, using Pd) “scenes” that generate music or process environmental sound and play it through the headphones. For example, one might identify that fan humming along at 300 Hz and re-inject overtones to change its timbre.

The first piece I have made to explore this concept features invisible wind chimes. When a person walks through the space occupied by the virtual chimes, they begin to ring. Should the actor stop to explore the source of the sound, he will uncover a fine-grained, predictable, knowable system. Using computer vision techniques to detect movement in a space, real, physical objects are able to interact with virtual, “physical” ones. Although not concrete, they are still physical because a physics simulation ensures the individual chimes still hang behave in ways we know and expect. They hang from the ceiling by strings, collide with each other (causing them to ring), swing and respond to gravity by eventually settling back to rest, and respond to touch by a soft body (muting any ringing).

The persistence of the lines could be affected by altering the opacity and the radius of the contour lines could be changed by setting the thresholding point, which is the cutoff point or “line in the sand” by which we determine what white and black are. Imagine flooding a mountainous area with water: everything above the water line is considered “dry” and everything below, “wet.”  If you walk along the shoreline, or contour, add more water and then walk the shore line again, you will find a much smaller ring the second time.  If you were to plot the path you took for each of these rings, you would have a topographic map.

Continuing with this idea, I automated the process and drew all the contours on each image, leading to a series of images like this:

Next I added all the points to 3D space and set up some basic camera controls to allow exploring the mesh and ended up with a video like this, which felt a lot like those old-school flight simulators from the 80’s.  There’s something very amusing about flying over your face…

The trackmate system examines an image (in this case coming from a cheap webcam) and looks for special coded labels. Each label contains a unique ID and also encodes its rotational direction. When trackmate identifies a label, it sends the id, position, and rotation to my software in processing which in turn draws lines.

The screenshot was taking during an interactive session where one person was controlling the color, the other was controlling the opacity, and the third was moving the brush.

The lines in the drawing are broken up due to the low resolution of the camera (this could have possibly been solved with interpolation). I need to calibrate the camera again (and really, buy a cam with a res higher than 640×480), and find a better surface that will filter some of the ‘noise’ from the scene. Still, trackmate behaved remarkably well given the circumstances.

The trackmate website has good tutorials on how to get a system up and running. My source code is provided to help get you started if you want to play around with the system, although I accidentally lost the final revision, so the code provided may not function as expected.

To use the code you’ll need a copy of Processing. Shine a bright flashlight against your thumbnail (I used a bright LED flashlight) and place your thumb over the webcam lens so it completely blocks it. Try to hold as still as possible.  The brightness of the red channel is measured each frame and a running average is kept. If the brightness drops below the recent average, a small circle is drawn representing a contracting heart.

If it is unreliable, you may need to tweak the navg parameter and/or find a way to stabilize the light source, finger, and camera, as small shifts will cause a change in the amount of light detected by the camera.

Download the Code

In a TED talk by Golan Levin, he presented an interactive project that highlighted the negative (or interstitial) spaces of cast shadows. This idea seemed like a natural way to segment a generic image, so I decided to use blobs (uninterrupted regions of an image) as the basis for generating tones. In this processing sketch, the system finds three areas of a particular size and assigns each a tone according to its size associating larger pieces with lower tones.  The colors indicate the three sound-generating regions.

The applet has been posted along with the source, although you may have to run it locally in processing to work properly.  You will need a video camera attached to your computer.  Listen to a sound clip from the synth.

An experimental, interactive audio synthesizer in which independent generators combine to create an ambient soundscape as they try to reach a state of equilibrium.

Click to listen to a sample clip, play with the applet, or view the source.

Once the data was collected, a number of scripts were written in Python to extract, convert, clean, and export the data in addition to performing some basic analysis. Finally, I used the Prefuse Flare library to create an animation at 40x real time of the interaction between people and key topics. Below is a still frame from the animation.  Click to load the animation.

Special thanks to David Friedlander, Chris Kennedy, Stephanie Nikolopoulos, and Reid Rogers for collecting the data (taking notes) during the event!

You can find a demo and the source here. Click four times to set the curve points and handles. Move the mouse left and right to alter the curve tightness.

In a nutshell, the algorithm computes the approximate length of the line segment by chopping it into smaller segments and computing the linear distance between points.  Once the length of the segment is known in pixels, textWidth() is called to obtain the width in pixels of each character. This width divided by the length of the curve determines the percentage of the curve the character occupies. Once this is know, the amount to increment the variable t in the parametric curvePoint(…) function is known. From here, we can find the starting point, determine the tangent, and draw the letter.

All straightforward, right? For whatever reason, though, the kerning is severely off under certain conditions. When there is a slight curve in the line segment, it seems to be balanced. When the curve tightness is altered and becomes very flat, or very curved, the letters become overly spaced out at one end, and overly compressed at the other.  Does anyone have any suggestions as to why this is happening?

This image was the first experimenting with this process. Pretty ugly. I don’t know what I was going for. It was late.

The second in the series came from a cell image. The fragments from this image seemed to lend themselves to new kinds of creatures. Or maybe it was just the weather, or what I had for dinner. Who knows.

And next, tasty:

Finally, some abstract patterns (one friend saw a headless, demonic horseman–wow):

And finally, these are some fragments from another van gogh, which I decided to turn into a fake map of somewhere.

I think I like this process for the same reasons I enjoyed shuffling around scraps of paper on my desk, or playing with my mashed potatoes as a kid.  Creating meaning from the abstract.

I am using the Carnegie Mellon Pronouncing Dictionary to encode each word. The CMUPD provides a list of 39 phonemes, the unique sounds that comprise spoken English. These are as follows:

Phoneme Example Translation
AA	odd     AA D
AE	at	AE T
AH	hut	HH AH T
AO	ought	AO T
AW	cow	K AW
AY	hide	HH AY D
B 	be	B IY
CH	cheese	CH IY Z
D 	dee	D IY
DH	thee	DH IY
EH	Ed	EH D
ER	hurt	HH ER T
EY	ate	EY T
F 	fee	F IY
G 	green	G R IY N
HH	he	HH IY
IH	it	IH T
IY	eat	IY T
JH	gee	JH IY
K 	key	K IY
L 	lee	L IY
M 	me	M IY
N 	knee	N IY
NG	ping	P IH NG
OW	oat	OW T
OY	toy	T OY
P 	pee	P IY
R 	read	R IY D
S 	sea	S IY
SH	she	SH IY
T 	tea	T IY
TH	theta	TH EY T AH
UH	hood	HH UH D
UW	two	T UW
V 	vee	V IY
W 	we	W IY
Y 	yield	Y IY L D
Z 	zee	Z IY
ZH	seizure	S IY ZH ER

Initially I gave each phoneme a different hue (from 1-360 degrees) by spreading each phoneme out on the color wheel equidistantly. Using this mapping, the phrase “Who knew sniffing glue could give you the flu?” translates to the following image: .  You can clearly see the phoneme UW in magenta appearing at the end of each word.  Unfortunately, there is a lot of visual noise introduced by displaying all the other phonemes.

Realizing not all phonemes are going to be equally represented in the corpus, I decided to find the distribution of each. The sound AH (the “uhh” in the word “hut”) is the  most commonly occurring, accounting for nearly 10% of phonemes (this may be different if the most common 10% of words are analyzed, haven’t looked). This also happens to answer the question I asked my linguistics studies cousin at Christmas dinner some years back after doing imitations of other languages: “So, what does a non-English speaker hear when an American speaks?” It sounds like the answer may be “uhh… duh.. buh… fuh.”

Phon.   Count   Prob.
AH	70564	0.0938934151927
N	53577	0.0712902826622
L	44148	0.0587439274124
S	43349	0.0576807671786
T	41698	0.0554839241923
R	40794	0.0542810495348
K	38174	0.0507948420096
IH	33779	0.0449467954168
IY	30957	0.0411918039527
D	28491	0.0379105109157
M	26330	0.0350350550142
ER	25871	0.0344243033905
EH	24564	0.0326851914686
Z	23955	0.0318748478111
AA	22175	0.0295063556757
AE	19151	0.0254825802726
B	18943	0.0252058126523
P	17305	0.0230262676423
OW	17147	0.0228160306999
G	12248	0.0162973548733
F	12147	0.0161629629038
EY	11851	0.0157691012903
AO	10059	0.0133846417922
AY	9838	0.0130905761956
V	9349	0.0124399061651
NG	8692	0.0115656930567
UW	8579	0.0114153337245
HH	8439	0.0112290478262
W	7737	0.0102949571077
SH	7730	0.0102856428128
JH	5461	0.00726648064689
Y	4392	0.00584405475209
CH	4378	0.00582542616226
AW	2932	0.00390135895563
TH	2597	0.00345560341329
UH	2021	0.00268917000318
OY	1124	0.00149560964056
DH	504	0.000670629233846
ZH	482	0.000641355735543

The next step is to take the frequency distribution into account… Before diving any farther into palette selection, I created a simple test application which you may play with (though be warned, it may be broken and may be an older version).

In this screen shot, syllable stress was taken into account as I experiment with the pre-attentive characteristic of height to aid visualization.

In the following version the previous phrase was repeated with a palette restricted to vowels only and colors assigned according to probability. Phonemes with a higher rate of occurrence received hues that were more distinct from others.

This change appears to have improved the legibility somewhat. More experimentation will be necessary. I suspect this linear format will be a failed experiment. It seems there is little improvement in recognition of rhyme, as the viewer must use working memory to hold what amounts to merely a new representation of sound rather than use pre-attentive factors to quickly match sounds together.

The music is “Glory” by The Acorn.

If you’re interested in a technical description of the code, read on. The main hurdle to getting this completed was implementing a fast “nearest neighbor” system. From the chroma-key footage (”green screen” as it’s often called), I threshold it (which means turn it to black and white). Next, I perform edge detection which leaves just the outline to the dancer. By altering the edge detector’s “kernel,” a mathematical term for a table of numbers, I can degrade the outline so it is a series of dots rather than one solid line. Next, I go through the image and make note of the locations of all the points. All that’s left to do is to connect them with a continuous line. However, if you simply connect the dots in the order they were found (top to bottom, left to right), you will get a zig-zag line that fills the area of the dancer. The next logical step is to take a starting point and ask for it’s nearest neighbor and connect the two. Then continue the line to the nearest neighbor after that, and so on. Unfortunately, this naive approach does not scale well and takes an enormous amount of time to compute. Enter the KD Tree, a method of storing the points for fast retrieval. I had started to write my own implementation last year and got distracted. Thankfully, I found a simple implementation for Java which worked perfectly. It’s beautiful to see a series of techniques converge to produce something entirely new.

The source code is provided in two files: dancers_outlined.pde and Tree.java for reference. Please be warned–it is very rough and not intended for public consumption, although I tried to comment as much as possible.

Below are a few useful tools:

Princeton’s Wordnet is a massive linguistic database containing not just definitions, but how words are related to each other.

The Python Natural Language Toolkit, which provides tools for parsing and understanding natural language semantics.

The Carnegie Mellon Pronouncing Dictionary, which breaks words down into their phonemes, providing information regarding pronouncing, rhyming, and syllable counts.

Having found no simple resource to provide syllable counts for common words, I want to share a quick solution I wrote in Python, which uses the CMU dictionary. Simply download the dictionary to the same directory as this script, naming it cmu_pron.txt.

f = open("cmu_pron.txt")

f = open("cmu_pron.txt")
lines = f.readlines()
f.close()
words = {}

for line in lines:
	pieces = line.split()
	if pieces[0] == ";;;":
		continue
	words[pieces[0]] = pieces[1:]

def num_syllables(word):
	global words
	key = word.upper()
	plist = words[key]
	return len(filter(lambda c: c in ("0","1","2"),"".join(plist)))

print "alphabet: %s" % num_syllables("alphabet")

The result for alphabet is indeed 3. There are probably more efficient ways to do this, but it gets the job done.

With increasing regularity I have heard the following message in a number of ways from a variety of sources: the key to producing is to… produce! In the context of research, the academic world has long used the phrase “publish or perish.” In the realm of design, my architecture studio instructor would tell me to crank out ten iterations of an idea by the next morning—quantity, not quality. Merlin Mann spoke about the distinction between doing and thinking about doing. Seth Godin, in a talk he gave at the NYC Jelly at the Brooklyn Treehouse, said that today we are not faced with a scarcity of ideas, production capability, marketing opportunities, or capital, but an an ability to “ship,” i.e. get work done and out the door. In the new book on the psychology of focus, Rapt, Gallagher argues that an increase in distractions has lead to a decrease in the ability to focus, and without steady focus we are unable to finish work.

So why has this become a recurring theme so recently? Perhaps it is an increased sensitivity to the message due to a period of declining output. Going back farther, I would pin it to the massive and sudden increase in the volume and availability of content, tools, and ideas on the internet. The same system that has catalyzed so much creative work and research in recent years has also sapped the ability of so many to create. I believe it works in several different ways.

First, it is easier than ever to simply waste inordinate amounts of time passively consuming content or talking to friends and acquaintances. We can still feel productive because we are learning, experiencing new things, and forming connections. Secondly, we waste more time than ever deciding what to do and how to do it. With so many new mediums, ideas, forms of expression, tools, platforms, and methods, it easier than ever to get distracted. Lastly, the quality and complexity of things being created today is greater than ever. Much of this is due to the availability of information, increased refinement of tools, and increased collaboration, bringing together the strength of disparate components in symphony. Viewed in the wrong way, as competition rather than inspiration, great work can be demoralizing.

How do you overcome the problem of production? Over the course of a long drive with long-time friend and software developer Jeff, we both agreed that setting deadlines and sticking to them is critical—the shorter the better. When you give yourself too much time to complete something, you will end up second-guessing yourself  and spending far too much time thinking about process, rather than doing work. Time spent will expand to fill the time given. Try giving yourself one day to create something rather than one week, knowing in advance you will probably have to throw it away. You will learn more, gain momentum, and start to realize you’re better equipped than you thought. Additionally, try to schedule work with someone else in advance. This may mean working with a partner or “foolishly” committing to a project you don’t feel quite ready for. Either way, you’re helping your future self avoid weaseling out. Lastly, turn down the volume on distractions. Limit your exposure to media streams, social bookmarking sites, RSS feed readers, and the like to once a week or less to limit the factors of distraction and demoralization. Instead, turn to first inspirations—the world around you.