Alex McLean (Goldsmiths), Dave Griffiths (FoAM), Nick Collins (University of Sussex) and Geraint Wiggins (Goldsmiths)
In this paper we outline the issues surrounding live coding which is projected for an audience, and in this context, approaches to code visualisation. This includes natural language parsing techniques, using geometrical properties of space in language semantics, representation of execution flow in live coding environments, code as visual data and computer games as live coding environments. We will also touch on the unifying perceptual basis behind symbols, graphics, movement and sound.
I’ve been thinking about visual languages and the morphology of symbols (as opposed to words) for a while. I had the opportunity to start putting some of these ideas into code at a really excellent openframeworks workshop this week, run by Joel Gethin Lewis and Arturo Castro.
Here’s what it does:
Makes the point nicely that symbols and spaces can intertwine.
Using opencv blob detection, the regularity, direction and area of the shapes map to envelope modulation, resonance and pitch. The drawing is then sequenced into a melody using the minimum spanning tree (from the boost library) of the shape centroids, where distance maps to inter-onset interval.
It also has a mode for projecting the red circles and highlights back on the drawing surface which worked well.
This is only the second thing I’ve made with openframeworks, and while I don’t really get on with the codeblocks editor recommended for linux, I’m impressed with how accessible it makes opencv and all that.
“Broadly, it is my desire to express emotions by means of timed variations of light and color in a manner analogous to that employed in the art of music. Such expression may either be for its own sake, or … as an accompaniment.”
In 1906, about 40 years after the invention of the commercial light bulb, Mary Hallock-Greenwalt (1871-1950) began work on her colour organ, the sarabet. She was an accomplished musician, but wanted to create an equivalent artform for colour which she called nourathar (derived from the Arabic for the essense/flavour/influence of light). Interestingly though, she came to the conclusion that colour wasn’t as important as brightness:
“In this art they, the darknesses and brightnesses, constitute the woof of the play. They carry a chief burden of the transmitted feeling. They also tend to make a oneness out of more than one colour, or colours, simultaneously produced.”
She was also quick to dismiss the idea of direct mappings between music and colour.
“… there is no octave to color. color has no harmonics. … Its pristine strength is such that no two colors can fit together as identical.”
While she composed nourathar pieces to accompany music, she was against cross-domain mappings in general.
“To seek to fasten the form of one art on the form of another art, is, on the face of it, a mistake, if not an impossibility. They are organically different things. They will speak in different ways.”
I’m not sure if I agree with this strong claim, the human senses are integrated after all. But I still think it insightful to reject the naive “colour scales” which others came up with — while synaesthetics can experience pitch as colour (or vice versa), I understand that no two synaesthetics experience the same scale.
All the quotes in this post came from Hallock-Greenewalt’s book “Nourathar: The Fine Art of Light-Color Playing”, which is a joy to read. She was not the first colour organist, but from what I’ve seen she was the most insightful and interesting of the bunch. Sadly however no video recordings can exist from back then, so we can only imagine what her performances could have been like, with only her notation to guide us.
I gave a quick dorkbot presentation about Mary Hallock-Greenewalt a couple of years ago, and one audience member jokingly accused me of inventing her to justify VJ culture with false history. Well her work is well documented in her writing and patent applications, but she should certainly be better known — I recently saw a talk about colour organists which didn’t mention her, despite her huge contribution to the field.
I’ll finish this post with one last quote from the woman herself.
“Is there no expressing of fervor in the deepening of a rose to red? Can quality of ardor not be suggested in the quickness or slowness with which this transition is done? Can zeal or eagerness not be expressed in the manner of change from blue to purple? Are colors not “warm” or “cold”? Is there not the fervid, the burning of intensity of feeling in the ray’s glowing into or embering back? So much there is to choose from.”
The space has two quality dimensions, “intensity” (X) and “disorder” (Y). Drum patterns are arranged along these dimensions, so more intense ones are towards the left and more ordered ones towards the top.
Draw a line from a high hat to a kick drum. If you draw a short line the rhythms will be more homogenous. Certain angles have certain feels to them. Maybe. It seems a nice way of playing with polymetric rhythms as vectors anyway.
The demo by the way is coded in haxe and compiled to flash. The source code is here.
Then run the Perl script over it to produce a .pd file, that you can then load into puredata to get this:
The ASCII syntax basically allows you to define pd objects and connect them together. Layout is preserved. Much like in ghostbusters, you can’t cross the lines, and there isn’t syntax for different box types (messages and numbers). Fixing this would be short work, but I ran out of train journey :)
There is a particular syntax for drawing the lines. You use - for going left and right, | for going up and down and \ and / for going diagonally. To change direction or fork a wire you have to place a * . Mark inlets and outlets with x .
I think this shows nicely how there is no real difference between patching and coding. Shades of pixels are an alphabet, anything can be a program if you define a suitable interpreter to go with it.
Sadly you can’t do live patching with this, but perhaps this could be a starting point for thinking about more interesting ways of programming with text.
I’ve returned to this subject, having many good ideas to explore from recent discussions with Tim Blackwell. We thought rendering some whole songs would work nicely. I didn’t fancy playing with my Java code again so wrote some Haskell, which I’m rather pleased with. The source is available (feedback welcome!). It does the the mapping using seeks on the output file, allowing impressive memory efficiency via Haskell’s lazy evaluation.
Some examples of some indie synth pop, disco, minimal techno (*3) and industrial gabba below, click on the images for the full versions but beware, they are rather large, around 5M each. Mouseover for the original track names.
Woven sound is an idea by Dr Tim Blackwell, where a one-dimensional stream of audio samples or midi events may be woven into a two-dimensional structure analogous to fabric. Tim has written this idea into his software, where (as I understand it) he uses flocking algorithms to seek out patches of high activity which are then unwoven back into sound.
Inspired by this I have made my own implementation of woven sound. It doesn’t produce very interesting audio output yet but so far the animated visualisation is pleasing.
My idea is to have autonomous agents running around the fabric at audio rate, changing the rules they follow on the fly. Not quite there yet.
As well as weaving the sound in a traditional manner (warp and weft?) my implementation can also weave in a Peano curve. I made a prototype which draws the Peano curve in processing, which helps see its structure. The movement is complex but the idea extremely simple; is to take a line, twist it in a figure of eight, then do the same with each new line segment recursively. Infinite recursions would fill a 2D square completely, but here I limit the recursions to 4 or 5.
These screengrabs give a general idea but to see the full effect and the relationship between the woven sound and the sound source, plug in your microphone, download my software and make some noise. The java sourcecode is in the jar file.
All of my software mentioned here is copyright 2006, available under the terms of the GPL version 2.0.