Archive for the ‘visualisation’ Category

PhD Thesis: Artist-Programmers and Programming Languages for the Arts

Wednesday, February 22nd, 2012

With some minor corrections done, my thesis is finally off to the printers.  I’ve made a PDF available, and here’s the abstract:

We consider the artist-programmer, who creates work through its description as source code. The artist-programmer grandstands computer language, giving unique vantage over human-computer interaction in a creative context. We focus on the human in this relationship, noting that humans use an amalgam of language and gesture to express themselves. Accordingly we expose the deep relationship between computer languages and continuous expression, examining how these realms may support one another, and how the artist-programmer may fully engage with both.

Our argument takes us up through layers of representation, starting with symbols, then words, language and notation, to consider the role that these representations may play in human creativity. We form a cross-disciplinary perspective from psychology, computer science, linguistics, human-computer interaction, computational creativity, music technology and the arts.

We develop and demonstrate the potential of this view to inform arts practice, through the practical introduction of software prototypes, artworks, programming languages and improvised performances. In particular, we introduce works which demonstrate the role of perception in symbolic semantics, embed the representation of time in programming language, include visuospatial arrangement in syntax, and embed the activity of programming in the improvisation and experience of art.

Feedback is very welcome!

BibTeX record:

@phdthesis{McLean2011,
    title = {{Artist-Programmers} and Programming Languages for the Arts},
    author = {McLean, Alex},
    month = {October},
    year = {2011},
    school = {Department of Computing, Goldsmiths, University of London}
}

RIS record:

TY  - THES
ID  - McLean2011
TI  - Artist-Programmers and Programming Languages for the Arts
PB  - Department of Computing, Goldsmiths, University of London
AU  - McLean, Alex
PY  - 2011/10/01

Attending to presentation slides

Friday, August 26th, 2011

I had some fun with my talk at ICMC earlier this month.

I started in the usual way with an outline slide, going through bullet points one by one outlining the structure of my talk.  Importantly, I tried to talk continuously while the slide was up.

On the next slide was a picture of a boy throwing a stone into the sea, I talked about it for a while, making the point that it was easy to perceive the image while listening to my voice.  The audience hopefully found they could attend simultaneously to the visual scene and my linguistic speech.

I then skipped back to the previous slide and pointed out that the outline slide actually had little to do with what I had been saying.  Here’s the contents of that first slide:

  • A live coding talk towards the end of the conference
  • Some strange programming languages were shown
  • He made a point about cognition that I didn’t quite get
  • The demo didn’t work out too well
  • I was a bit tired but he seemed to be trying to say something about syntax

This got some laughs.  There were quite a lot of people in the room, and the slide had been up for a while, but as far as I could gather no-one had managed to read any of it.  My contention was that they couldn’t read it while listening to my voice, it’s too difficult to attend to two streams of language at once.  I didn’t really know what would happen, but from talking to audience members afterwards it seems at least some people got a sense that something was wrong, but couldn’t work out what it was until I told them.

This was a nice practical demonstration of Dual Coding theory, and lead into my argument for greater integration between visual and linguistic elements of computer languages.  However there’s probably a point in there about the design of presentation slides.  If you want people to listen to what you’re saying, put short prompts on your slides, but not real sentences, because the audience won’t be able read them while listening to your voice.

 

Workshop output

Monday, February 7th, 2011

The Text live coding workshop went really well, surprisingly well considering it was the first time anyone apart from me had used it and (so I found out after) most of the participants didn’t have any programming experience. The six participants took to the various combinators surprisingly quickly, the main stumbling block being getting the functions to connect in the right way… Some UI work to do there, and I got some valuable feedback on it.

Once the participants had got the hang of things on headphones, we all switched to speakers and the seven of us played acid techno for an hour or so together, in perfect time sync thanks to netclock. Here’s a mobile phone snippet:

The sound quality doesn’t capture it there, but for me things got really interesting musically, and it was fun walking around the room panning between the seven players…

Text update and source

Monday, January 31st, 2011

I’ve updated Text a bit to improve the visual representation of higher order types (you’d probably need to full screen to view):

I won’t be touching this until after the workshop on Saturday.

I’ve also made the source for the visual interface available here under the GPLv3 free license. To get it actually working as above you’d also need to install my tidal library, Jamie Forth’s network sync, my sampler, the nekobee synth, and somehow get it all working together. In short, it’s a bit tricky, I’ll be working on packaging soonish though.

Text

Monday, December 13th, 2010

Text is a experimental visual language under development.  Code and docs will appear here at some point, but all I have for now is this video of a proof of concept.

It’s basically Haskell but with syntax based on proximity in 2D space, rather than adjacency.  Type compatible things connect automatically, made possible though Haskell’s strong types and currying.  I implemented the interface in C, using clutter, and ended up implementing a lot of Haskell’s type system.  Whenever something changes it compiles the graph into Haskell code, which gets piped to ghci.  The different colours are the different types.  Stripes are curried function parameters.  Lots more to do, but I think this could be a really useful system for live performance.

Visualisation of Live Code

Saturday, May 29th, 2010

I wrote a paper with Dave Griffiths and Nick Collins on the visualisation of live code, exploring ideas around live coding interfaces, accepted for the EVA London 2010 conference in July. A HTML version is below, or see the PDF Preprint.


Alex McLean (Goldsmiths), Dave Griffiths (FoAM), Nick Collins (University of Sussex) and Geraint Wiggins (Goldsmiths)

Abstract

In this paper we outline the issues surrounding live coding which is projected for an audience, and in this context, approaches to code visualisation. This includes natural language parsing techniques, using geometrical properties of space in language semantics, representation of execution flow in live coding environments, code as visual data and computer games as live coding environments. We will also touch on the unifying perceptual basis behind symbols, graphics, movement and sound.

(more…)

Acid sketching

Sunday, November 8th, 2009

I’ve been thinking about visual languages and the morphology of symbols (as opposed to words) for a while. I had the opportunity to start putting some of these ideas into code at a really excellent openframeworks workshop this week, run by Joel Gethin Lewis and Arturo Castro.

Here’s what it does:

Makes the point nicely that symbols and spaces can intertwine.

Using opencv blob detection, the regularity, direction and area of the shapes map to envelope modulation, resonance and pitch. The drawing is then sequenced into a melody using the minimum spanning tree (from the boost library) of the shape centroids, where distance maps to inter-onset interval.

It also has a mode for projecting the red circles and highlights back on the drawing surface which worked well.

This is only the second thing I’ve made with openframeworks, and while I don’t really get on with the codeblocks editor recommended for linux, I’m impressed with how accessible it makes opencv and all that.

Update: sourcecode

Mary Hallock-Greenewalt

Tuesday, March 24th, 2009
Hallock-Greenwalt at the sarabet

Hallock-Greenewalt at the sarabet

“Broadly, it is my desire to express emotions by means of timed variations of light and color in a manner analogous to that employed in the art of music. Such expression may either be for its own sake, or … as an accompaniment.”

In 1906, about 40 years after the invention of the commercial light bulb, Mary Hallock-Greenwalt (1871-1950) began work on her colour organ, the sarabet.  She was an accomplished musician, but wanted to create an equivalent artform for colour which she called nourathar (derived from the Arabic for the essense/flavour/influence of light).  Interestingly though, she came to the conclusion that colour wasn’t as important as brightness:

“In this art they, the darknesses and brightnesses, constitute the woof of the play.  They carry a chief burden of the transmitted feeling.  They also tend to make a oneness out of more than one colour, or colours, simultaneously produced.”

She was also quick to dismiss the idea of direct mappings between music and colour.

“… there is no octave to color.  color has no harmonics. … Its pristine strength is such that no two colors can fit together as identical.”

While she composed nourathar pieces to accompany music, she was against cross-domain mappings in general.

“To seek to fasten the form of one art on the form of another art, is, on the face of it, a mistake, if not an impossibility.  They are organically different things.  They will speak in different ways.”

I’m not sure if I agree with this strong claim, the human senses are integrated after all.  But I still think it insightful to reject the naive “colour scales” which others came up with — while synaesthetics can experience pitch as colour (or vice versa), I understand that no two synaesthetics experience the same scale.

An example of Hallock-Greenewalt's patented notation system

An example of Hallock-Greenewalt's patented notation system

All the quotes in this post came from Hallock-Greenewalt’s book “Nourathar: The Fine Art of Light-Color Playing”, which is a joy to read.  She was not the first colour organist, but from what I’ve seen she was the most insightful and interesting of the bunch.  Sadly however no video recordings can exist from back then, so we can only imagine what her performances could have been like, with only her notation to guide us.

I gave a quick dorkbot presentation about Mary Hallock-Greenewalt a couple of years ago, and one audience member jokingly accused me of inventing her to justify VJ culture with false history.  Well her work is well documented in her writing and patent applications, but she should certainly be better known — I recently saw a talk about colour organists which didn’t mention her, despite her huge contribution to the field.

I’ll finish this post with one last quote from the woman herself.

“Is there no expressing of fervor in the deepening of a rose to red? Can quality of ardor not be suggested in the quickness or slowness with which this transition is done?  Can zeal or eagerness not be expressed in the manner of change from blue to purple?  Are colors not “warm” or “cold”?  Is there not the fervid, the burning of intensity of feeling in the ray’s glowing into or embering back?  So much there is to choose from.”

Posted on the occasion of Ada Lovelace day 2009.

Rhythm space

Friday, April 18th, 2008

I’m working with Jamie Forth on ideas around spaces of rhythm. Here’s a demo (which might not work in feed readers):

[kml_flashembed movie="http://doc.gold.ac.uk/~ma503am/software/space/audio.swf" height="300" width="400" bgcolor="#000000" /]

The space has two quality dimensions, “intensity” (X) and “disorder” (Y). Drum patterns are arranged along these dimensions, so more intense ones are towards the left and more ordered ones towards the top.

Draw a line from a high hat to a kick drum. If you draw a short line the rhythms will be more homogenous. Certain angles have certain feels to them. Maybe. It seems a nice way of playing with polymetric rhythms as vectors anyway.

The demo by the way is coded in haxe and compiled to flash. The source code is here.

Textual patching

Monday, January 7th, 2008

I wrote a perl script that allows you to compose puredata patches in a text editor. You define the patch using ASCII art like this:


    *------------------------*
    |           .--------.    \
  .-x--------.  | osc~ 5 |     *
  | osc~ 500 |  `-x------'     |
  `-x--------'    |            |
    |           .-x------.     |
    |           | *~ 300 |     |
    |           `-x------'     |
    *---*         |            |
        |         *------------*
      .-x------.
      | *~ 0.2 |
      `-x------'
        |
        *
        |\
        | *
        | |
      .-x-x--.
      | dac~ |
      `------'

Then run the Perl script over it to produce a .pd file, that you can then load into puredata to get this:

patch.png

The ASCII syntax basically allows you to define pd objects and connect them together. Layout is preserved. Much like in ghostbusters, you can’t cross the lines, and there isn’t syntax for different box types (messages and numbers). Fixing this would be short work, but I ran out of train journey :)

There is a particular syntax for drawing the lines. You use - for going left and right, | for going up and down and \ and / for going diagonally. To change direction or fork a wire you have to place a * . Mark inlets and outlets with x .

I think this shows nicely how there is no real difference between patching and coding. Shades of pixels are an alphabet, anything can be a program if you define a suitable interpreter to go with it.

Sadly you can’t do live patching with this, but perhaps this could be a starting point for thinking about more interesting ways of programming with text.

If you are curious you can download the script (and patch) here.