The project is called Live Notation: Transforming Matters of Performance, and the first event will be a performance involving Hester and I on Thursday 22nd March as part of the soon-to-be-announced LoveBytes festival (more on that in my next post).  We are not sure what we will do yet, except it will be in a large cinema and involve sound-based dialogue in some way.  It will be an experimental performance (as in risky and prone to failure) and we’ll learn something whatever happens.

Later on we will be holding workshops leading to a big conference/performance event around June/July.

Despite now living in three different cities, slub will manage to perform together on 30th September in La Maison Rouge Paris, as part of the Sony CSL 15th anniversary. Really looking forward to this one.

Then on 28th October I’m honoured to be invited by Aarhus University to give a talk on Artist-Programmers.

On the 15th March 2012 I’m doing some kind of live coding performance at the Life Centre in Newcastle although details aren’t set for that yet.

I’ll also be co-organising another dorkbotsheffield in the next month or so.

That’s it for now..

Experimenting with webcam overlay. Video recorded using gstreamer, source for screencaster here (screensave.c).

UPDATE, here’s another from a different angle to appease douglas.

The last few days I’ve been thinking about revision control.  Revision control systems are really excellent and have a great deal to offer artist-programmers, particularly those working in groups.  What I’ve been wondering though is whether they assume a particular conception of time that doesn’t always apply in the arts.

Consider a live coder, writing software to generate a music performance.  In terms of revision control they are in an unusual situation.  Normally we think of programmers making revisions towards a final result or milestone, at which point they ship. For live coders, every revision they make is part of the final result, and nothing gets shipped, they are already the end users.  We might somewhat crassly think about shipping a product to an audience, but what we’re `shipping’ them isn’t software, but a software development process, as musical development.

Another unusual thing about live coding revisions is that whereas software development conventionally begins with nothing and finishes with a complete, complex structure, a live coder begins and ends with nothing.  Rather than aim for a linear path towards a predefined goal, musicians instead are concerned with how to return to nothing in a satisfying manner.  Indeed perhaps the biggest problem for Live Algorithms is the problem of how to stop playing.  The musician’s challenge is both how to build and how to deconstruct.

There are two ways of thinking about time, either as a linear progression and as a recurrent cycle or oscillation.  Here’s a figure from the excellent book Rhythms of the Brain by György Buzsáki:

“Oscillations illustrate the orthogonal relationship between frequency and time and space and time. An event can repeat over and over, giving the impression of no change (e.g., circle of life). Alternatively, the event evolves over time (pantha rei). The forward order of succession is a main argument for causality. One period (right) corresponds to the perimeter of the circle (left).” (pg. 7)

This illustrates nicely that these approaches aren’t mutually exclusive, they’re just different ways of looking at the same thing.  Indeed it’s normal to think of conventional design processes as cycles of development, with repeating patterns between milestones.  It’s not conventional to think of the code itself ending up back where it started however, but this can happen several times during a music performance, we are all familiar with chorus and verse structure for example, and performances necessarily begin and end at silence.

So where am I going with this?  I’m not sure, but I think there’s plenty of mileage in rethinking revision control for artist-programmers.  There’s already active, radical work in this area, for example the code timeline scrubbing in field looks awesome, and Julian Rohrhuber et al have some great research on time and programming, and have worked on non-linear scheduling of code changes in SuperCollider.

As far as I can see though, the revision control timeline has so far been treated as a linear structure with occasional parts branching and remeeting the main flow later on.  You do sometimes get instances of timelines feeding back on themselves, a process called backporting, but this is generally avoided, only done in urgent circumstances such as for applying security fixes to old code.

What if instead, timelines were of cycles within cycles, with revision control designed not to aid progression towards future features, but help the programmer wrestle their code back towards the state it was in ten minutes ago, and ten minutes before that?  Just questions for now, but I think there is something to be done here.  After all, there is something about artist-programmers, the way they’re caught using general purpose languages and tools in specific, unusual circumstances.

Once the participants had got the hang of things on headphones, we all switched to speakers and the seven of us played acid techno for an hour or so together, in perfect time sync thanks to netclock. Here’s a mobile phone snippet:

The sound quality doesn’t capture it there, but for me things got really interesting musically, and it was fun walking around the room panning between the seven players…

I won’t be touching this until after the workshop on Saturday.

I’ve also made the source for the visual interface available here under the GPLv3 free license. To get it actually working as above you’d also need to install my tidal library, Jamie Forth’s network sync, my sampler, the nekobee synth, and somehow get it all working together. In short, it’s a bit tricky, I’ll be working on packaging soonish though.

I have had a bit of time to make Text, a visual language I mentioned earlier, a bit more stable, here’s a test run:

A bit of a struggle, partly due to the small screen area I gave myself for the grab, but also due to some UI design issues I need to sort out before my workshop at Access Space in Sheffield next week, on the 5th February. Access Space is a really nice free media lab, but will turn nasty unless I free the workshop software, so expect a release soon.

In case someone is interested, here’s the linux commandline I use to record a screencast with audio from jackd:


gst-launch-0.10 avimux name=mux \
! filesink location=cast.avi \
ximagesrc name=videosource use-damage=false endx=640 endy=480 \
! video/x-raw-rgb,framerate=10/1 \
! videorate \
! ffmpegcolorspace \
! videoscale method=1 \
! video/x-raw-yuv,width=640,height=480,framerate=10/1 \
! queue \
! mux. \
jackaudiosrc connect=0 name=audiosource \
! audio/x-raw-float,rate=44100,channels=2,depth=16 \
! audioconvert \
! queue \
! mux.


It’s basically Haskell but with syntax based on proximity in 2D space, rather than adjacency.  Type compatible things connect automatically, made possible though Haskell’s strong types and currying.  I implemented the interface in C, using clutter, and ended up implementing a lot of Haskell’s type system.  Whenever something changes it compiles the graph into Haskell code, which gets piped to ghci.  The different colours are the different types.  Stripes are curried function parameters.  Lots more to do, but I think this could be a really useful system for live performance.

Alex McLean (Goldsmiths), Dave Griffiths (FoAM), Nick Collins (University of Sussex) and Geraint Wiggins (Goldsmiths)

Abstract

In this paper we outline the issues surrounding live coding which is projected for an audience, and in this context, approaches to code visualisation. This includes natural language parsing techniques, using geometrical properties of space in language semantics, representation of execution flow in live coding environments, code as visual data and computer games as live coding environments. We will also touch on the unifying perceptual basis behind symbols, graphics, movement and sound.

1. Introduction

Live coding, the improvisation of video and/or music using computer language, has developed into an active field of research and arts practice over the last decade (Wang and Cook; 2004; Ward et al.; 2004; Collins et al.; 2003). Live coding is made possible by dynamic language interpreters, which allow algorithms to run while they are being modified, taking on changes without any break in the audio or visual output generated by the code. The development of software becomes part of the art in a very real sense; at the beginning of a typical live coded performance there is no code and no audiovisual output, but the output grows in complexity with the code.

A frequent criticism of computer music is the lack of performance, where an artist hides behind their laptop screen, and the audience is unable to see any activity that might ground their experience of the music (Cascone; 2003). Solutions continue to be explored, with many researchers focussing on developing tangible interfaces which bring the computer closer to a traditional instrument. However, a live coding tradition has developed taking the straightforward approach of projecting whatever is on the artist’s screen: the code, moving cursors, the debugging output… The audience is then able to see the human movements and code structures behind an improvisation.

This tradition of projecting screens is itself open to criticism; the audience members may feel distracted, or perhaps even excluded by the projection of code written in language they do not necessarily understand. The alternative of showing nothing, hiding behind a laptop screen, is felt to be untenable, but perhaps more should be understood about the practice of projecting code. Watching the articulations of a live guitarist may enhance the experience of a listener who does not play a musical instrument themselves. Can a live coder elucidate the more abstract thinking gestures of their practice? The search is on for ways of visualising code development that allows non-programmers to enhance their enjoyment and understanding of a live coded piece.

2. Perceiving code

Generally, a programmer cannot work with their eyes closed; a programmer’s text editor is a visual interface¹. Text editors have gained many features over the last few decades, to the point where we no longer call them text editors but Interactive Development Environments (IDEs). The visual presentation of code has developed its own aesthetic; colour is used to highlight syntax, fonts have been designed for code (e.g. ProFont, proggy), and visual tools for navigating around tree-like code structures. Nonetheless computation is fundamentally about symbol manipulation, and the composition of symbols lies at the heart of every IDE. When our eyes saccade across code, the shapes on the screen are categorised into these symbols, and we perceive them as the tokens (words) and statements (sentences) making up our program. The computer interprets code as a one dimensional string of discrete symbols, but humans perceive it as symbols within a spatial scene. Expert programmers may be able to chunk larger blocks of code as meaningful entities; less experienced live code audiences may become stuck on small details, but an elaborate dance of spatial change to code is evident over time.

Our perception of source code is aided not only by spatial organisation, but also by colour highlighting, in-line documentation and the well chosen names given to abstractions and data structures. These features are collectively known as secondary notation², being that ignored by the interpreter but of benefit to programmers in understanding and organising their code. A challenge to those pushing the boundaries of programming language design is to find ways of taking what is normally secondary notation as primary. For example the ColorForth language uses colour as primary syntax, replacing the need for punctuation. Even more radically, the instruction set of the Piet language illustrated in Figure 2 is formed by first order colour relationships within a two dimensional grid; instructions include directional modifiers so that control flow travels in two dimensions. Piet, among many other esoteric languages, is inspired by the two dimensional syntax of Befunge shown in Fig. 1, a textual language where arrow-like characters change the direction of control flow. Some languages bordering on mainstream, such as Haskell and to a lesser extent Python have a syntax that takes two dimensional arrangement into account when grouping statements, although this is otherwise unusual.

Secondary notation is of great importance to human understanding, despite being ignored by the computer interpreter. Without spatial layout and elements of natural language a program would be next to unreadable by humans. Humans live an embodied existence in a spatial environment, and while we are perfectly able to perform computation, our spatial ability still supports such thought processes (Gärdenfors; 2000). As a result source code, as Human Computer Interface, is a half-way mixture of geometrical relations and symbolic structures. This is true even of the ‘patcher’ dataflow languages in common use in the digital arts (Puckette; 1988), such as Max and PureData. Patcher languages are often described as ‘visual’, but in fact all the functions are defined textually, and the visual arrangement is purely secondary notation ³.

Visualisation of code may either act as secondary notation in order to enhance code comprehension for human viewers, or go further as primary syntax to enhance meaning for both humans and computers. The latter is of particular interest, as to some extent it requires making models of human perception the basis of computer language.

2.1. Morphology of Sound, Shape and Symbols

TurTan is a geometric visual live coding language introduced by Gallardo et al. (2008), using the technology of the Reactable (Jordà et al.; 2007). The functions of the language are manipulated as physical blocks that are placed on a tabletop interface, with nearest neighbours forming a sequence, and relative angle mapping to the function’s parameter. The functions describe turtle graphics operations, and the resulting recursive forms are continuously updated on the table surface display.

TurTan inspired a system by Alex McLean and introduced here, with the working title of Acid Sketching. In Acid Sketching, a sound is specified simply by drawing a shape, where morphological measurements are mapped to parameters of an acid bassline synthesiser. The area of a shape is mapped to pitch, its regularity (perimeter length vs area) mapped to envelope modulation, and relative angle of central axis mapped to resonance. Several such shapes are drawn in an arrangement, where a minimum spanning tree of their centroids is taken as a polyphonic sequence, where distance equals relative time. Feedback may be projected back on to the drawing surface, so shapes flash red as they are triggered. A static figure would not make this clearer, however illustrative video is available online at http://yaxu.org/acid-sketching/.

While Acid Sketching and TurTan are far from what is typically understood as live coding, both lead us to challenge understanding of the role of symbols, shape and geometry in computation. Investigating how such concrete forms of interaction could be married with the abstractions of general, Turing complete programming languages could be an interesting research topic itself.

Critically connected to live coding engagement with time-based media, is the time-based revelation of code itself. For electroacoustic music, Pierre Schaeffer’s theories of sound timbre have been further dynamised into the time-variant sonic gestures of Denis Smalley’s spectromorphology (Landy; 2007). For live coding, we might analogously dub ’codeomorphology’ as the changing shape of code over time. Examples include the accumulating code revisions referenced on the edge of ChucK language Audicle documents, or SuperCollider’s ‘History’ class to document a live code performance. More visual representations of change over time would include accessible visualisations of programmer activity. Metrics might be displayed to characterise changes per second, from coarse keystroke counts to the depth of parse tree disruption; this brings us to self-evaluating performances, and coder re-coding of their very visualisations…

3. Visual experiments in live code

This section serves to introduce four novel visual/geometric live coding systems by Dave Griffiths, namely Scheme Bricks, Betablocker, Al-Jazari and Daisy Chain, along with some of the systems which inspired them. All of these languages were constructed within Fluxus, a game engine designed for live coding performances and experiments and available under a free (GPL) license from http://www.pawfal.org/fluxus/.

3.1. Execution flow and operational events

vv  <      <
    2
    ^  v<
 v1<?>3v4
    ^   ^
>  >?>  ?>5^
    v   v
 v9<?>7v6
    v  v<
    8
 .  >  >   ^
^<

Computation is a metaphorical movement, where algorithmic processes operate on data (which can include the algorithms themselves) in memory in the discrete time steps of the CPU. Ways of visualising memory as it is changed have been developed for conventional debuggers, particularly in microcontroller applications where memory is small enough to be viewed in its entirety. More novel visualisations also exist, such as Tierra, an artificial life simulation where code evolves in a Darwinian competition which can only be appreciated when viewed as such, or Core War (Fig. 3), a game where player/programmers write code which fight over memory address space.

Live coding has the unique opportunity to visualise the movement of an underlying process while it is being formed. This helps an audience appreciate a live coding performance in a more meaningful way – as it bridges the gap between an abstract description of a process (the code) and process itself (the generated pattern of movement through memory). Betablocker (Fig. 4) is a raw visualisation of an imaginary 8-bit processor operating in 256 bytes of memory. This brightly coloured live coding environment is operated by writing assembly code with a gamepad. The processes are visualised while they operate on the memory addresses and trigger sound events. Processes are able to modify themselves and each other, resulting in highly dynamic relationships which are challenging to control.

A more traditional method of programming is employed in Scheme Bricks (Fig. 6), a geometric interface for constructing Scheme programs. Scheme Bricks takes advantage of the isomorphism of code and data in the Scheme programming language, and is inspired by the Scratch language designed for use by children (Resnick et al.; 2009). Scheme Bricks allows you to drag, drop and plug together programs rather than typing. This has some potential side effects; in a performance situation, it is impossible to have a mismatched parenthesis error, as is common in other lisp-like languages. It is quicker to change the overall structure of the program as sections can be removed and reinserted easily by drag/drop actions. Unwanted sections are pulled out of the program and set aside rather than being deleted, and accumulate around the program as ‘spare parts’ which are often later ‘recycled’ by being pulled back into another section.

Scheme Bricks uses visual feedback to relate sound events to the code; the instruction which triggered a sound event flashes as the sound is played. This minimal approach to process visualisation makes the relationship between sound and code structure clearer than Betablocker’s more complete visualisation, and is useful for the performer to immediately locate the code generating a particular sound event.

Daisy Chain (Fig. 7) is an attempt to embrace less rigid structures while maintaining enough of a computational basis to qualify as a live coding performance. It follows a processing system based on Petri nets (Petri; 1966), where executable instruction tokens move around a directed graph. Daisy Chain programs create and modify the graph topologies that they inhabit, producing sounds as a side effect of the computation. The look of the performance was designed to be as far from conventional programming as possible, hand animated flowers and drawn instruction symbols moving around graphs constrained by spring models.

The nodes of a Daisy Chain graph have a fixed lifetime, which was introduced in order to counter a common problem with live coding where the audience watching and performer concentrating on programming tend to perceive time differently. Daisy Chain prevents musical structures from persisting too long, keeping the performance moving forward at a rate the performer can control beforehand.

3.2. Computation in game worlds

Code has a long tradition of use in games as a gameplay mechanic, an early example being Core War developed in the mid 1980s and discussed above in §3.1. More recent games such as Carnage Heart and Marionette Handler are mainstream games for the Playstation which employ programming environments using icons. These programs are used to control robots which battle it out in large virtual arenas. Popular game titles such as Little Big Planet allow the player to construct machines as part of game worlds, complex enough to support Turing complete computation. Kodu, a research project at Microsoft goes even further, as an end-user games programming environment on the XBox.

Al-Jazari is a deliberate attempt to fuse games and live coding performances. It was designed to use a similar visual process to BetaBlocker, but this time mediated through the actions of robotic agents moving around a 3D world, triggering sounds as they do so (Fig. 5). The use of visual agents following commands rather than abstract processes is intended to make the performance more immediately understandable for the audience. Al Jazari has been expanded as an art installation, audience participatory performance and recently as a facebook game – with the aim to increase the accessibility of live coding to the point where anyone can become a live coder.

4. Conclusion

Visualisation is central to live coding. In this article, we have confronted how code is perceived by performers and audiences, and in what ways visual elements contribute to the primary syntax and semantics of a programming language meant for live coding. Consideration of visual elements of code have also become essential as live coding has formed the basis of virtual game worlds. We have introduced a number of novel systems, presented here as explorations of these themes. Visualisation of live code however remains under-investigated in terms of the psychology of programming; while Blackwell and Collins (2005) lead the way into HCI, evaluation protocols are yet to be adapted and applied to experience of live coded performances. This is however fertile ground for practice based research, and we anticipate the changing shapes of code over time, a codeomorphology at timescales from individual performances to lifetimes of artistic and technological development.

I’ve been through a few linux distros over the years, neatly getting progressively easier to install and configure as I get less willing to spend time recompiling kernels, culminating in ubuntu, enjoying the attention to detail and simplicity of use.  Recently though, I’ve had to give ubuntu up and go back upstream to the rather higher maintenance Debian again.  Linux suffers from creeping featurism in its layers of audio APIs, it started with OSS, a straightforward API based on files, then came ALSA, a wildly complex API with broken documentation in a wiki you can’t edit, and an architecture that somehow means only one OSS application can write sound at a time.  It seems to me that it’s a failing of ALSA that further layers of abstraction are piled on top of it, creating a rather complex landscape for sound hackers to navigate.

Ubuntu has joined in the fun by shipping with PulseAudio, which is probably great for general users but a pain for those needing to work with audio on a low level without using loads of CPU.  Pulse is not straightforward to remove, and when I removed it had problems with volume controls not working, and the likelihood that future system upgrades wouldn’t work so well.  That’s why I switched to debian sidux, but then I couldn’t get laptop hibernation, or my firewire sound card working, and had the stress of maintaining an unstable distribution.

However this week Puredyne carrot and coriander came out, and it’s really great.  The kernel is optimised for realtime sound, and jack audio runs solidly without any drop outs, something I haven’t seen before.  My firewire sound works reliably, better than I managed under ubuntu.  It has a really nice logo and clean look, with no plump penguins in sight.  It comes with all the best a/v software beautifully packaged, including all the live coding languages.  The people behind it are super friendly and helpful.  It’s downstream from ubuntu, so all the software is available.  It’s a dream!

They make a big deal out of it being good for booting off a USB key, and I think have worked out some nice practicalities of working that way.  This makes it great for doing workshops and running linux in a non-linux lab etc.  It installs and works just as nicely on a permanent hard drive though, and that’s what I’ve done.

Anyway, heartily recommended, a dream come true, congratulations to all those involved.