I’m very happy to have a paper “Bricolage Programming in the Creative Arts” accepted to this year’s PPIG (the Psychology of Programming Interest Group). You’re probably better off reading the PDF version, but I’ve put it here too:
Bricolage Programming in the Creative Arts
Alex McLean Geraint Wiggins
Centre for Cognition, Computation and Culture Department of Computing Goldsmiths, University of London
Abstract:
In this paper we consider artists who create their work by writing algorithms, which when interpreted by a computer generates their plotted drawings, synthesised music, animated digital video, or whatever target medium they have chosen. We examine the demands that such artists place upon their environments, the relationships between concepts and algorithms, and of cognition and computation. We begin by considering an artist’s creative process, and situating it within the bricolage style of programming. An embodied view of bricolage programming is related, underpinned by theories of cognitive metaphor and computational creativity, and finally with consideration of the bricolage programmer’s relation to time.
Here’s a tech demo of my current Haskell live coding environment, an emacs mode adapted from Rohan Drape’s haskell supercollider mode, with flash on eval adapted from the syntax tail highlighting minor mode.
A few people on twitter found this useful so here it is in full:
I’ve been writing a few papers lately and going through the cycle of write -> print -> proofread -> write, generating a lot of paper. I’ve text to hard to read on screen, and raw LaTeX somehow feels too malleable to read as a document. Then I thought the obvious; why not format the document for the screen.
I came up with this, two columns with minimal margins:
This fits nicely on my laptop screen but adjust for your particular aspect ratio and so on. Then view in full screen or presentation mode and hey presto. Evince in linux is great in presentation mode (rather than full screen mode, which keeps a menu bar), and automatically picks up changes when you recompile your PDF.
If you still find yourself with eye strain, rather than reading from paper, consider adjusting the position of your monitor. There’s a lot of hype around e-paper, and it does look lovely, but I’m unsure that the evidence shows that LCDs are significantly worse for eye strain if at all… Is light really that different when it reflects off something rather than emitting from something? I reckon distance between eyes and monitor is a bigger factor at least.
Without interpreters, we wouldn’t have software, but yet interpreters are also software. This is why we talk about `bootstrapping’, where software pulls itself from the floor by its bootstraps, a paradox settled by the existence of hardware microcode.
Any piece of software exists as a combination of two parts, some instructions in a computer language and an interpreter of that language. Alone they do nothing, put them together and they can notionally do anything. Often there are intermediary steps, commonly interpretation into another language called `bytecode’ to produce `binaries’, but these are just translations into another language, which still needs interpreting for the magic to happen.
Interpreters are fantastic, they allow us to try out ideas beyond our imaginations, adding some instructions, interpreting them to get output rendered as sound or light to our senses, perceiving otherwise impossible worlds, and returning to the source code to twist the encoded structures into new contortions inspired by the results so far. We humans expand the realms of perception through computation, not creating things but writing about things in order to magic them into existence. We’re only scratching the surface of what’s possible, artistic and otherwise, from marrying high speed computation with embodied human experience.
It’s a shame then that the freedom of thought that interpreters give us happen to threaten business models of large companies, who are accordingly searching for the power to make free access to them them illegal. `Console’ computers (a misnomer if I ever saw one) are those where the end user is not allowed access to an interpreter, without paying for a restrictive license and/or expensive hardware. You are not allowed to modify code, certainly not allowed to modify the interpreter, and so must be satisfied with using whatever programs the manufacturer allows you to.
What is frightening is that these business models are spreading — from computer games, to handheld computers and now to tablets. The iPhone (and now iPad) was a particular shock as a device from a company producing hardware traditionally marketed at the creative. You certainly can be creative with an iPad, but only using a surface level interface; you can touch the screen to make a mark, but you can’t write a program about touch, or about marks. iScratch, a project that allowed children to interpret their programs on the iPhone, has been rejected from the apple store. Such interpreters have been banned from the iPhone from the start (apart from the concession of a javascript web browser VM without access to, for example, sound synthesis), but a big media stir has only been made since interpretation was locked down on the development side too, having impacts on some crappy tools by a large company.
This creep towards centralised control over interpretation is deeply worrying, and arguments that end-users get confused or threatened by higher order thought are frankly sinister. Can we have our languages back, please?
It’s Ada Lovelace day today, and it turns out my main influences lately have been two women.
One is Sherry Turkle, via her book The Second Self: Computers and the Human Spirit, and her work on Epistemological Pluralism and the Revaluation of the Concrete together with Seymour Papert. Two decades on, some aspects of this work are dated, but the main drive reads like a message from the future; to fully accept computer programming in our creative lives we should consider the feminine, do away with purity and black boxes and work within our code rather than upon it. Computer programming languages and culture is biased towards abstract approaches and against anthropomorphism.
Perhaps live coding is a swing back towards a more feminine approach, where humans are immersed in the same time flow as algorithms, fully engage their perceptual and aesthetic faculties while writing code, and have their whole program before them ready to be manipulated, rather than abstracting parts in black boxes according to grand designs.
Here’s a choice quote from the epistemological pluralism paper:
The conventional route into formal systems, through the manipulation of abstract symbols, closes doors that the computer can open. The computer, with its graphics, its sounds, its text and animation, can provide a port of entry for people whose chief ways of relating to the world are through movement, intuition, visual impression, the power of words and associations. And it can provide a privileged point of entry for people whose mode of approach is through a close, bodily identification with the world of ideas or those who appropriate through anthropomorphization. The computational object, on the border between the idea and a physical object, offers new possibilities.
My other recent influence is Laurie Spiegel, in particular her writing on the manipulation of musical patterns. I’d already found myself making a pattern language when I found her paper, and found it a great inspiration. In this 1981 paper she recognises computation as pattern transformation (which after all is all that it is, a Jacquard loom head that transforms), and applies this insight directly to music; computation then becomes music pattern transformation, the structure of computation forming the structure of music. There’s much more inspiration for musician-programmers to be gained from her other writings.
Conceptual metaphor theory holds that our understanding of the world is largely structured by metaphor. This presumably includes our understanding of computer programs, which is the basis of Metaphors we Program By: Space, Action and Society in Java, a paper by Alan Blackwell. The paper shows analyses of documentation for some standard Java libraries, looking for the metaphors that structure human understanding of the library components and their interactions. I’ve taken the liberty of extracting the metaphors related in the paper but if you’re interested you should go and read the whole thing, it’s a good one. I feel I could meditate on this list for some time, and I’d love to see comparisons with the metaphors used in the documentation of other languages.
Components are agents of action in a causal universe.
Programs operate in historical time.
Program state can be measured in quantitative terms.
Components are members of a society.
Components own and trade data.
Components are subject to legal constraints.
Method calls are speech acts.
Components have communicative intent.
A component has beliefs and intentions.
Components observe and seek information in the execution environment.
Components are subject to moral and aesthetic judgment.
Programs operate in a spatial world with containment and extent.
Execution is a journey in some landscape.
Program logic is a physical structure, with material properties and subject to decay.
Data is a substance that flows and is stored.
Technical relationships are violent encounters.
Programs can author texts.
Programs can construct displays.
Data is a genetic, metabolizing lifeform with body parts.
Software tasks and behaviour are delegated by automaticity.
Software exists in a cultural/historical context.
Software components are social proxies for their authors.
I’ve written about Alan Blackwell’s research before.
Geoffg is playing a lawyer in declaring his interpretation of the GPL as the truth, lambasting the Free Software Foundation for not updating some old articles, accusing them of spreading FUD and declaring the iphone as a missed marketing opportunity.
He makes the mistake however of reading the GPLv2 as a definition of freedom, rather than an implementation of it. As this fine person pointed out last year, it’s no good being able to change software if you can’t run it. This is the freedom to change software and use it as toilet paper.
Fine, maybe you can run GPLv2 software on your iphone, but you can’t exercise all the freedoms which the FSF campaign for. According to them, GPLv2 software in closed DRM is exploiting a loophole which GPLv3 was in part written to close.
I can see it’s difficult for people to reconcile their love for free software and their love for Apple products, but maybe it’s best to be mindful of the fact that you’ve bought a computer that you can’t run software on without the permission of a large corporation.
There’s a lot of weirdness in the computer music community around this. RJDJ is a port of the free PureData language to the iphone. Many of those involved are free software developers, but have to give up all their rights to RJDJ, a venture capital funded company. RJDJ in turn license the code back to them under the GPLv3. This means they are free to change and run the code under open operating systems, but not on the iphone. This is weird, using the GPLv3 as a firewall to protect commercial interests on a closed platform, while exploiting the work of a free software community.
Just to clarify; I’m not saying that people shouldn’t run open source software on the iphone, particularly when a permissive license like MIT or BSD is involved. I’m just objecting to ignorance of the FSFs definition of software freedom, which I happen to subscribe to and which is incompatible with the iPhone. I find the RJDJ case weird, but while I know of at least some cognitive dissonance in the RJDJ team, I don’t know that any of the authors are otherwise unhappy with the situation.
UPDATE: Michael from RJDJ contacted me to point out something I got wrong. RJDJ don’t release their whole iphone port of PureData, just part of it — rjlib. This is indeed released under a GPLv3 though; if you want to contribute towards rjlib, you have to transfer your rights to RJDJ or fork the code.
BTW they are also releasing a rather nice looking cross-platform gui-less version of PureData under the LGPLGPLv3 and LGPLv3 called zengarden.
I had this feeling that a nice visualisation of polyrhythmic structure would be to plot patterns in a spiral. I finally got it to work, and here’s the first few results, rendered with cairo in Haskell:
{red blue orange purple, yellow brown pink} [red blue orange purple, yellow brown pink] {orange tomato, blue ~ purple ~ ~} {orange yellow, blue green ~}
Where colours co-occur, they’re mixed, thanks to the lovely Haskell Colour library.
I was surprised by the rhythmic gestalt these give. In retrospect I realise that Mick Grierson would not be surprised at all by this, and neither would John Whitney. Now I’m a spiral lover too.
I’ll release the code along with the rest of my pattern library next week.
hackpact9
Played around with spiral rendering, but came to conclusion that the spiral mapping itself was beautiful and not the patterns I’m trying to visualise, because just about anything you put into it looks good. Although you can see the pattern length in the spiral, if you mix different pattern lengths into a polyrhythm, it’s very difficult to perceive more than one pattern length, one comes across more strongly than the others. Perhaps a peano weave would be better, but I think I’ve gone far enough with this, back to music tomorrow.
hackpact10
My lack of headphones bit me so I couldn’t make music tonight. Instead I played with context free art, which I’ve been meaning to do for years. No great masterpieces, but healthy, grammatically constrained fun.
Ok back to music finally. I’m going to try to add a function to my pattern library before making a quick screencast of the first time I use it to make music patterns. This time I made a ‘breakbeat’ function, which takes a pattern of integers and uses it to trigger playback positions in another pattern. I also found a bug which breaks things at the end. I could add exception handling so these bugs wouldn’t break everying, but hey, it’s good to livecode on the edge.
hackpact12
Added a couple of things:
stuff to my ghci interface so I can plug one active pattern into the composition of another.
an ‘onsets’ function that turns a pattern into a pattern of bools, true for the first beat of a phrase.
Used these things together to apply functions to effects that are responsive to the sound sample pattern. It takes me a while to work out how to do it, and the end result isn’t particularly musical. I’ll be playing around with this some more, but in the spirit of showing the first time I use a function to make sound, here’s the screencast.
(or it will be once the vimeo flash conversion finishes)
hackpact13
Implemented and played around with bpm patterns. Also implemented simple ‘tween’ pattern for linearly tweening between two values over a given period.
Did some good live coding practice for Sunday’s gig in Plymouth with Dave. We didn’t record anything though. Instead for hackpact I made a swirly processing patch while thinking about pattern visualisation.
I’m very happy to be putting together the evening programme for knowledge futures on October 16th at Goldsmiths. Excitingly, my superheroes Leafcutter John, Finn Peters and Yee-King and Spacedog UK (Sarah Angliss + co) have agreed to play. I’m going to play too as part of slub.
Details are coming together over here. Buy your ticket early!
In the tradition of daily live coding practice a few people have formed a hack pact to make something new every day in September. I’ve got a London Haskell User Group talk coming up on September 17th and a livecoding gig at the planetarium in Plymouth on 21st September, so my daily hack will be focussed on getting my Haskell stuff in order and practicing with it. I’ll document my progress on this blog.
In other news, a couple of film crews attended the last pubcode, and the bbc feature has appeared on their website. Features excellent livecoded algorithmic dancing from Yee-King and Click Nilson.
In the tradition of