Web developer. Art/Tech hacker. I ♥ music, film, live video, art and the social web. By David Wolf
It’s been done for a couple of months, but now that I’ve done the ceremony with the funny hats and received my piece of paper it’s time to show the world what I’ve been working on for the last five years!
Vidgets: The Development and Use of Interactive, Network Based Video Works by David Wolf
Big thanks to family, friends and collaborators and especially big thanks to my supervisor Adrian Miles.
Last year an update to QuickTime meant that Quartz Compositions could no longer access RSS feeds when played within the QuickTime Player application. This change was made to stop maliciously created compositions from sending information to remote sources without the user’s knowledge and was made in the name of security. It was very annoying as I was using Quartz Composer to create ‘network aware’ works which accessed RSS feeds, bringing in text and photos from various sources online.
Now with the latest Mac OS software update another of my favourite features has been taken away from Quartz Composer. With 10.4.7 a Quartz Composition cannot use the ‘Image with movie’ node to nest compositions within compositions. Previously I could create a composition, save it in a .mov wrapper and then import it back into another composition as I would a normal movie file. This would let me quickly build up complex interactive compositions, layer and manipulate them in custom Vidgets.
I came across this problem on the Quartz Composer Developer List a few days ago but it was only when I went to play some of my more recent compositions last night that I realised how much of a problem it is. I had come to rely on this feature.
Pierre-Olivier Latour writes that they had to disable this feature so as to prevent people from creating compositions which reference themselves, crashing the computer (another potential security risk). I understand the need to make the software as stable and secure as possible but it seems like there should be another solution which prevents the crashing without ruling out a whole set of features.
It should also be noted that similar features such as network access (including the parsing of remote XML files) and the nesting of movies within movies have been key features of the QuickTime architecture for years without any such restrictions.
“This is brush, a small Max/MSP/Jitter patch that Ive compiled as a standalone application. Its aimed at visualists who are just starting out and looking for software to play with. Programmatically, its very simple. Video from a live camera (or a movie file) is fed back on itself so that light stays on the canvas (screen). Thus, you can paint with the light in the room youre in. Decay (fade time), tolerance (lower luma threshold) and color inversion are adjustable so you can adapt your performance to any lighting conditions.”
Dan Winckler’s blog – brush 0.1: a tool for visual performers
This is a great little piece of software, what I would call a Vidget. A small scale application which lets you manipulate digital media in real time for improvised performance. It is very easy to use and entertaining to play with.
Today I played live visuals for Legs For Fish at Thornbury High school. Kent Macpherson from the band is a music teacher there and organised a performance / presentation for the kids from music, media and IT classes. We played three improvised AV sets and the kids seemed not to hate it 
As Paul from the band said, it makes you acutely aware of how self indulgent this sort of improvised performance is when are doing it in front of a room full of kids who aren’t necessarily there by choice. Some of the kids seemed to dig it and some asked some good questions. “Why do you do this?” “What does it mean? The pictures just looked like colours and shapes.”
I used the Quartz Vidget I prepared for the Liquid Architecture gig in July as it works fairly well and made it easy for me to show the kids what I was doing without the confusion of showing a huge messy Quartz Composer patch. After each set we took questions. I showed how I can grab any two video files or sequences of still images and mix them together, manipulating colour, brightness, contrast etc. as well as layering multiple copies over each other. After the second performance I showed very briefly how Quartz Composer works by dropping in a video file, connecting it to a Billboard and then running it through a couple of effects.
Looking back at the QC presentation I gave at Electrofringe after this one I realise I should have shown heaps more examples and started much more slowly in Newcastle. Assuming no prior knowledge meant I explained things much more clearly, and I introduced things in a much more logical order (having slept the night before also helped!).
It’s coming up on that time of year again. Electrofringe is a new media arts festival held in Newcastle as part of the This Is Not Art arts festival. Lots of good stuff on and definitely worth the trip.
This year I’m presenting / performing the following:
Realtime Video Manipulation using Isadora and Quartz Composer
“Introduction to the use of two software platforms which allow the creation of realtime video installations which can respond to the audience or other available data.”
This will be a introductory ‘tips and tricks’ panel presentation with Luke Toop (Adelaide), Steve Huon (Melbourne), Khalid (Melbourne) and myself. I’m probably going to focus on the basics of how create a Quartz Composer patch and then turn it into a stand alone application with Xcode and a little bit on network access and RSS etc.
Spac{v}e dpwolf vs Doktorb Robotnik
A live improvised AV performance with my frequent collaborator Doktorb Robotnik. Audio and video feedback crossed with data pulled from the network in real time. Followed by a discussion.
Re – Imagining (live) Video – Narrative
A panel presentation / discussion with Jean Poole and Anna Helme.
From the blurb:
“Audio has been easily sampled, processed and manipulated live for decades. Although hardware and software now allow video to become just as malleable, it is used in a limited number of ways. How can current-day video tools be used to composite video at live events differently? How can theatre and storytelling better integrate live video? What storytelling possibilities lay beyond recreating cinema, music videos or ‘wallpaper’? How do live video and sound work best together? What video is most worth having live?”
Technorati Tags: Electrofringe, Quartz Composer
Hmm, nothing like retro-promotion… this happened a few weeks ago.
Friday 26th of August: A night of live AV performances with dpwolf and David Sevo, Bits of Clay and Canvas City at Glitch Bar & Cinema in North Fitzroy.
David played an ambient set of prerecorded tracks mixed with musique concrete style found sounds and samples and I projected very abstract, fluid, generative images created using analog to digital feedback through realtime effects in Quartz Composer.
I had just been down to see the excellent white noise exhibition at ACMI and was inspired by the ideas of abstraction on display. Curator Mike Stubbs’ essay on the exhibition is definitely worth a read: white noise : a leap into the light.
Here is a quick diagram which shows how my setup was plugged together:
The video output of the laptop was connected to both the analogue input of the MiniDV camera and the projector. The FireWire output of the video camera was connected back into the computer. Displaying the DV signal from the camera through colour controls and halftone line filters, back out through the computer’s video output I created a feedback loop. Rather than using mouse based on screen controls as I have with previous Vidget setups, for this performance I chose to use a MIDI controller with 8 knobs and 8 faders. This gave me much needed ‘hands on’ control so I was able to manipulate the feedback by adjusting various patch variables.
Click image to play.
Here is a fairly large (29 Mb), long (8 min) video created playing around with the setup in preparation for the gig which should give some idea of how it looked. This kind of play is an important (and fun) part of my research.
The video is released under an Attribution-NonCommercial-ShareAlike 2.5 license however the music is copyright .
Technorati Tags: Quartz Composer, vidget
The main tools of trade when using Quartz Composer are patches. Each of these modular objects has a particular function when plugged together to create flows of information. For example a patch may import or download an image, calculate a mathematical operation, affect an image or function as a switch, selecting between other patches and signals. This flow ends up rendered on screen by a renderer patch.
While you can plug these patches together in almost any way, until recently you could only play with the patches Apple has included in the program by default. You could not create your own. This meant that while QC talks directly to the graphics card and renders all images as OpenGL surfaces, users were limited to rendering onto flat surfaces, spheres, cubes and a teapot shape.
Recently ClockSkew worked out how to enable disabled patches and write your own custom patch. With this information ?? at Quartzcomposer.jp developed a custom patch which allows you to load 3D Studio Max objects into Quartz Composer and render images onto their surfaces.
Via Roger Bolton: Quartonian Custom patches for Quartz Composer now a reality. Follow the above links for all the downloads how-tos.
I’ve been playing with rendering onto 3D Studio Max .3ds objects in QC for the past week or so but have had mixed results. The fighter jet model included by ?? works well but most other objects I’ve found online have either crashed QC or rendered incorrectly. I’m sure these issues will be worked out down the track.
Here is a quick demo Quicktime movie showing the fighter jet model in QC with cloud effects generated via particle systems based on this smoke effect by Noise Industries.
Click image to play (1.9Mb, 30sec Quicktime).
Technorati Tags: Quartz Composer
mDimension Technology have developed a plugin for Safari which lets you not only view Quartz Compositions in a web page, but also interact with them via JavaScript. This is very neat!
So far I’ve been experimenting with creating custom applications (Vidgets) in Xcode which control Quartz Compositions, this plugin appears to give me most of the same functionality within a web page. To control and manipulate the composition, first you must follow the same methods of ‘publishing’ inputs and outputs from a composition as described here.
View source on these pages to see how to send and receive information to and from these inputs via JavaScript. I’m still working it out myself!
Here is a demo page from mDimension with a Quartz Composition embedded.
Here is a quick demo I made up based on the above page. It allows you to load an image by entering its url into a field and clicking outside of the field (don’t need to press enter), and change the spinning text similarly.
Technorati Tags: Quartz Composer, vidget
Ok, back to the blog after a month…
I’ve been busy working full time and finishing a paper for DAC_05 – Digital Arts & Culture. A conference held in Copenhagen, Denmark in December this year. In 2003 DAC was held here at RMIT in Melbourne. Check out the melbourneDAC site and all the papers.
My paper focusses on the processes of play common to both the creation and use of Vidgets. Rather than looking at my work in terms of its relationship to theatre, literature or cimema (as is often the case when looking at new media art), I draw parallels with the history of sound art practice. In particular I compare the idea of the all data being explored in digital media to the all sound explored by sound artists.
I hope they like it
QuickTime 7.0.1: Security enhancements
This sucks. QT 7.0.1 no longer lets you access remote web locations from within a Quartz Composition when played using Quicktime Player. No more playing such compositions from the Quicktime Plugin in a web page either. This means no RSS feeds unless you run the composition in Quartz Composer itself or in a custom app.
This is the direction I’m heading anyway (making custom apps) but it means you can no longer distribute a .qtz file or a composition in a .mov which accesses network data and expect it to work on machine running Tiger unless the user has Quartz Composer installed.
These were some of the coolest features of Quartz which I was only just starting to explore before the plug was pulled. Hopefully these features will return some time soon.
Update 18/7/05
I don’t think I made this clear enough but the change was made in response to the potential security issue whereby network access from within a composition wrapped in a .mov file to be used to leak information to a malicious third party when used in combination with Quartz Composer’s ability to access sensitive information about the host computer (computer name, local ip address, current username, results of spotlight searches etc). See the original security report from David Remahl here:
Full-Disclosure: [Full-disclosure] [DR018] Quartz Composer / QuickTime 7 information leakage
Regular (non Quartz Composer) wired Quicktime movies have had the potential to expose some information about the host computer via network access for many years, however it was never to this level and I guess never seen as a security issue. As much as I hope to see things like RSS access return to Quartz Compositions when wrapped in .movs I don’t think it is likely.
It would be virtually impossible to modify Quicktime’s handling of QCs to differentiate between allowing nice, friendly information to be sent (get me links to all the recent images of “x” from flickr.com) and preventing nasty information such as (here is my username and ip address, please start trying to hack into my machine).
Using Quartz Compositions as screen savers expose the same potential risks but I guess in this case you are making a conscious decision to install a piece of software, as opposed to playing a .mov file unaware of its hidden code.
Technorati Tags: Quartz Composer, Quicktime
Roger Bolton from eskatonia has put together a great Quartz Composer VJ patch complete with on screen previews and keystroke triggering of clips.
Quartonian is designed to be run in Quartz Composer at full screen and thus has very few on screen controls, in fact with a press of the “~” key you can toggle the whole onscreen display on and off. This makes for a fairly steep initial learning curve for users but, you soon get a feel for using the keyboard and mouse.
Clips (up to 24) must be loaded into ‘Image with movie’ nodes from within a macro patch using Quartz Composer’s editor window to begin. *Hint* it’s the little blue one at the top
Also on the eskatonia site is a guide to Quartz Composer for VJ’s with lots of useful links and examples.
Until visiting this page I didn’t realise that you could load .qtz files into ‘Image with movie’ nodes. For example, the ‘Audio Barcode’ .qtz file from the site which reacts to audio input can be loaded into a patch and used as a mask on another clip.
Both Quartonian and the other examples on the site are released under attribution – non commercial – share alike creative commons licenses so users are free to modify and create derivative works.
Technorati Tags: Quartz Composer
This week I spent a day improvising and recording audio with Doktorb Robotnik (Adrian Lucas) and was reminded of one of the key aims driving my research. I am developing software devices which allow the user to manipulate audio, video and other data in real time. By creating these works I am attempting to give the user the same feeling of control that I experience when performing live audio, of creating order out of chaos and letting it fall apart again.
Adrian and I have been making improvised sound art / noise / music together, for about 9 years. During this time we have developed various methods for collaborative performance and our production methods have evolved considerably.
Initially we would assemble, manipulate and sequence sound objects on computer in to create finished ‘tracks’, often to accompany short video works. While our process was improvisational in many ways, the combination of unlimited levels of ‘undo’ afforded by the computer software and the goal of producing a piece of a set duration meant that hours of work went into seconds of sound.
A significant shift occurred when we agreed to perform live at a music festival at uni: Rusfest 98 . With no idea of exactly what we were doing, we assembled a very basic setup consisting of two multi-effect guitar pedals and two basic synthesisers. Each of the pedals was set to a 2 second delay which let us play various sounds on the synths and have them repeat endlessly. Rather than spending hours obsessing over a few seconds of meticulously cut up audio, we were improvising in real time – in front of an audience. It was exhilarating. We had no interest in melody and our pedals were keeping time, we were free to play with sound. We had constructed a kind of machine for the generation and manipulation of sound in real time and given ourselves a set of variables with which to control it. While our individual noise making machines have diverged technologically since then to consist of a series of interconnected guitar effects pedals which feed back on themselves and a computer running live audio sequencing / processing software, the processes at work in our first performance are still employed.
We build ‘machines’ with fixed number of variables and manipulate them to generate audio in real time. This is exactly what I am doing with images and data in both my Quicktime and Quartz Composer Vidgets.
Technorati Tags: vidget
Electundra is an annual festival of live experimental audiovisual performances. It was held at Loop, Melbourne, Australia from Sunday 12th of June through to Wednesday 15th, so I’m a little late in promoting it! Since last year the festival has grown from two to four nights and featured around 40 artists performing. It was great to see the range of visual styles and techniques represented from live camera switching and physical object manipulation to computer based performances.
I went with the latter option for my own set, performing visuals with Quartz Composer on one laptop whilst running Ableton Live on another laptop for sound, collaborating with Doktorb Robotnik (Adrian Lucas) on feedback electronics. I knew that doing two things at once was going to be quite a challenge so I developed a visual patch which used live audio input to modulate and manipulate images in a number of ways – see this post for a simplified version of my setup. Hopefully this gave the imagery a ‘liveness’ and ‘directness’ which meant that it wasn’t too obvious when I was focussing my whole attention on what I was doing with the audio side of things.
Technorati Tags: Electundra
For the past few weeks I’ve been playing with Quartz Composer live at Plug ‘n Play on Thursday nights at the Kent St Caf on Smith St in Collingwood (Melbourne). It’s a great place to experiment in a comfortable environment with a projector and sound system and the only place I can imagine that gives you free drinks for sitting in the corner programming!
Last Thursday I attempted to use my QuickTime vj application Vidget 3.5 (I’m actually up to version 3.6 but haven’t got around posting it) to mix Quartz compositions saved as .mov files. This produced some interesting (unstable) results. The way the vidget works is to layer up to three Quicktime movies (movies in a movie) over the top of each other with different transparency/opacity graphics modes – like video Photoshop layers which are rendered in real time by Quicktime. With the standard ‘Dither Copy’ mode my compositions played reasonably well but whenever I switched to some other modes the video flickered very fast and bright and whenever I resized the window to go to full screen Quicktime unexpectedly quit.
Since I couldn’t use the vidget I decided to see what happened when I dropped Quartz compositions saved as .mov files into the composition workspace of a new file, to my surprise rather than appearing as ‘image with movie’ nodes (as QuickTime movies usually do), they were added as ‘macro patches’ meaning I could double click and manipulate their inner workings, excellent!
Here’s an image of my original composition space with the new macro patch highlighted:
Double clicking on the macro patch reveals the composition within:
Technorati Tags: Quartz Composer
I’ve been having a good play with Quartz Composer over the last couple of weeks. It is very exciting and a bit scary at this late stage of my MA research: “Does it make what I’ve been doing for the past two years redundant?”; “Should I drop the whole interactive QuickTime thing and start from scratch in this new environment?”; “Should I ignore it for now and continue with QT because it is cross platform and more accessible?”.
In many ways it lets me do what I have been doing, experimenting with, and wanting to do (real time interactive online video) much more quickly and with exciting new visual results. In some ways it makes basic QT redundant but it is quite a different beast.
QuickTime excels on the network. Child movies can be sourced from anywhere, XML and QTlists while a pain to set up sometimes are very powerful and I’ve only really scratched the surface of their potential when combined with server side scripting such as php. Quartz Composer is much more at home on the desktop. It can import still image files from a URL but not movies. It can read RSS very easily, but is designed for human readable text and requires custom scripting to deal with generic XML files and attributes. I have had some success getting QC to load movies from the network via a local QuickTime link file pointing to a URL, but the targeting it is local, relative to the QC composition. It seems this link is lost if the composition is exported to a .mov file.
Here is a quick example (requires Mac OS 10.4). Apologies for the cheesy kaleidoscope imagery once downloaded and unzipped, the .qtz file should play in Quartz Composer, importing link.mov which points to a video file on my server. The zip file is about 4k.
The cool thing is, in many ways this (Quartz Composer), builds upon what I have been doing in QuickTime and is mostly playable by both the QT player and plugin. While the linking to movie files online is problematic at this point, surprisingly live video and audio inputs are supported even in the QuickTime browser plugin! Here is an example which takes a live feed from a FireWire camera, layers it over itself on 3 differently coloured layers (red green and blue) and scales in real time based upon audio input from the computer’s built in microphone. Link to livergb.mov. This has been tested in Safari with a Sony HandyCam and my PowerBook’s built in microphone. Here’s the source .qtz file. While live video input into a movie playing in a browser is pretty exciting, unfortunately more simple things like keyboard and mouse input are missing.
Stay tuned for more examples as I play more…
Ok, this version (Vidget 3.5) follows on from previous versions:
Vidget 1
Flickr Image Viewer (Vidget 2), and
Vidget 3
Vidget 3.5 is an experimental interactive audiovisual performance device which allows the user to manipulate video in real time online. As well as mixing a number of video clips together, the user may search for still images from the Flickr photo sharing site and mix them together. For a instructions on usage see previous versions above.
I arrived in Brisbane this morning VERY early, caught the 6am flight from Melbourne which (due to regional time distortion fields) arrived around 7am local time even though it took just over two hours.
I’ve been battling with my latest revision vidget over the past few days and the saga continues. To get photos from flickr I use the site’s RSS feeds, only they keep changing on me! On Tuesday I had everything sorted, they started using the
On the plane I accidentally opened my LiveStage Pro project in an old version of the app (4.5 instead of the latest 4.5.4) and saved and now while the flickr search works – I can’t get it to load any movies. D’oh! I have, however got inter-movie communication working really well so that I have a separate movie for output and controls. This means I can resize the output to 640*480 really nicely to go to a second monitor.
BTW, I’m writing this from a very cool cafe called The Alibi Room on the corner of Brunswick St and Annie St in Bris-vegas. It has yummy food, good coffee and FREE WIFI!
I recently completed work on an experimental internet radio program with Hannah Miller and Kate Eccles. Hannah and Kate are final year Media students at RMIT majoring in Radio and TV production. My role in the production was to take various pieces of audio, video, still images and text, and create an interface which would allow the user to mix and match the elements in an exploratory, non-linear way.
The result of this work is a program called “Inspiration”, which features interviews, live footage, sound recordings and lyrics from Reset://0 a Japanese influenced Melbourne band.
The program was authored in LiveStage Pro and is a Quicktime file that consists of a sprite track, several movie tracks and a text track which features lyrics. The above image shows the partially completed work as I was assigning sounds to various non-square shaped roll-over buttons. The idea was that rather than presenting the user with a list of options, or even a grid of non-labelled options, the work should encourage the user to explore the screen space with the cursor, almost like they are feeling their way in the dark. To give the users some feedback, and a little direction as to where may be a good place to explore, I used Hanna’s fire twirling image as a guide. I placed invisible sprites over the background image which reacted to the “MouseEnter” event, triggering sounds which played in specific movie tracks, and changing the sprite image for the background so that different parts of the fire twirling would be illuminated and hi-lighted.
You can view the completed work in context on the interadio site. Or, to go straight to Inspiration(requires Quicktime, a fairly recent computer and a decent broadband connection – 15Mb)
In Flickr each image tag has an associated RSS feed automatically generated from a URL with the following structure:
http://www.flickr.com/services/feeds/photos_public.gne?tags=###&format=rss_200
where ### is your search tag.
So to do a search for “factory” I take the string “http://www.flickr.com/services/feeds/photos_public.gne?tags=” and add it to the contents of the text box and then add “&format=rss_200″ to the end. This give me the URL of the feed. I then use the “LoadQTListFromXML” command to load it into the movie. The RSS file is structured as below, it has a lot of information for each picture.
http://www.flickr.com/photos/
http://www.flickr.com/photos/
http://www.flickr.com/photos/42039133@N00/560315/
/><br />I’m not sure whether the quotation marks
add to the effect, or detract from it.
http://www.flickr.com/photos/rae/547742/
/><br />Watching the precious bottles of Cascade beer get packaged up.
http://www.flickr.com/photos/rae/547738/
/><br />Apparently this cat lives at the Cascade factory. It was there the last time
Tony visited.
http://www.flickr.com/photos/rae/547736/
/><br />The Cascade beer brewery in Tasmania. Yum!
…
If you compare this with the previous XML example you can see some similarities. Where I have the whole list surrounded by the
Ok I just realised it would have been much easier to find the image details from in the enclosure tag, but I used the description tag. I’ll change this next version. Anyway…
For each item, there is a url for a medium sized jpg image. After some quick snooping around I found that the images are coded with a letter to describe their size. For example here is an image from DocManhattan‘s Flickr photostram showing the various sizes and their urls (note the how only the last letters change):
http://www.flickr.com/photos/550041_75d3362292_s_d.jpg
http://www.flickr.com/photos/550041_75d3362292_t_d.jpg
http://www.flickr.com/photos/550041_75d3362292_m_d.jpg
http://www.flickr.com/photos/550041_75d3362292_d.jpg
http://www.flickr.com/photos/550041_75d3362292_o_d.jpg
The RSS points to the “m” sized image but what I need is the “t” thumbnail sized version so I remove the last few letters and add “t_d.jpg” to the end.
This post is getting really long so I’ll continue with part 3 soon. Please leave a comment if I need to explain anything more clearly.
Vidget 3 reads from a very simple XML file to load thumbnail images, this file is structured as follows:
…
Each of the clip names refers to the name of both the thumbnail and its associated video clip. For example, when
Loading the Flickr thumbnail images is a little more complicated.
Twitter