a work in progress
Jack Ox
Graphics programming and architecture
Dave Britton
My long term interest has been to create and solve problems encountered in the visualization of music as an abstract phenomenon. This self-authored system for translating extant compositions into visual performance involvesInformation theory with a complex layering of systems. The art works which have emerged from this process have embodied principles of Intermedia, as defined by Dick Higgins, the late avant-garde theorist and Fluxus artist. Intermedia is a completely different concept from multimedia, although it can be included in a multimedia environment. While with multimedia, content is presented in more than one medium simultaneously, intermedia is a combination of structural elements or syntax which come from more than one medium but are combined into one.
I will describe the journey from the 2D world of my earlier visualizations of music into the 3D, fully immersed, stereoscopic world of the CAVE, a virtual reality theater originally developed by the Electronic Visualization Lab at the University of Illinois, Chicago. This project has received support from NCSA at the University of Illinois at Urbana-Champaign including time in their CAVE with a team led by Donna Cox and also funds for software support. SGI has given hardware support and EAI has given Sense8’ s World Tool Kit. Ars Electronica provided the initial research and development money. Robert Putnam from the Scientific Computing and Visualization Group at Boston University is doing the interactive, kinetic sound placement and 3D localization of sound. Absolute Reality Corp. is doing all of the graphics programming and 3D modeling.
In 1996 I began to work on the visual translation of Quanta and Hymn to Matter, which is a large scale symphonic and choral work from the 1970’s by the American composer, Dary John Mizelle. This work is heterophonic and timbre based in structure. Heterophony is a term which describes a non-harmonic polyphony. It is a widely used principle in the second half of the 20th century in western music, but has always played a large part in the genres of primitive, folk and non-Western music. Timbre is the quality of a tone played on a specific instrument which is distinct from the same tone on another instrument. It also describes the different kinds of sounds produced by playing one instrument in different ways. These qualities are a product of the harmonics generated by a tone.
Timbre literally means the color of sound. A vital component of music which is structured on timbre is the combination of specific and differentiated sounds much like a painter can choose to use color. In my previous visual translations of music I worked with other color systems: For instance, I created a color-harmony wheel for my work on Anton Bruckner’s Eighth Symphony which expressed harmonic movement and quality by moving around the color wheel as keys changed and changing between pure hues and colors which were mixed with specific percentages of complimentary colors, creating grayed out versions of the primary color to equivalent variations in dissonance and consonance. These past works used color to perform the harmonic structure of the music. In this work the actual sound becomes the structure, so this is the first time that I have used a system which translates actual sound into color equivalences.
In order to properly express a wide range of possible timbres I mixed up over 130 RGB colors [link to RGB timbre chart in Acrobat] which come from an orchestra,including an extended percussion section, and vocals. In the string section each instrument has its basic hue and then 7 more with additions of other colors expressing the changes in the color of sound which result from playing the instrument in different ways. For instance, the basic hue for violin is a a warm yellowish green. When it is played sul ponticello, or on the bridge, it has more yellow mixed in because the sound is warmer. When it is played on the finger board, or sul tasto, the addition is blue, to created a cooler sound; yet another blue is used with a mute, and so on. The deeper toned stringed instruments are bluer colors; violas are green, cellos, blue green and contrabasses are a deep blue as their base color, taking on the same relative additions as the violins. Brass instruments operate in the range of yellow orange for trumpet to a deep warm red for tuba. Each of these colors are further modified by the addition of relational color mixtures which are visual equivalences for the use of mutes and other playing techniques such as half valve. Woodwinds range from yellows in piccolos and flutes to mixtures between complimentary hues being dominated by greens, violets, or purples for the groups of instruments in the clarinet and oboe families.
The large and varied family of instruments which are all grouped under the general heading of percussion further divides into keyboard percussion, including piano, celeste, marimba, xylophone, and kalimba thumb piano; glass percussion such as glass wind chimes, glass harmonica, and cloud chamber bowl; earth being rocks and sandpaper blocks,and beating objects which includes wood blocks, bamboo wind chimes, temple blocks, kin, snare and bass drums, timpani, cymbals, thunder sheet, tam tam (further changing with different mallets), tubular chimes, and various sets of bells. Percussion instruments made of metal and sounding metal are also given a metal reflection in their timbre and glass instruments have a glass effect included in their colors.
The timbre color changes its saturation becoming lighter and darker depending on the change in dynamics; louder has more saturation and softer has less. This leads directly to what this complex system of color is applied to; what the surface is modifying.
Mizelle based his composition on a poem by the Jesuit scientist and philosopher, Pierre Teilhard de Chardin called Hymn to Matter . In the poem Chardin describes a very harsh landscape where the task of merely surviving engenders amazing acts of creation. I am making this place in drawings which are transformed into 3d through modeling and placement in the VR world of the CAVE. This hand drawn and modeled world is the canvas and the image source for surface textures on which the timbre color lays. I went to real deserts in Arizona and California and photographed landscapes that expressed in their sculptural qualities sound making concepts of different instrument families. There is no vegetation in this world, just varied formations of rocks and earth, drawn in graphite gray.
The model for Strings was definitely the most complex piece, and also the one which is the largest landscape. Because it is a real place we were able to begin with a topological map of Fonts Point and make an accurate drawing of the 3 washes which are the basis for the land formations. Rodriguez used Animi Tek WorldBuilder 1.0 to build it initially, but because the 3D geometry was far too large at 500,000 polygons. He then created a height map from it, applied this as a displacement for a lower resolution geometry, which came in at 4,000 polygons. It is made from 24 equal parts (8x4), each one having a resolution of 1024 x 1024. The final version was done in 3D Studio Max 2.5 Our aim for the CAVE environment is to have a very low poly count with a high resolution texture map.
In order for me to make an orthographic texture map which would cover all surfaces of the model, Rodriguez gave me screen shots of the model from 4 different angles of view. The last view was straight overhead, and this one I tiled and printed out to fit my drawing table, put tracing paper over it and drew all of the water carved desert hill textures.
Sandpaper texture has become a ground cover for everything without another specific texture. It mirrors itself on all 4 sides so that it can be repeated seamlessly and was only modeled on a bump map, the rocks being portrayed are so close to the ground it is useless to make actual objects out of them. Where „sandpaper‰ runs into other textures Elastic Reality was used to make the blending gradient..
Because the Vocals landscape exists on the ends of the VR space their modeling is modified. The original drawing was cut out, points were placed on these sections and pushed in and out, representing spatial points in the original rock structure. This allows for a very low polygon count, always a goal when working in a real time VR situation like the CAVE. Caligari trueSpace 4.2 was used for both Vocal and Woodwinds because it permits polygons that are larger than 3 points, a feature not common to all 3D programs and one which supports low polygon numbers. Some of the rocks towers in Woodwinds were modeled by themselves out of cylinders, unwrapped into flat versions of the wire frames and sent to me. I printed out 12-15 inch high frames, over which I made an unwrapped texture map on my drawing table, scanned it back in to the computer, where it was wrapped around the model. Rock groups were made into planar models, mirrored on the backside and with a modified modeling as in the vocals.
In all of the musical visualizations that I have made during the last 20 years there has been a very close relationship between the images, through which the structure of the music is viewed, and the music itself. My work has always used landscape and/or architectural images which related structurally, chronologically, or in the case with my visual performance of Kurt Schwitters' Ursonate, with the large scale installation art works of Schwitters himself. In the case of Mizelle's music, he had created his composition using the text of Chardin's poem. Therefore I to went to Hymn to Matter for direction in creating a meaningful context in which to perform Quanta and Hymn to Matter. It turns out that this is a far more flexible world, containing elements which directly connect to other music, by other composers. I decided to keep all of the landscapes in gray scale so that the only color you see takes your eyes directly to the structure above the landscape which is created by and as the music plays. This is a world which is about different lines and shapes and is a flexible musical environment. The individual landscape units are each connected to a family of instruments, so compositions using instruments belonging to these groups can be performed here, deriving meaning from the images embedded in their structure, and in the environment from which they came. For instance, I could see music from Edgar Varese in this world.
Problems with Quanta; Solutions with Q
When I began working on Q&HM , this music only existed as a paper score, having never been performed or realized before in any form. This meant that first the score had to be laboriously put into MIDI files (the international Musical Instrument Digital Interface standard used to connect synthesizers and other musical electronic components), then sound patches had to be made from all of the instruments and voices, followed by a synthesized version of sound files when played through a Kurstweil synthesizer connecting to the recorded sound patches. The tremendous labor and cost on this part of the project made it impossible to proceed in the CAVE project with Q&HM . The music is over twenty minutes long, about twice as long as a CAVE piece should be, and I discovered later that the wonder of Q&HM , its obsessive details in ever more complex layers, would not even be visible in the low resolution environment of high performance computing.
At this point Charlie Morrow, who composes in MIDI and sound files in a state of the art sound studio, entered the project. He played for me a dynamic 8 minute piece , including kinetic sound,which is both heterophonic and timbre based. Morrow originally composed it to be played in collaboration with Jerry Rothenberg jubilantly reciting his poem Paris Elegies. This work has literally been pluged and played in the desert environment created orginally created for Q&HM , and in this new incarnation is called Q. It has 41 tracks of sound, including woodwinds, strings, brass and a very large representation from different percussion instruments. These include bamboo wind chimes, wood blocks, xylophone, snare drums, timpani, bass drum, triangle, tubular chimes, thunder sheet , rocks, sand paper blocks, glass harmonica, glass wind chimes, piano, celeste, and kalimba.
