“The digital spring” paper
part one
by
Michael Nitsche

STILL SERIOUSLY WORK IN ETERNAL PROGRESS !!

version 19/12/01
words: 8.356
Approach
Digital media develop extremely fast and spread very wide. Moore’s law that the computer processing power can double every 18 months and the growth of the now ubiquitous Internet serving as the most famous example. Within these fast extending media there are – and one might say they are more and more growing – unexplored possibilities for artistic expression. The basic new possibilities of digital media, such as interactive access and varying forms of content representation, have paved the way for their early successes and lead to the extremely successful world of computer games. Stepping beyond this primary attraction and introducing new content and design methodologies proves difficult.
The step to “next generation” game consoles – as seen in the introduction of the PlayStation 2 (2000) – displayed this void as very often the same content is represented using “just more polygons” [quote EDGE!].
This paper will explore this present void. It will investigate how to apply narrative techniques to real-time three-dimensional virtual environments. It focuses on real-time three-dimensional virtual environments (RT 3D VE) accessible on consumer platforms at the moment of writing (November 2001) as one form of digital media. 

Three research projects running on two different consumer platforms (PC and Sony PlayStation) – all conceived and conducted within 1999-2002 – will be the practical examples and testing grounds for the theories developed in this paper. These projects are the Common Tales project (1999-2000; co-operation of the National Film and Television School London NFTS, Sony Computer Entertainment Europe SCEE, and the Cambridge University Moving Image Studio CUMIS), the Wayfarer project (2000-; in co-operation with the West Herts College, CUMIS, and the Centre for Applied Research in Education Technology CARET), and the Haven project (2001-; CUMIS in cooperation with CARET).
These projects serve as current examples used to discuss the wider field of other RT 3D VE. Using practical examples avoids a historical summing up of the field into an Aristotelian “Poetics” of virtual spaces. Such an attempt had to be avoided as after mere 25 years of frantic development there is simply no final shape or definite theory about interactive media in sight. What this thesis can offer though is to develop structural qualities for this ongoing development. 

Structural qualities are necessary, as it is proving more and more difficult to fill the technological advances with content. New technical possibilities (such as connectivity, processing power, smaller computing devices) demand to be implemented in meaningful structures. 
This paper is addressing the issue of formalising a way to make content in RT 3D VE more meaningful by filling them with narrative elements. RT 3D VE are envisioned as carriers for narrative structures that evoke a meaningful experience in a user who participates in them. The approach this paper will take is an interdisciplinary one, as it will refer to architecture, film, narratology, and to some extent to theatre in the discussion of real-time virtual environments. How can techniques from these traditional media be used as narrative elements in the structuring of a RT 3D VE? 

Part one: Analysis of crucial elements of RT 3D VE
0 Introduction
I Spatial Structure
One defining part of a real-time three-dimensional virtual environment is its spatiality. Each of these virtual environments delivers some kind of spatial structure to the user. This structure might be generated on the basic of sheer mathematical rules by the program “on the fly” like in the space exploration and fighting game Elite (Braben/Bell, UK 1984) or might be predefined by a designer like in most other RT 3D VE. Both cases offer a spatial virtual environment and the user can explore this virtual space. This chapter puts the spatial structure of VE in relation to architecture and film. This relation will be of importance for later references between the fields of film, architecture and RT 3D VE in the following.

I.1 Architecture and VE

Virtual worlds are no physically real spaces nevertheless they can be experienced as places in their own right. Both the solid architecture’s and the virtual world’s space are confined and structured and these qualities can be experienced by the human interactor who enters them. Both spaces offer the impression of a presence within and exploration of the space.
Presence in the terms of this investigation is understood as the mental state where a user subjectively feels physically present within a computer-generated VE. 
Being in a virtual world and being active inside this world are considered important elements in the level of presence in this world (Slater 1999; note that the question of how to measure presence is still in discussion: e.g. Witmer/Singer 1998 versus Slater 1999). ‘The experience of being transported to an elaborately simulated place is pleasurable in itself, regardless of the fantasy content.’ (Murray 1997 : 98) This thesis argues that is not regardless but the pleasure closely connected to the content of the environment. Heim even talks about the ‘psychological quality of any given virtual architecture’ (Heim 2001). This thesis avoids a psychological approach and will use architecture and film as points of reference, instead.
A VR experiment at the Georgia Institute of Technology Atlanta logged acrophobics in a very simple virtual world where they would travel upwards with an elevator. The acrophobics under these circumstances are ‘almost as afraid in this virtual space as they are in the real world. They are frightened even though this is not a very realistic scene, even though the shapes are very simple, the colors are artificially bright, and there is no sense of atmosphere.’ (Bolter 1997) This gives a clear indication how the content of the space influences the user who enters it. This effect is due to the feeling of presence inside the space. This feature of virtual worlds is why Computer Aided Design (CAD) walkthroughs through non-existent buildings are useful. This is also why this paper has to look towards criteria for structuring architectural space that might be of value for the virtual counterpart.

From an architect’s / spatial designer’s point of view presence is extremely important, because one’s subjective being in a virtual place generates the very existence of it. Virtual spaces are because we can go there. As Wertheim phrases it: ‘Despite its lack of physicality, cyberspace is a real space. I am there – whatever this statement may ultimately turn out to mean.’ (Wertheim 1999 : 229 [italics Wertheim’s]). What it means for the spatiality of VE is the reverse logic: if I am there, then there has to be a there, which is the virtual space. But in contrast to architecture, such an impression of a coherent and closed space in virtual environments is an illusion. 
If a certain part of the VE polygonal structure is not in the field of view of the user it is not rendered. That means the computer program does not create this structure’s spatial representation. This location might still influence the action (e.g. by using sound) but the structure itself is not generated by the system. The computer can even generate this spatial structure on demand the very moment a user enters it or it gets into view (see for example the randomly generated dungeons and wilderness areas in Diablo 2 (Blizzard, USA 2000)). Thus, a RT 3D VE can consist of many separate interlinked spatial pieces. 
Even these links can defy the nature of natural architectural logic as they can link between spaces that do not have any connections in the overall spatial structure. 
In a virtual world the different locations can be stored as separate computer files that have to be uploaded by the engine that, then, generates the space and every feature of the VE. There is no spatial connection between these data files, only a mathematical one. A designer can interlink them in any way.

Figure 1 – The Cube Club level in Common Tales and the connection of separate spatial units
One environment for the Common Tales project, the Cube Club, consists of several separate structures that are interlinked. But the whole structure can be experienced as one closed spatial location. Each single standing structure in fig x is connected to another via invisible “warp zones”. Characters, including the user, are “teleported” from one location to another when, for example, stepping through a door or crawling out from a tunnel. Thus, a seemingly coherent space can include, in fact, various spatially illogical twists and alterations that belie this coherence but pass unnoticed by the user during the spatial exploration.
Each separate 3D VE file – as well as their connections towards each other and relationship with each other – needs to be structured. This is the task of a designer or architect. ‘[C]yberspace is architecture; cyberspace has an architecture; and cyberspace contains architecture’ (Novak In: Benedikt 1991 : 226). The programmer designs the software architecture, the modeller creates the models that are existent in virtual space, and a designer assembles and shapes these models. This three-some use of the word architecture already indicates that technical support and dependencies are important factors for the shaping of virtual environments. In the case of this thesis the speed of the graphic rendering of a 3D image is of crucial importance for such virtual environments. 
With the dawn of faster graphic cards and increasing processor performances these environments can be rendered in real-time (as in the Virtual West Cambridge Project at the Department of Architecture at Cambridge, CUMIS 2000- )

Figure 2 – Virtual West Cambridge; this RT 3D VE is modelled for the planned extension of the University of Cambridge (using the Green Dino engine)

In the case of the Virtual West Cambridge project the RT 3D VE serves as a tool for the architect’s work that will be realised later. This model is not broken down in separate locations as an overall view is one of its priority functions. Real-time models allow a fast update of the environment as well as immediate access to them. One can actually experience the space during different planning stages. This specific model delivers an intermediate visualisation of a planned site. This offers the possibility to test different options but the model as such has no interactive features implemented in it. 

Other virtual environments do not necessarily need to be represented in the physical world to gain autonomy. They are representatives of architectural work themselves. Real-time rendering makes them explorable for human beings – which, in turn, make these structures functional. ‘[A]rchitecture has become transmissible’ (Novak, 1996a). This paper will not follow Novak towards the definition of a new architecture through the possibilities of virtual environments. Instead, it will push the specifics of RT 3D VE as media into the foreground.

The Virtual Reality Modelling Language standard (VRML) allows worlds to be accessible through the Internet. Other 3D systems – such as computer games – allow the download of spatial structures for use on the client/user side. 
These structures are not intended to leave the virtual form. They can mirror places from the physical world (e.g. the model of Notre Dame built by Digitalo Studios Inc. in 1999 http://vrndproject.com/vrndfr.htm  using the Unreal game engine), they can reference to physical worlds without copying any specific location (e.g. the original levels of Counter-Strike http://www.counter-strike.net/ based on the Half-Life engine), or they can approach structures that do not depend anymore on physical ones (e.g. some Quake III Arena environments by ID software).

The question, here, is not how architecture is being extended by digital media, but what role architecture plays in the extension of digital media to narrative virtual environments. Structuring the virtual worlds for these new media relates back to architectural concepts. In the long run, the result may well be new, evolving spatial theories; but this paper does not aim at this development for the sake of an extension of architecture. Instead it looks for a use of architectural concepts in the structuring of narrative elements.

I.2 Cinematic Space and VE

The interdisciplinary approach of this paper relates to Ted Nelson’s statement, that ‘[m]aking software is like making movies because both are about how moving presentations affect the mind and feelings of the viewer. (...) I would like to consider software and movies both as parts of the same field, which I call virtuality.’ (Nelson In: Laurel, 1990 : 238) 

Nelson draws film and interactive digital media even closer together ‘Designing for the little screen on the desktop has the most in common with designing for the Big Screen (directing theatrical films). Interactive software needs the talents of a Disney, a Griffith, a Welles, a Hitchcock, a Capra, a Bob Abel.’ (Nelson In: Laurel, 1990 : 243; see also EDGE, Aug, 1999 for direct comparison of games and Hollywood films) 
Nelson is by far not the only one who states a familiarity between film and interactive virtual worlds. 

Computer monitors use a higher Hertz-rate and higher pixel resolution than most television-sets. This way a computer monitor can deliver a sharper and brighter picture. The recipient’s average distance from the TV set is about 2,50 meters (Fröhlich 1991). Computer users sit nearer – sometimes too near (Ankrum 1996 mentions 40inches as preferable distance to a computer monitor but this is rarely achieved in practice) – to the screen thereby widening the view angle. A wide view angle, the Big Screen, is one of the characteristics of cinema where the projection is much bigger than in the world of television. 
In this case, the computer screen is nearer to the cinema screen than to television. That means that it should be used more accordingly in cinema than in television. For example, while there is a draw back from wide establishing shots and multiple split-screens in television productions as the details might get lost on a small screen – this is no problem in cinema or computer visual representation because of the higher quality of the projection. The Windows operating system as well as the Macintosh OS – as a typical computer applications – use the split-screen accessibility to the extreme by opening new screens for each application within the projection field. 

Within this development of the interactive digital media, the use of film elements holds an important position. Digital video played a significant role in the shaping of the interactive media from early research works such as Andrew Lippman’s Aspen Movie Map (Lippman, USA 1978) and artistic experiments like Weinbren’s The Erlking (Weinbren/Friedman, USA 1986) to consumer products such as Psychic Detective (Sanborn, USA 1995); growing network capabilities allowed streaming video delivered through the Internet (like the RealPlayer 1995) and DVD capabilities of next generation game consoles (like the Sony PlayStation 2 released 2000) pushed the data storage and visual presentation further into cinematic domains. Technically the computer as medium has developed towards an incorporation of cinematic media into its capabilities. Due to this development we get more and more familiar with cinematic visual presentations delivered by modern interactive titles.

Interactive media developed from text capabilities (e.g. the first text adventure ADVENTURE (Crowther/Woods, UK 1976)) towards more and more elaborate 2D graphics (e.g. Pac-Man (Iwatani, JP 1980)), 3D graphics (e.g. through the introduction of the original Sony Playstation game console in 1994) and video capabilities that use bigger, faster data storage.
3D capabilities are improving at this time of writing (December 2001) reaching more and more photo realistic 3D graphical representation that dependent on graphic card performance (e.g. through the release of Nvidia’s GeForce 3 graphic card 2001). In addition, different forms started to mix with each other shaping new models such as interactive television, mobile phone gaming, 3D virtual chat rooms, and full-motion-video blurring into 3D game environments. 
But the importance of digital moving images as video is not the focus of this paper. 
Graphic performances allow 3D rendered virtual events to unfold in a graphical quality close to video. Elaborate 3D titles such as Half Life (Valve, USA 1998), Shenmue (Suzuki, JP 2000), Black & White (Lionhead Studios, UK 2001), or Max Payne (Remedy, USA 2001) use their 3D engine to stage and generate film scenes in real-time. The visuals are not based on digital video but created on the fly and, therefore, they can be manipulated. 
These titles do not depend on digital video – but they still quote the cinematic traditions. This paper investigates the cinematic visual presentation of space and objects within this space through real-time rendered 3D virtual environments.

Cinema creates its space by a careful mise-en-scene of its separate shots (e.g. Katz 1991; 1992). The shots generate a screen space consisting of the ‘visible space within the frame’ (Bordwell 1993 : 72). While watching the film the audience assembles the screen spaces to a the story world space (Bordwell 1993; Branigan 1992 : 33f). Only small parts of the story-world space are visible at any one time and the audience unconsciously adds the missing parts already introduced earlier. 
This assembly is done in the post-production process of editing. 3D real-time environments generate the view on the fly; they lack the post-production phase. But the technical capabilities of the real-time rendering programs allow an assembly of two different virtual camera points of view. Technically editing is possible in RT 3D VE and even allows limited (at the time of writing, December 2001) advanced features, such as fading (into or from a still).

This flexibility of VE space allows it to function in a comparable way to film space. It can dramatise the space through artificial lighting, special sound design, visual effects, and change of the space itself. In this regard VE can appropriately be seen as a stage (see also Murray 1997) or a film set. But there is an important difference between a film set and a VR world.

Film sets – in contrast to traditional architectural buildings – do not have to be coherent and closed spaces. The cinematic space gets assembled only through the camera-work, the editing process, and finally the perception during which the audience unconsciously constructs the space of the cinematic action from the given shots. But if this audience were to step into the architectural space of the film set and take control over the camera they would see a modern film studio at the point where the set ends. The space would have no coherence and the illusion would be broken. 
It is a defining part of real-time digital worlds that they allow this step into the architectural space. The set does not end but gets rendered/generated by the program when the user “turns around”.
A virtual environment’s space resembles therefore the cinematic qualities of mise-en-scene, editing, and dramatisation of space with the apparent coherence of architectural structures. In their spatial logic, RT 3D VE allow users to step into cinematic space.

I.3 Conclusion

[NOTE could this be a moment to talk about space to place debate?]
This paper concentrates on three-dimensional real-time virtual environments (RT 3D VE). A RT 3D VE is seen as a category of computer mediated virtualities.
We define as a RT 3D VE a computer-generated representation of a spatial structure, which an observer can enter – not physically, but mentally. The observer can interact with computer-controlled objects and optionally also with human-controlled ones in this virtual space and/or with the space itself. This interaction functions in real-time that means, without any delay between the user’s input and the system’s reaction to that.

Virtual environments have a representational quality comparable to cinematic space but differ in the technical process of their generation as they do not re-assemble a spatial structure through the audience’s reception, but actually generate this structure in the system. Due to this space generation they can allow a user to enter such a space and create the illusion of a closed architectural space. Thus, RT 3D VE worlds combine features of real space with elements of dramatised film space, the narrative power of drama and the functional qualities of architecture. Virtual space is a stage and can be used as an active element when needed. 

II Mediation
II.1 Moving Images
There is no “natural” point of view in a virtual environment. Each view in a RT 3D VE has to be rendered and generated by positioning of a virtual “camera” inside the VE. The role of the camera, then, is comparable in importance to the camera in cinema – and its expressiveness can build on the visual language established by the medium film. Certain camera style are understood and read in connection to certain genre and content. These visual styles can be transferred into interactive virtual environments.
Bolter argues ‘[w]e understand and appreciate Myst [= a visually rich CD-ROM computer game] precisely because we know what happens in detective films.’ (Bolter 1997 : 4) Alike, Max Payne offers the interactive feature of occasional slow motion that emulates the cinematic style of John Woo’s action films. 
The camera in film has been accepted as a genuine own narrative force. The goal of this chapter is to describe the visual representation in a RT 3D VE comparably as an own layer.

II.1.1 Film Camera
The camera work during the recording of a film is defined by two main elements: the position of the camera including its movements and orientation and the functionality of elements within the camera.
The position of any camera aligns itself along the three axes that define the camera’s angle and its position in space. Movements are possible along any of these axes and can combine movements on different axes. At any given position a camera can also change its angle towards each of the axis by tilting, panning, or rolling.

Figure 3 – Freedom of the Camera	
In addition to its spatial positioning functional elements within the camera define the recording of the picture. This includes, for example, various lenses (enabling zooms and change of focus), different filters (changing the colour range or adding effects), a variable film shutter (defining the speed of the recording), the camera’s iris, and different film stock (that can have different grain, colour range, light sensitivity).
Depending on their specialisation different cameras may offer features that adjust to their special needs (e.g. high-speed cameras). 
[NOTE: this would be a nice place to put in some pictures – e.g. of a film set where the cameras are mounted on giant cranes versus the 3D editor of Zanzy; plus maybe some shots from different cameras in VE]

II.1.2 Virtual Camera
There is no physical camera existent in a RT 3D VE. Therefore, virtual environments allow all of the mentioned camera movements along the axes in total freedom. The camera is not limited by any physical constraints such as collision with walls, dependency on the movement mechanism, or on the studio settings. Instead, it can pass through representations of walls, move in on a miniature detail, or spin into extreme heights.
On the other side, as a virtual camera is not a physical device it lacks the functional parts of a real camera. There are no lenses, filters, shutters, iris, or film material in a virtual camera. The effects of these elements have to be reproduced as part of the rendering of the picture. Their reproduction is technically possible but needs sufficient processing power. For example the film material’s grain can be implemented in a virtual picture by using a shader – that is a special rendering algorithm – during the rendering of the picture. 
Implementing fake camera effects such as a lens flare, which depends on the optical lenses of the camera, has become widespread in games (such as Zanzarah) thus imitating the camera device’s technical functionality. Thus the production method of the effect is change but the visual effect stays available. They are even more available than in real film camera work as these kind of camera related effects can be implemented without any connection to a physical limitation whenever needed in a VE. As a result the rendering program that simulates a virtual camera can change the picture’s “grain” during a single take, switch a lens flare effect on or off independently from any light source, or change the colour balance. 
This mirrors post-production special effects of films, but in a real-time environment these effects have to be presented immediately to the user.
It is this immediate presentation that limits the range of available features in the virtual environments available to wide range consumers. For example, camera effects such as motion blur and changing focus still demand a lot of processing power and became available only recently. 
[NOTE: I should describe the cameras with pics and loads of examples from Common Tales  - maybe mention here already the spatial, conditional, user controlled (interactive) camera functionality?]

II.1.3 Research in Camera Control
Academic research in the field of VE camera is less dependent on the technically limited side of consumer electronics. Due to the more complex input/output devices available in research labs – such as head mounted displays, motion tracking devices, and large projection screens – the VE camera can be approached in different ways. 

Galyean uses a head mounted display in his “Dogmatic” installation. But in order to enable this free first person pov of the environment he has to limit the amount of user freedom. The interaction becomes “guided” (Galyean 1995) while the user encounters the VE as a closed space in which one can look everywhere at any time.

In 1992 Galyean was also involved in the development of the CINEMA system that headed towards a camera control system that allowed the user “to build higher level procedures from simpler primitives” (Drucker/Galyean/Zeltzer 1992 : 2). This system was tested during courses taught by Davenport and Zeltzer at the MIT MediaLab.

Another of the researchers involved in the CINEMA project, Steven Drucker, developed an intelligent camera control implemented in different example settings (Drucker 1994). He used camera primitives that define camera behaviour and can be combined in camera modules. A modular approach that specifies tasks to separate camera operators is a concept that will guide this paper’s approach, too. His conversation filming Camera Agent (Drucker 1994 : 118f) mirrors the functionality of a stage director in a TV studio. But especially this Camera Agent is limited to a very small amount of possible camera positions. 

Figure 4 – over the shoulder shot generated by Drucker’s conversation filming camera agent

Figure 5 – two shots from Drucker’s Virtual Football Game camera example

The Camera Modules developed by He, Cohen, and Salesin at Microsoft Research (He/ Cohen/ Salesin 1996) fulfil a comparable function. They consist of predefined states that can be activated by a Virtual Cinematographer (VC) using a Declarative Camera Control Language (DCCL). The activation of a Camera Module, here, depends on programmable states that relate to the actions inside the virtual environment. They delivered an online “party” application that used the basic camerawork as a demonstration.

For his PhD research in the ALIVE system Blumberg filmed the user and projected him onto a large video screen working as a virtual mirror (Blumberg 1997). Here the camera is functioning like a technical mirror device.

Figure 6– a view at the running ALIVE system by Blumberg

Blumberg already included the option of “switching into” the main character’s camera view. But in his research this point of view was used to avoid obstacles and as a matter of orientation. 

Figure 7  – the dog’s camera in ALIVE (note that this is not a true first person pov)

   

Blumberg’s work is carried forward in the MIT MediaLab’s Synthetic Characters Group. Within this group Tomlinson’s research is the closest to the approach taken in this paper. Tomlinson introduces the idea of a camera agent that reacts to the emotions of the visible characters, a “CameraCreature” (Tomlinson 1999). The camera position is adjusted in relation to the emotional state of the visible character. 

At the Synthetic Character Group a character has six basic emotional states: anger, disgust, fear, happiness, sadness, and surprise. Checking these variables allows a virtual camera to continuously re-define its position and develop an appropriate style. The camera, thereby, interprets the action. It includes the same six states not as animated expressions but as viewpoint and camera variables (Tomlinson 1999 : 49). Tomlinson treats the virtual camera as an expressive being and this approach will be carried forward in this paper. 

Technically the performance gap between researchers’ media labs and consumer electronics is shrinking. Some research projects already use game engines (e.g. the Quake II engine in the Virtual West Cambridge Project at Martin Centre). On the other hand it the implementation of some of the theoretical approaches into the world of commercial game development is getting easier.  

II.2 Sound effects in VE
Next to the visual richness of RT 3D VE the audio output stands out as the second main expressive feature of mediation by RT 3D VE. From classic computer games – such as Myst (Rand&Robyn Miller, USA 1993) that uses sound effects in its basic puzzles – to titles such as Rez (Mizuguchi, Tetsuya, JP expected 2001) that combines shooting action and music creation in one experience – sound is growing more and more important.
Sound is read more unconsciously than the picture. In cinema sound can achieve a lot of strong effects, yet stay relatively unnoticeable (e.g. Bordwell 1993 : 292). But implementing sound carefully in the title allows a focalisation. ‘By selecting certain sounds, the filmmaker guides our perception of the image and the action.’ (Bordwell 1993 : 299) Virtual environments, like cinema, combine visuals with sounds that get unconsciously combined in the audience’s/users’ brain and generate a combined impact. ‘The result is that we see something on the screen that exists only in our minds’ (Murch in: Chion 1994 : xx) That way sound and visuals merge into an active mediation of the world. Within this mediation the sound level can be an important element for the user to understand the VE, construct a meaningful event, an unfolding story from the action, and attach the user emotionally to this event.

‘The eye blinks when you turn your head to look at something else, but the ear never blinks. We are working hard with the programmers to make certain that there are no seams in the soundtrack - no moments of dead silence that will break the spell of the sense of place.’ (O'Donnell, 1997 : 638) O’Donnell already mentions the importance of sound to create a ‘sense of place’. Applying elaborate sound effects to virtual objects and spaces can enhance and enrich the manoeuvring in virtual 3D spaces. The sounds of a virtual waterfall add significantly to the sensual quality and presence of its visual representation in virtual space. Visually represented elements and designated sounds become extremely closely connected.
If, for example, Lara Croft – the virtual heroin in the Tomb Raider series (Core, UK 1996 -) – pulls a virtual lever not only this lever’s position changes but also a recognisable sound effect is triggered. The sound effect signifies the interactive access. This sound signification can be enhanced to a degree that no graphical representation is necessary. As long as the lever sound is audible the lever does not have to be in view – users will understand that this lever has been activated [Murch].

But not only specialised interaction with individual objects gains through the sound level. Spatial interaction can be intensified to a great extent through the proper use of sound and may serve as an even more valuable feature in the absence of tactile effects. In a RT 3D VE the user not only navigates between shapes but also between sounds. Thus, soundscapes become integral part of the space represented in VE. Current interactive titles offer spatial sound features that allow implementing sound sources in a virtual space (see e.g. Poole 2000 : 81 about the use of spatial sound in Perfect Dark (Rare, UK 2000)). Some titles already start to implement physical representations of sound heading towards a “sound perspective”. 

Sounds will add to the user’s understanding of the virtual world. For example, based on the work on cognitive maps done by Kevin Lynch Darken\Silbert introduced an ‘acoustic landmark’ (Darken\Silbert 1993 : 3). Their experimental world showed that the combination of visual and audio clues did enhance orientation in virtual environments.
Visitors seem to read and imagine a space much easier, when the visual structure of the space is enhanced through a sound layer. E.g. combining the sound of driving cars with a busy street might speed up the user’s understanding of this special space as “the street”. Sound helps to distinguish objects within the environment. Ambulance cars get identified by their sirens from far away, a person can be identified by her voice, an invisible bird can be recognised by its singing etc. As the objects become more distinct, the VE becomes more focussed and accessible and the space is easier to understand. [MT wants more examples]

II.3 Music in VE
Music as a special part of the sound mediation offers a variety of representational features. It functions on a very basic reception level ‘“because it is almost directly plugged into the psyche.”’ (Gorbman 1987 : 58 quoting Tallon) Thus it operates as a hidden but powerful mediation tool.
Music in film mediates. Its nonverbal and nondenotative status allows it to cross all varieties of “borders”: between levels of narration (diegetic/nondiegetic), between narrating agencies (objective/subjective narrators), between viewing time and psychological time, between points in diegetic space and time (as narrative transition). Finally, the connotative values which music carries, via cultural codes and also through textual repetition and variation, in conjunction with the rest of the film’s soundtrack and visuals, largely determine atmosphere, shading, expression, and mood.

(Gorbman 1987 : 30 for mood in a VE see also Poole 2000 : 83) Potentially all these functions can be implemented in virtual environment. Spatial sound, dolby surround, live sound broadcast, adding sound from a CD, filtering and distorting sounds, can be available in a RT 3D VE. Microsoft’s DirectX sound or middleware products such as the Miles Sound System by RAD Gametools allow most of these possibilities – and more. Still music in VE is rarely as elaborated as in film. A main reason for this lack is, that the timing and even the occurrence of events is not strictly predefined in a VE as it is in film. A timing of soundtrack and visible event is not as easy as in film. 
Microsoft’s DirectMusic (part of its DirectX system) offers a dynamic change of a playing musical piece that relates to the changes in the presented event.
 but even elaborate systems like that (or LucasArt’s elder iMuse system) cannot yet achieve the precision of film music in relation to the picture. 

[include a discussion of Koanx? http://www.yamaha.co.uk/xg/html/software/s_koanx.htm (well I am not so sure about the quality of this thing –MIDI?) Deliver some outlook to how things will evolve in the future – and relate to other interactive chaps (like Aarseth) and film people (maybe Bordwell?) – this all feels already very much as part of the narrative discussion and makes more sense in the context of diegetic and non-diegetic stuff and narrative layers]

Therefore, two far more important effects of music’s mediation in VE are the definition of the mood and the connection of spatial locations by establishing motifs (for a further discussion of motifs in VE see below). Setting a special mood improves the immersion of the user and connecting different events through the music raises the structural qualities of the piece.  Like film music ‘bonds: shot to shot, narrative event to meaning, spectator to narrative, spectator to audience’ (Gorbman 1987 : 55) music in VE can connect event to event, events to narrative, user to narrative, user to user within a RT 3D VE as this bonding does not necessarily depend on the precise coordination of picture and tune but works on the more general level of re-using a special melody or motif. Thus, the memorable main theme of Tomb Raider gets adjusted to an ‘Italian version’ (see also Poole 2000 : 83) when the trail of the adventure reaches a virtual reproduction of Venice. Of comparable importance to the content is the original John Williams soundtrack for Star Wars (Lucas, USA 1977) as a unifying element in the spreading media universe of Star Wars articles that include RT 3D VE such as Star Wars: TIE Fighter (Holland/Killham, USA 1994) or Star Wars: X-Wing (Holland/Killham, USA 1993). Here the music combines the different media of film and RT 3D VE as well as different virtual environments to each other. Users know what kind of universe they enter when they hear the familiar tune and this familiarity transcends between different media.

III Functionality

III.1 Interactive Access
The programming code of a VE limits the world to certain rules. ‘Beyond the fantasy, there are always the rules. (...) When you play a video game you enter into the world of the programmers who made it.’ (Sherry Turkle in: Provenzo 1991 : 22). This programmed world is always ´”rule-governed”´ (Sherry Turkle in: Provenzo 1991 : 66) and the machines it runs on ‘are not merely law abiding; they are embodiments of law’ (Weizenbaum 1984 : 40; see also ‘code is law’ Mitchell 1995 [: 111]; also Lessig 1999 : 6). Within these rules the interactive access of the user to the environment is defined it is part of the computer program.

The computer programmer, however, is a creator of universes for which he alone is the lawgiver. So, of course, is the designer of any game. But universes of virtually unlimited complexity can be created in the form of computer programs. Moreover, and this is a crucial point, systems so formulated and elaborated act out their programmed scripts. They compliantly obey their laws and vividly exhibit their obedient behavior. No playwright, no stage director, no emperor, however powerful, has ever exercised such absolute authority to arrange a stage or field of battle and to command such unswervingly dutiful actors or troops. 

(Weizenbaum 1984 : 115 see also in: Walker 1994 : [445])
Empowered through the feature of programming interaction is a defining feature of computer-based media. It is an offspring of digital data handling by the computer and is single most important point where software differs from film or architecture in the functionality of the represented moving images.
We define as interaction that kind of cooperation of at least two entities that relates to each other. Interaction can happen on various levels with various other entities at the same time. In the case of computer-generated virtuality the new part of this interaction is the one between the human user and the contents of the virtual environment, through the medium of the computer and vice versa (for “computer as medium” see Laurel 1991; Laurel 1993). 
Here we concentrate on the physical part of this interaction, what Espen Aarseth (Aarseth 1997) defines as ‘ergodic’ participation. During the interactive participation with the virtual world ‘the user will have effectuated a semiotic sequence, and this selective movement is a work of physical construction that the various concepts of 'reading' do not account for.’ (Aarseth 1997 : 1) Examples for this movement are a mouse-click, pressing a key on the keyboard, or moving a joystick during an interaction with the virtual world. The user’s physical input is the ‘ergodic’ participation in the title. It can be any kind of physical activity (e.g. Char Davies’ Osmose uses the expanding chest when users breathe, console games use the varying console controlling devices, PCs mainly use the keyboard).

Notably navigating in a virtual 3D world lead to new interface designs to optimise this participation, such as the joystick or the JoyPad. 
As ‘ergodic’ is one of Aarseth's main defining terms the lack of discussion of different input-output devices is striking. Not only does the joystick offer new ergodic interaction but also it has been developed further into a force-feedback device that means the game/text has a physical output to the user. In addition to the joystick, there are plenty more devices such as steering wheels, data-gloves, ergonomically designed gamepads, trackballs, dance-pads, and even maracas that offer different interaction methods. 
Aarseth also does not cover the features of analogue controls. Analogue controls can be found in most game console interfaces. They are touch sensitive devices: depending on the force the player uses to press a button the result will vary. This has direct influence on the ergodic participation as the player's input will be measured in a much more accurate way. As Aarseth concentrates on the usually insensitive keyboard and mouse input devices he does not discuss this option and its use for ergodic literature although its importance is obvious. 
If one compares the difference of a real piano allowing different strengths of hitting the same key with an insensitive electronic keyboard, one immediately realises the different qualities of the emerging ergodic participation and the evolving musical piece. 
Nevertheless, Aarseth’s notion of ‘ergodic’ participation proves useful as it puts emphasis on the physical action needed by the user to generate the text in collaboration with the virtual world. Aarseth did focus on the main important point of necessary interaction.

One has to build-in the possible interactive access or there will not be any manipulation at all.
On the one hand this is a severe limitation as VE cannot build on established patterns; on the other hand, this limitation is the basis of shaping the unfolding experience one can have with VE and therefore a powerful narrative option. Interactors co-create the unfolding event accordingly to a set of predefined rules in the virtual environment. What a narrative 3D virtual environment, then, has to accomplish is a dramatic structure of this co-creation, these readings of virtual space. 

III.2 Interface

The technical backbone of interactivity is the interface. The interface consists of input device and output device both parts are active elements in the event of the interaction itself. Their functionality has to be defined in the process of interface design. The ‘goal of all our [the interface designers’] efforts, is to empower the user. But empowerment itself is a notion that must be unpacked.’ (Mountford/Laurel in: Laurel, 1990 : xi) The unpacking is done within the code that defines the interface.
Interface design has to fulfil three basic quality criteria:
-	fit to the user – it has to be appealing, usable, and understandable 
-	fit the technical specification – it has to relate to a computers’ technical equipment 
-	fit the environment – it has to relate to the whole environment’s design 
There are various devices that include input and output features in one, such as force-feed-back joysticks or touch-screens. Arranging the features of each interface, that is, presenting the user’s opportunities in the most appealing way is the task of the human-computer-interface design. 
John Walker describes different generations of interface:
‘knobs and dials’; the physical direct control using switches and other physical input devices (e.g. the computer built by Alan Turing for the decoding of the Enigma code 1939-40) 
‘batch’; a user gives a request to the computer operator and receives an answer without direct contact with the PC (e.g. the use of cards that hold necessary information)
‘timesharing’; allowed the user direct interaction with the computer and immediate results (e.g. typing in a line of mathematical commands and receiving the result)
‘menus’; hide the program behind a textual interface (e.g. the pull-down menus of many word-processors) 
‘direct manipulation’ systems; widespread examples of this Graphical User Interface, GUI, are: the Apple MacOs or Microsoft’s Windows (Walker in: Laurel 1990 : 442 f; see a single-standing version of this article: Walker 1994; also: Ramm 1994 : 28; also further description for interface design and history. Walker does not discuss the 3D interface experiments such as Henderson/Card’s “Rooms” interface (see e.g. Shum 1990 : 10) nor sound interfaces (see e.g. Shneiderman 1998 : 275ff)). 

This paper does not follow the historical development of computer systems, but in stepping into the real-time three-dimensional realm, will question this development. As current (December 2001) examples of the MacOs and the Windows operating system show, these criteria headed mainly towards the functionality of computer desktops and depended on direct manipulation. Shneiderman himself distinguishes between application and game interface design: 

Studying game designs is fun, but there are limits to its applicability to the lessons. Game players seek entertainment and focus on the challenge of mastery, whereas application users focus on their task and may resent too many playful distractions. The random events that occur in most games are meant to challenge the user; in nongame designs, however, predictable system behavior is preferred. Game players are engaged in competition with the system of with other players, whereas application-systems users prefer a strong internal locus of control, which gives them the sense of being in charge.

(Shneiderman 1998 : 197) Shneiderman basically does not apply the idea of the computer as a medium and stays on the level of the computer as a tool. Nevertheless, for a basic interface discussion, his findings are helpful and will be related to later. In this context Ben Shneiderman outlined three key criteria as quality standards of direct manipulation:
‘1. Continuous representation of the object of interest
 2. Physical actions or labelled button presses instead of complex syntax
 3. Rapid incremental reversible operations whose impact on the object of interest is immediately visible’ (Shneiderman in: Laurel 1993 : 8).
At other places (e.g. Shneiderman 1998) Shneiderman outlined more criteria but for the argument of this thesis the mentioned three key criteria will serve as helpful guidelines for a later discussion of narrative 3D environments. 

The virtual environments this paper discusses have been defined as three-dimensional virtual spaces. Thus, the interface has to relate to this and has to serve two main goals: movement inside the virtual world (see Murray 1997 : 129f for the pleasures of navigation without special interaction) and specialised interactions with parts of the virtual environment. With this in mind one has to return to the question of interface design. 
Some VE, such as Char Davies work Osmose (Davies, 1995) limit the interactive access to the movement. Osmose lets the user explore a 3D world by using the breathing as the main input device that regulates the movement during this exploration.
Other pieces, such as the Wayfarer VE navigation is the designated main interaction supported by other features. Wayfarer asks users to find certain checkpoints within a virtual environment. There are limited interactions at these checkpoints that add to the experience but the main task remains the pure exploration of and navigation in the VE.
Finally, most game environments (such as Half-Life, Shenmue, or Black & White) combine spatial exploration with specialised interaction. These interactions can include access to objects within the world (e.g. picking up virtual objects in Shenmue), access to the space itself (e.g. change the virtual natural environment by designating corn fields in Black & White), or interaction with other users (e.g. chatting with other players in a online session of Half-Life).
On the level of these specialised interaction most 3D VE use a ‘direct manipulation’ approach that relates back to a representation of Walker’s physical ‘knobs and dials’. 
So, in the last forty years we’ve taken the computer user, who was initially in direct control of a dedicated computer, operating it by switches and gazing at huge arrays of blinking lights, to greater and greater distances from the computer and direct interaction with it, then back again to contemplating a virtual control panel on a glowing screen filled with slide pots, radio buttons, meters, all providing direct and expressive control over what’s going on inside the computer. 

(Walker 1994 : [444]) 
Direct manipulation has a very low level of abstraction as the operator has to directly interact with the operating machine. ‘Since the user [of the knobs and dials computer era] was the operator of the machine and controlled it with little or no abstraction, there was essentially no mediation between the computer and its expert user.’ (Walker 1994 : 440) The computer stays a tool with immediate functionality.
Virtual representations of knobs and dials attempt to lower the level of abstraction within the virtual environment. The give the impression of immediate, unmediated active elements. Shooting a virtual gun has an immediate reaction within the environment. This makes the object (e.g. here the gun) easier to understand and also easier to put into relation to the environment (e.g. if the user shoots at some element within the VE this element might receive an impact). This immediacy makes virtual representations of old-fashioned solid input devices in RT 3D VE valuable as it makes the interaction with the RT 3D VE appear to be direct. (Tegolo also hints towards this direction although his Dynamic Visual Icons are still 2D; in Brusilovsky 1996 : 136ff). 3D games do this step explicitly: see for example the enormous levers in Tomb Raider (UK, Core, 1996) or the colour-changing buttons in Half-Life. As these virtual spaces are shaped like representations of physical worlds their interface’s appearance and functionality relates back to the days of physical interaction. Thus, ‘[t]hese games provide a field of action that is visual and compelling. The commands are physical actions – such as button presses, joystick motions, or knob rotations – whose results are shown immediately on the screen.’ (Shneiderman 1998 : 196)

Both interaction methods – navigation and specialised concentration – have to be combined in a VE while the use of menus, type-in boxes etc. should be avoided as spatial movement intrinsically differs from these input methods.

Interaction – as it is envisioned in this paper and outlined in the introduction – can happen on direct and indirect levels. It can combine them and influence different levels of a VE. It can include different kinds of manipulations during one ‘ergodic’ interactive act. This is a significant difference to the way operating systems (or practical software applications such as Microsoft Word, Netscape Navigator, or Alias Wavefront’s Maya) work. The reason for this complex approach to interaction methods is the equally complex aim of the VE this interaction is set in: to implement narrative into RT 3D VE.
Developing a narrative is very different from providing a purely functional operating system and Walker’s as well as Shneiderman’s criteria for interface design have to be discussed in this new field.