WavRide An Audio Game Experiment

1. WavRideAn Audio Game Experiment CSCE 4250/5265 Project Spring 2013 Sam Johnson

2. Papers Studied • T. Drewes, E. Mynatt, M. Gandy, “Sleuth: An Audio Experience.” • S. Papetti, D. Devallez, F. Fontana, "DepThrow: a physics-based audio game." Proc. of the International Conference on Auditory Displays, Paris, France. 2008. • T. Stockman, N. Rajgor, O. Metatla, L. Harrar, “The design of interactive audio soccer.” In Proc. of the 13th International Conference on Auditory Display (pp. 526-529). June 2007. • T. Roden, I. Parberry, “Designing a narrative-based audio only 3D game engine.” Proc. of the International Conference on Advances in Computer Entertainment Technology, ACE 2005 • N. Röber, M. Masuch, “Playing Audio-only Game: A compendium of interacting with virtual, auditory Worlds. In Proc. Of Digital Games Research Conference 2005, Changing Views: Worlds in Play 2005 • D. Archambault, "The TiM Project: Overview of Results." In Computers Helping People with Special Needs, pp. 248-256. Springer Berlin Heidelberg, 2004.

3. The Objective • Rather than focus on implementing a specific algorithm to solve a particular problem, the goal of my project was to conduct an experiment by utilizing various ideas from the researched papers to create an original game with a unique approach to the idea, and to study a range of elements of player interactions and responses to the game. • The topic of the project is audio-only games.

4. The Objective • To create a computer game with which the player interacts solely through audio (and possible controls such as a keyboard or mouse). No video is used at all in the game. • The game must strive to convey all elements (things like environment, movement, game objectives, control feedback, success/failure, etc.) in a way that is as universally and quickly understandable as possible. Depending on the type of game, these elements can be numerous, and present a unique challenge when working with only audio.

5. The Objective • To carefully design the type and elements of the game to find a balance between synergizing well with an audio-only interface, while avoiding being too limited by said interface. • To create a new, unique contribution to this type of game design. Attempt something that hasn’t been done. • In many ways the audio-only interface can be restrictive, but I discovered it can also allow for new possibilities for freedom in design.

6. The Objective • Finally, to use the game to study in as many ways as possible how various players interact with it. • The focuses of study included how players create a world space from the sound environment, how they learn and adapt to different situations, how they identify objectives, how they tend to move, and how they can adapt to spacial navigation without the use of vision, among other things.

7. Prior Work • The works I studied in particular were the three previously developed complete audio-only software games: “Sleuth,” “DepThrow,” and “MudSplat (aka the TiM Project).” • The gameplay in Sleuth focuses on listening to conversations and other audio cues to ultimately solve a mystery. • DepThrow is a physics-based game where the player uses a Wii Remote to throw a ball at a certain velocity to hit a target. • Mudsplat is an arcade style game where the player moves left and right to shoot monsters in a line. This game was specifically developed for vision-impaired children.

8. Contribution of My Paper • Full, real-time 3D environment. • Fast-paced movement and gameplay. • Subtle changes in many layers of audio effects to convey emitter position. Lots of research in audio engineering went into developing this. • Subtle positional changes in many layers of emitters to create a realistic feel to movement. • Library of original audio files created specifically for this game, actively layered to create a realistic world space. • Active background music system that evolves according to actions and progress. • Zero narration or any otherwise explicit instructions for what to do. This was the most important priority for me throughout development and presented the biggest design challenge, as well as in my opinion the most significant achievement of development. • Unique tracer system to intuitively identify objectives. This evolved from AudioSnipe, a different, earlier version of the project. • Player control types that have (as far as I know) not yet been done, i.e. point and click (in first project) and fast 3D movement (in final).

9. Your Implementation • As I was researching the games in the beginning of the project, I noticed a trend. There was usually a catch due to the audio-only setting which could be avoided with video. Many were either narrative-based or relied on spoken language to convey information. The ones that did use a 3D environment tended to have a restriction such as the limit of rotation to 90 degree angles. • I wanted to attempt to design an audio-only game involving a player interface that would be considered restricted by lack of video. (I also noted the significant chance that it would fail). • The interface I chose was a mouse. This led to my 1st project…

10. Your Implementation (AudioSnipe) • The first project was a game called AudioSnipe, a First Person Rail Shooter. • The core challenge was to allow the player to use the mouse to ‘look around,’ like any typical computer FPS, and pinpoint a target in the crosshair, without seeing anything on the screen. • For this project, I used DirectSound. • After many iterations of ideas I developed a tracer system, where multiple emitters are positioned based on the cursor’s relation to the target. The most successful of these used a circle of eight tracers, with rapid pulsing of different pitches in a specific pattern

11. Your Implementation (AudioSnipe)

12. Your Implementation (AudioSnipe) • I tested this with 12 or so people. I did not give them any information or tell them what the object was, only that the mouse was the controller. I started the program, turned of the monitor, had them put on headphones, and told them to let go of the mouse and tell me when they thought they had succeeded. • What resulted was that 100% of the time the cursor was hovering over the target. This was a far bigger success than I had predicted. • This success made me want to study how people learn and respond to abstract situations like this one, which led to the main project…

13. Your Implementation (WavRide) • As I was looking at ways to convey positional audio with greater control over DSP effects for positional audio, which involves real-time variation of levels of reverb, stereo panning, and most importantly, selective filtering and EQing, which regular DirectSound didn’t seem to do very well. I discovered xaudio2, a newer audio subset of DirectX which allows for real-time multichannel DSP for any effect imaginable. • Using xaudio2, I built WavRide. • The idea behind the game is that the player is sliding forward in first-person (very fast) down the side of a mountain, on a snowboard or whatever you like. The controls allow you to move left and right, and jump. There are two objects: To collect powerups along the way, and to successfully jump off ramps along the way.

14. Your Implementation (WavRide) • The powerups are signified by a modified version of the tracer algorithm from the other game, using string samples that produce a chord. In this, I discovered a really cool benefit of working in an audio-only game. Rather than restricting player movement within a certain path, the player is free to slide indefinitely in either direction. Instead of being stationary, the powerups are spawned a certain distance in front of the player with a random x value within boundaries set by the player’s position. With video, this sort of dynamic spawning would become easily apparent to the player in such a fast-moving environment. But in audio, the detail is hidden. • The ramps are signified by a rising audio cue, and if the player times it right, he will perform a successful jump. • There are success (rising major sine wave scale) and failure (falling chromatic sine wave scale) sounds to that play when an objective is hit or missed.

15. Your Implementation (WavRide) • There are multiple layers of wind and snow contact sounds, positioned accurately in relation to the player. If you turn in a direction, the wind emitters and slide emitters will adjust as if you turned that way on a snowboard, as well as queue small swoosh of the snow to the opposite direction. Same thing with the sliding sounds. Jumping queues jump up and impact sounds, removes sliding, and subtly adjusts wind filtering and reverb to emulate rising and falling y-positioning. This was all done by trial-and-error until it ‘felt right’. The end product is it really feels as though you are moving through 3D space. The audio-only aspect tends to enhance the immersion. • There is also an active music system that consists of a rhythmic and a melodic part. The rhythmic part is fast-paced cinematic drums that is always on except when the player jumps, in which case the drums stop, and resume on the downbeat upon landing, to add to the impact. The melodic part starts off silent and actively adjusts its volume in relation to how successful the player is throughout the game.

16. The Experiment (WavRide) • I created a website to host the game and asked people to download and play it, and created a survey on the site to provide feedback for the research. • I also implemented the ability for the game to record certain aspects of the player’s progression, and it saves all of this data in an xml file. Part of the survey asked players to attach this XML file. • There is no narration in the game. No languages cues at all. All sounds are either environmental or musical. • The only information I provided players who tested the game was what buttons on the keyboard were considered controls, and I did not tell them what those controls did. I did not tell them what to do, what the environment was, or anything. • There is a short ‘tutorial’ introduction I added at the start of the game to teach the player the controls. This tutorial also does not contain any information. During the tutorial the player is not sliding but rather walking through the snow. The object is to walk to two powerups on each side then time a jump. Once the jump is completed the real game starts. I was afraid some people might get stuck on this, as I did not mention anything about it.

17. Results – actual quotes • Question 1) Describe the environment your character was in at the START of the game, and how you moved through it. • “The environment sounded like a windy, snow filled field.”“It felt like a 2d side scroller to me so it felt like I was moving in a straight direction either left or right. I remember the music switching from ear to ear and using that to get my bearing.” • “just wading into the water of an ocean and seeing the waves crashing beneath me.””It felt like you were kind of floating through the world” •  ”There was a slight build-up that gave the idea of an introduction. Desolate and cold, like a tundra.” ”Quickly, the pace of footstep sounds implied some sort of haste.” • “seemed like i was in a sewer at first because of the splash of the jump, but as the music progressed it felt like i was leaving the sewer or something.” “I felt like I was trying to climb something towards the beginning....”

18. Results – actual quotes • Question 2) How would you describe what each of the 3 controls did? • “A moved me right, D moved me left, and space made me jump.” • “although true direction is impossible to determine without visual, it would be most easily described as left-to-right and jump.” • “A moved left, D moved right, space was jumping or swimming over crashing waves.” • “I didn't realize you could use space/a/d. I was using the numeric pad 1/4. And I figured one moved one direction and four moved the other (left and right). I don't know what space did. Let's say jump because those are the bare functions you expect in a game.”

19. Results – actual quotes • Question 3) What would you say were the objective(s) were at the START of the game, if any? • “I wasn't sure what the first objective was, but i felt like i was climbing something, like a series of platforms in which you had to jump from right to left and left to right etc. all i know was that i obtained something in the game.” • “To find the loudest point of the sounds or music. The first 2 had very clear objectives and I figured them out very quickly. The 3rd was the hardest for me taking me about twice as long as the first two levels.” • “ The objectives at first seemed to be just follow the sound. Much like Legend of Zelda: Ocarina of Time and going through the lost woods. Then the harsher sound of the wave coming at me gave me the idea that space was probably a jump or a dodge sort of move.” • “the objective was to orient the character in a specific location using only audio clues. this involved determining which direction the sounds were coming from. The objectives felt clear once you were moving.”

20. Results – actual quotes • Question 4) How would you describe the environment your character was in during the MAIN part of the game? • “It felt like i was on top of something watery, as, again, the jump sound sounded like splashing. The music was very intense and made me feel as if i needed to hurry.” • “A 2d side scrolling snowy field.” • “A vast beautiful ocean with seemingly no end and there was probably a tribal island behind you.” • “it felt confined, potentially due to how fast the character moved.”

21. Results – actual quotes • Question 5) How do you feel your character moved through/interacted with the environment during the MAIN part of the game? • “Falling/sliding. The sound of shoes sliding on ice/snow” • “either to the left or right like a 2d scroller. The sounds getting louder as I traveled in either direction and the jump sound effect when I jumped.” • “I kept moving to the right over and over. The music would get louder and a heartbeat sound would increase in speed, but after continuing to move right, it'd stop with a pause and then a splash from a jump; i'd do it over and over again (about 3 times) until i realized that i was falling off of something. I knew spacebar meant jump, and i knew i had to jump, but i didnt know exactly when. So when i moved right the last time, i mashed the spacebar over and over until i got to the goal.” • “Falling off of cliffs and going through some kind of portals. Audio clues were collision-like sounds for the falling and ringing synths and pitch shifting for the portals.”

22. Results – actual quotes • Question 6) What would you say were the objective(s) were during the MAIN part of the game, if any? Were the objective(s), if any, clear or vague? Were the objective(s), if any, easy or difficult to achieve? • “I felt that the objective was to try and obtain something, as the pulsing noise that i heard towards the end of the game compelled me to go towards it and reach it.”“The objectives during this part of the game were pretty clear; the sound/noises pointed me in the right direction. It was frustrating how I kept falling over and over.” • “ To find the loudest point of the music and/or sounds. Sometimes it required just running to them, or jumping. The sounds switching from ear to ear and getting louder.”“objectives were very clear in the first 3, the last one I'm not positive of the objective (although I assume it was the same).” • “  Follow the sound, jump over the waves. Crashing wave sounds gave me the thought of wading in the ocean.”“The objectives were easy enough to get. Like I said that harsher sound made me think it was something to dodge and space is usually the jump or dodge button in PC games. The difficulty did get harder. I could quite get the timing.” • “to get to the point where the sounds are strongest and clearest, sometimes requiring you to be there at a time coinciding with a specific sound.”“The objectives were pretty clear with a little experimentation. it became a little difficult to navigate the environment accurately towards the end.”

23. Results – actual quotes • Question 7) Finally, how well do you think you scored in your play through the game? Why do you think this is?  • “I think I scored pretty well, as I feel like I caught on quickly.” • “I think I scored well. I think I figured out how to play the game pretty quickly and I successfully completed all of the levels.” • “I think I did bad near the end. I just could not get the timing down.” • “Poorly, mostly because I sat going in one direction for a good 4 minutes before I figured that I might be going the wrong way.”

24. Results – Stats (PowerUps Collected)

25. Results – Stats (Ramps Jumped)

26. Results – Stats (Path Taken)

27. Conclusion • I believe this project has provided several conclusions and raised many more questions. • The most significant conclusion I can make from this research I think is that there definitely exists a stronger capacity for intuitive spacial recognition through sound in the human brain than I would have thought before performing this experiment.