Butterfly Effect : An Augmented Reality Puzzle Game

1. Butterfly Effect:An Augmented Reality Puzzle Game Marleigh Norton Blair MacIntyre Steven DowMaribeth Gandy

2. Game Overview • Spatial puzzle where physical world is not modeled • Virtual butterflies in physical environment • Catch them! • Butterflies are stationary • Challenge comes from reaching the butterflies • Bring butterflies into reach by rotating the virtual world • Virtual world rotates in 90 degree chunks about a player defined axis.

3. Equipment

4. Head-Mounted Display

5. Tornado Stick

6. Rotations • Tornado Stick controls Virtual Axis • First Button – Place / Remove Axis • Second Button – Pause / Unpause Axis • Third Button – Rotate Butterflies

7. Tornado Stick • People had trouble understanding rotations • Not everyone is used to 3D geometry • Though 3D puzzles have worked before • Change the metaphor, not the interaction • “Tornado Stick”

8. Butterfly Effect

9. Butterfly Effect Video

10. Design Context • Start with medium, then design game • Mapping physical and virtual worlds is hard • AR works best when user is moving slowly • Tracking more accurate • Safer for player – Cognitive Tunneling

11. Design Context • Home environment • Run on next generation game consoles • Must be easy to set up • Minimize use of special equipment

12. 1st Prototype: 2D Paper

13. 2nd Prototype: 2D Interactive

14. 3rd Prototype: Video

15. Tuning • Two Major Challenges • 90 degree rotation constraint • Understanding virtual space and physical space • Possible Changes • Loosening rotation constraint • Adding more manipulations (translations) • Adding better depth perception cues

16. Open Issues: Depth Perception • Static Depth Cues

17. Open Issues: Depth Perception • Remove Automatic Depth Cues

18. Open Issues: Depth Perception • Remove Unavailable Depth Cues

19. Open Issues: Depth Perception • Texture probably the best

20. Without Grid

21. With Grid

22. Is Occlusion Really Impossible? • Ad hoc modeling • Automatic modeling during calibration • Fiducials • Real time depth • Depth from stereo • Something to be explored

23. Virtual Obstacles - Bees • No consequences for using poor strategy • Bees behave like butterflies • “Sting” if caught • Penalty • Lose life • Butterflies escape

24. Questions?

25. Old Slides from Past Talks (In case people as a question I have slides for.)

26. Butterfly Effect:An Augmented Reality Puzzle Game Marleigh Norton Technical Advisor: Blair MacIntyre

27. Person Walks Through Game Area

28. Person Walks Through Game Area

29. Person Walks Through Game Area

30. Derive Walls from Path

31. Derive Walls from Path

32. Why this is Fun • Fun in the same ways Rubik’s Cubes are fun • Caillois’s Four Categories of Games [1] • agôn, alea, mimicry, ilinx • MacTavish’s “astonishment at visual and auditory technology” [2]

33. Open Issues • Scoring • Capture Interaction • Depth Perception

34. Open Issues: Scoring • Score based on efficiency, not speed • Penalized for moving too much • Accrue score as you walk • High score bad • Problem: High Score Traditionally Good

35. Open Issues: Capture Interaction • Location-Based Capture • Capture butterfly by getting close enough • Pros: Technically simple. No additional cost. • Cons: Precise head tracking more important. Less thematically appropriate. Less fun. • Butterfly Net Capture • Capture butterfly in a net • Pros: More active, thematically appropriate interaction. • Cons: Requires an additional tracking system, likely expensive.

36. Open Issues: Depth Perception • Judging depth in virtual worlds is difficult • Overcoming this difficulty is part of the game, yet it should not be impossible • Problem: What can we do to help the player understand the relative locations of the butterflies? • Research on Depth Perception yields several techniques [4] [5] [6] [7]

37. Questions & Feedback • Scoring – Efficiency • Capture Interaction • Location-based capture • Butterfly Net Capture • Depth Perception • Which cues to use? More ideas? • Other Feedback

38. References: Game Theory • [1] Caillois, Roger, “Man, Play, and Games,” trans. Meyer Barash, University of Illiniois Press, 1961 (orig.1958) ch 2. • [2] Mactavish, Andrew, “Technological Pleasure: The Performance and Narrative of Technology in Half-Life and other High-Tech Computer Games”, in: Geoff King and Tanja Krzywinska (eds.) Screenplay: Cinema/ Videogames/ Interfaces (London: Wallflower Press, 2002) 33-50

39. References: Depth Perception 1 • [4] Cutting, J. E., and P.M. Vishton, 1995. “Perceiving layout and knowing distance: The integration, relative potency and contextual use of different information about depth”, in Perception of Space and Motion, W. Epstein and S. Rogers, Eds. Academic Press, New York, 69--117. • [5] Furmanski, Chris, Ronald Azuma, and Mike Daily. “Augmented-reality visualizations guided by cognition: Perceptual heuristics for combining visible and obscured information”, in Proc. IEEE and ACM Int'l Symp. on Mixed and Augmented Reality (ISMAR 2002) (Darmstadt, Germany, 30 Sept. - 1 Oct. 2002), pp. 215-224.

40. References: Depth Perception 2 • [6] Sinai, M.J., W. K. Krebs, R. P. Darken, J. H. Rowland, and J. S. McCarley, 1999. “Egocentric distance perception in a virtual environment using a perceptual matching task”, in Proceedings of the 43 rd Annual Meeting Human Factors and Ergonomics Society, 43, 1256-1260. • [7] Surdick, R. T., E. T. Davis, R. A. King, and L. F. Hodges, 1997. “The perception of distance in simulated visual displays: A comparison of the effectiveness and accuracy of multiple depth cues across viewing distances”, in Presence: Teleoperators and Virtual Environments 6, 5 (October), 513--531.

41. References: DART • [8] MacIntyre, Blair, Maribeth Gandy, Steven Dow, and Jay David Bolter. "DART: A Toolkit for Rapid Design Exploration of Augmented Reality Experiences." To appear at conference on User Interface Software and Technology (UIST'04), October 24-27, 2004, Sante Fe, New Mexico.

42. Implementation Status • It’s playable!* • Populate the world with a pre-defined set of butterflies • Butterflies can be captured • Tallies of caught and remaining butterflies are displayed • Butterflies can be rotated about the axis • Physical axis interface * When all of the equipment is working.

43. The Screen

44. Feedback • Rotation Interface • Depth Perception • Score • Aesthetics

45. Rotation Interface • Problem: Rotating about an axis? Huh? • Suggested alternatives: • Randomization • Jar, Mini-version of gamespace • (Rotation about a point) • “Tornado Stick” • Changes: Tornado Stick • Axis/Tornado Stick only interface in which all manipulations are valid moves • Arrows pointing to butterflies’ final location

46. Depth Perception • Problem: It’s hard to tell where the butterflies are, especially when they’re “behind walls” • Complication: can’t change appearance of butterfly based on presence of wall(or can we…?) • Changes: Add grid with drop shadows • Also, it’s less of a problem when you’re actually playing

47. Score • Problem: for Efficiency-based score, lower was better (confusing players) • Suggestions • Timer • Subtract from High initial score • No Score • Changes: No Score • Score isn’t really that important • Was causing more problems than it solved

48. Aesthetics • Problem: What will the game look like? • Collecting the butterflies should be compelling • Changes: Aesthetics more fully fleshed out • Childlike graphic design with scanned crayon textures • Player is frog, eating butterflies

49. Frog has Butterfly in Sights

50. Frog Catches Butterfly