1 / 69

Multi-Modal Visualization Methods

Multi-Modal Visualization Methods. The use of Haptics, Graphics, and Sonification to Better Interpret Multi-dimensional Data. By Doanna Weissgerber 7/19/01. Overview. Goal of my research Purpose of scientific visualization Methods of visualization Why multi-modal methods?

hyacinth-roy
Download Presentation

Multi-Modal Visualization Methods

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multi-Modal Visualization Methods The use of Haptics, Graphics, and Sonification to Better Interpret Multi-dimensional Data By Doanna Weissgerber 7/19/01

  2. Overview • Goal of my research • Purpose of scientific visualization • Methods of visualization • Why multi-modal methods? • My Previous and present research in visualization • State of the art in visualization • Proposed research University of California at Santa Cruz

  3. Goals Passive haptics is an information channel which has not been researched • I am most interested in passive haptic information displayed on it’s own information channel • non-redundant • I will employ usability testing to prove effectiveness of haptic mappings • In combination with other modalities when appropriate • Watching for cross-modal problems and enhancements • Alone when appropriate • Watching for enhancement of a performed task • My starting basis will be in scientific visualization • Increasing the amount of information able to be perceived in a period of time • Great possibility of moving to flight simulation if funding found University of California at Santa Cruz

  4. Purpose of scientific visualization • Accurately transmit data from computer to user • Enhance user’s ability to understand presented data • Develop mental image of the data University of California at Santa Cruz

  5. Methods of visualization • Does not specifically require sight • Graphical - sight • Color • Texture • Shape • Sonification - sound • Decibel level • Different instruments • Haptics - touch • Tactile • Kinesthetic University of California at Santa Cruz

  6. Why Multi-modal methods? • "Current graphics and visualization technology cannot cope with the volume and complexity of the data produced by the simulations that will be carried out on the high-end computer platforms in the next two to four years". • They say that new techniques that merge visual, sonic, and haptic data representations need to be developed. Department of Energy University of California at Santa Cruz

  7. Multi-modal benefits • Possible to display more data at the same time • Possible to display data in a more intuitive fashion • Earthquake model • locations of the earthquakes • Graphically represented • distance traveled • Graphically represented • depth of the earthquake • Sonification using musical notes • magnitude of the quake • Haptic vibration University of California at Santa Cruz

  8. Multi-modal benefits(cont) • Cross-modal stimulation can lower detection threshold • Tactile stimuli can reinforce and/or clarify stimuli from other modalities • More natural mappings • Vision and audition usually dominate haptics • Haptics dominate with texture or surface judgments University of California at Santa Cruz

  9. My previous and present visualization research • ProtAlign • Protein structure prediction • Graphical visualization • Sonification • Passive Haptics Unit University of California at Santa Cruz

  10. Predicting Protein Structure • Why predict the structure? • Protein crystallization is difficult and expensive • Why use alignments to predict structures • Proteins with similar amino acid sequences will likely have same structure and function • Huge databases of protein sequences exist • Why Study Structure • Gain Insight into protein functions • Guide biological experiments • Discover the genetic basis of disease University of California at Santa Cruz

  11. How is protein made • DNA is comprised of Nucleic Acids • Codons are groups of three nucleic acids which code for amino acids University of California at Santa Cruz

  12. Protein Biology • Protein • Polypeptide chain made up of Individual Amino Acids • 20 Different Amino Acids Distinguished by the R group University of California at Santa Cruz

  13. Protein Structure • Hierarchical Structure • Goal understand function of protein from primary structure • Sequence of protein is relatively easy to obtain University of California at Santa Cruz

  14. Assessing an alignment • Heuristics for assessing prediction • Amino Acid similarity • four basic types of amino acids • Hydrophobic,Charged Acidic, Charged Basic, Polar (hydrophilic) • Size of amino acid • Core conservation • Want the core of the protein to be most similar • Environmental preferences University of California at Santa Cruz

  15. Two Dimensional Alignment Evaluation • Belvu • Possible to Edit 2-D alignment and see results • Color coded to assess positions along the alignment • Must remember protein 1 letter codes • Only one color mapping possible at one time • User must memorize properties of amino acids University of California at Santa Cruz

  16. Why visualize the prediction? • Some heuristics lack numerical methods • Combining heuristics is difficult • Sanity check University of California at Santa Cruz

  17. Other tools for evaluating structure predictions • RasMol • SAE • DINAMO • CINEMA • SwissPDB • SwissMODEL University of California at Santa Cruz

  18. ProtAlign introduced • Combines features of previous tools • Adds new features University of California at Santa Cruz

  19. ProtAlign allows quick assessment • Ribbon mode scoring allows quick assessment of overall alignment Good Bad University of California at Santa Cruz

  20. ProtAlign Main Structure Visualization Modes • Backbone • Ribbon • Strand • Streamline / rungs • Shows alignment mismatches • -Uses program MeasureShift University of California at Santa Cruz

  21. ProtAlign Main Structure Visualization Modes(cont) • Cartoon • Much like ribbon • Shows secondary structure • Helices • Loop structure • Beta strands • Invisible • Allows focus on specific areas • Removes extraneous information University of California at Santa Cruz

  22. ProtAlign Alignment tools • Low resolution glyphs • Edit alignment in three dimensions • Updates two and three dimensional display • Multi-modal visualization • Environmental mapping • Amino acid environmental preferences University of California at Santa Cruz

  23. Glyph Representation • Amino acid size • Amino acid type • Hydrophobic • Cylinder • Polar/Hydrophilic • Square • Charged Basic • Pyramid • Charged Acidic • Cone University of California at Santa Cruz

  24. Alignment Representation • Evaluate positions along alignment • Color • Rainbow analogy • Red, yellow, green, blue • Shape • Size University of California at Santa Cruz

  25. Evaluating a position along the alignment • Rasmol • Histimine is polar • Phenylalanine is hydrophobic • Easy to confuse as good substitution • ProtAlign • Cylinder and square peg indicates the types differ • Red indicates this is very unlikely to substituted in nature A = histimine B=Phenylalanine Bottom = histimine Top = phenylalanine University of California at Santa Cruz

  26. Possible to edit via 3d interface • Edit alignment via 3d interface University of California at Santa Cruz

  27. ProtAlign Scoring metrics • Blosum 62 matrix • Likelihood that the substitution occurs naturally in nature • Environmental • Calculate environmental probability • Exposure preferences • Buried • Partially buried • Exposed • Preferences for neighboring amino acids • Structural preferences • Coil • Strand • helix • Exposure • None University of California at Santa Cruz

  28. Environmental Sonification added to ProtAlign • Environments • Used to evaluate amino acid exposure preferences • Buried • Tend to like protein core • Partially exposed • Tend to like protein area between core and loops • Exposed • Tend to like loop regions University of California at Santa Cruz

  29. Sonification of ProtAlign • Beachhouse in Santa Cruz • Buried • Wind • In the house near the beach • Wind rattles the windows • Partially exposed • Waves • Open door walk toward beach • Hear the waves crash in the distance • Exposed • Seagulls • Next to the shore • The seagulls greet you University of California at Santa Cruz

  30. ProtAlign Usability testing • Color mapping tested • Environments used to color amino acids • Possible color pairings tested • Blue/red – excellent/bad • Blue/yellow – excellent/poor • Blue/green – excellent/good • Green/yellow – good/poor • Users asked if they thought if substitution was • Much worse • A little worse • A little better • Much better University of California at Santa Cruz

  31. ProtAlign Usability testing • Sonification mapping tested • Exposure • Buried • wind • Partially Buried • waves • Exposed • seagulls University of California at Santa Cruz

  32. ProtAlign Visual Testing • Excellent/bad pairs recognized perfectly and quickly • Excellent/poor pairs moderate accuracy • Good/poor pairs recognized with surprisingly good accuracy Blue/Green was skewed by single user allowing 45 seconds to elapse while answering University of California at Santa Cruz

  33. ProtAlign Aural Testing • Exposed amino acids • More quickly recognized • Accurately recognized University of California at Santa Cruz

  34. ProtAlign Aural Testing (cont) • Partially exposed amino acids • Slowest recognition • Moderate accuracy University of California at Santa Cruz

  35. ProtAlign Aural Testing (cont) • Buried amino acids • Slow recognition • Moderate accuracy University of California at Santa Cruz

  36. ProtAlign Visual Testing • Visual accuracy in multi test is fairly close to single testing University of California at Santa Cruz

  37. ProtAlign Aural Testing • Time for exposed consistently faster University of California at Santa Cruz

  38. ProtAlign overall results • Faster to present all information at once • Time to evaluate a position for environment and for exposure when presented with all information less than time to present information individually • Both environmental and exposure info mapped to color University of California at Santa Cruz

  39. Passive Haptic Unit • No hands required • Currently controls motor filled chair pad • ACCESS RDAG-128H • Serial communication • Eight DACs • Digital to Analog converters • Control variable voltage items • Motors • Fans • Heating elements University of California at Santa Cruz

  40. How haptic unit is controlled • Functions to make voltage changes to single DACS using easy function calls. • RunDAC(theDAC, voltage_level) • Pulse (theDAC, voltage_level) • Causes DAC to turn on to voltage_level than turns it off • Wave (theDAC, increments, level) • DAC starts at 0 and increases voltage until levels in the number of increments • Wave(0, 2, FULL_POWER) • causes the motor to start at power 0, go to half power, then to full power, than half power, ending with off. University of California at Santa Cruz

  41. Multiple motor control • Possible to turn motors off and on in any sequence University of California at Santa Cruz

  42. State of the Art Immersive Multi-modal Visualization • CAVE (University of Illinois) Room • SGI • 4 projectors (3 walls and floor) • Stereographic LCD glasses • Head Mounted Display (HMD) • Wearables • Belt-pack PC • HMD • Wireless communications hardware • Input device • Touch pad • Chording keyboard University of California at Santa Cruz

  43. State of the Art Non-Immersive Multi-modal Visualization • PHANToM • force feedback device requires at least one finger to operate • tends to have low degrees of freedom • Fan applied to hand by Ogi et al • Information given using fans on the hand • Unable to use hand for anything else • Air jets • Low bandwidth • Jets must be spaced to sense seperately • Tacticon 1600 • electrodes on user’s fingers provide electrical pulses University of California at Santa Cruz

  44. Past “Normal” Active Haptic Units • Require hands be used for output • Haptic mice and haptic joysticks • Require user to maintain contact • Input pens • Must maintain contact University of California at Santa Cruz

  45. Proposed Research • Multi-modal visualization • Concentration on tactile haptics • Use of haptics • Showing independent information • More information at the same time • Showing redundant information • Information more quickly understood • Information better understood • User testing • Did the user understand the mappings? • Did the haptics help understanding? University of California at Santa Cruz

  46. Proposed Research (cond) • Passive haptics • Sit back and absorb the information University of California at Santa Cruz

  47. Perceptual Bandwidth University of California at Santa Cruz

  48. Skin Layers • Dermal papillae • Interdigitations of epidermis and dermis • Fingertips • Soles of feet • Dermis • Papillary layer • Nerve endings • Reticular layer • Bulk of dermis • nerves University of California at Santa Cruz [36]

  49. Tactile sensory receptors • Thermoreceptors • Changes in skin temperature • Mechanoreceptors • Pressure • Vibration • Slip • Nocioreceptors • Pain University of California at Santa Cruz

  50. Tactile sensory receptors (cont) • Successiveness limit • 5 ms to perceive as separate • 20 ms to perceive order (not including time for cortex to process the order) • Adaptation • Slowly adapting • Respond throughout stimulus • Joint angle from skin stretch • Rapidly adapting • Respond to start/end of stimulus • Block out extraneous signals • wearing gloves would be constant stimulus which is adapted to University of California at Santa Cruz

More Related