1 / 20

Case Study: Debugging Multicast Problems from an Applications Perspective

Case Study: Debugging Multicast Problems from an Applications Perspective. Steven Senger, Ph.D. Dept. of Computer Science University of Wisconsin - La Crosse. HAVnet Project. Parvati Dev, PI, Stanford SUMMIT National Library of Medicine, NGI & SII programs since 1999.

louvain
Download Presentation

Case Study: Debugging Multicast Problems from an Applications Perspective

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Case Study: Debugging Multicast Problems from an Applications Perspective Steven Senger, Ph.D. Dept. of Computer Science University of Wisconsin - La Crosse

  2. HAVnet Project • Parvati Dev, PI, Stanford SUMMIT • National Library of Medicine, NGI & SII programs since 1999. • Applications of high-performance networks to anatomical and surgical education. • http://havnet.stanford.edu • http://visu.uwlax.edu

  3. Immersive Segmentation

  4. Remote Stereo Viewer

  5. Nomadic Anatomy Viewer

  6. Other Apps and Components • Information Channels • Multicast based announcement/discovery mechanism. • Supports other app requirements such as logging. • Access Grid

  7. Testbed

  8. Network/App Monitoring

  9. Potholes Along the Way • Stanford / CENIC • Multicast setup delay • WiscNet • Conflict between sender and receiver • Michigan / Merit • Multicast setup delay • Inbound flow stops after 209 secs

  10. Stanford / CENIC … • Longstanding problem (observed in ‘01). • Large delays (~15 min) in multicast setup. • Stanford / La Crosse / NLM • Significant delays except for La Crosse / NLM • Originally thought to be at Stanford Border and RP. • 04 hardware/ios upgrades at Stanford. • Situation improved.

  11. Stanford / CENIC … • Only Michigan to Stanford delayed, ~6 mins. • Oct 04, Phone calls, Stanford, CENIC, Vendor support, La Crosse. Escalate through 3 layers of vendor support. • Test/Debug every couple of weeks through March ‘05. • Identified as MSDP propagation delay related to encap/unencap data received by MSDP.

  12. Stanford / CENIC • Delay occurred at each CENIC router. • At some point problem had been internally found and resolved by vendor. • Solution: upgrade OS on CENIC routers.

  13. La Crosse / WiscNet … • First observed spring 05 using AccessGrid. • La Crosse sender and Stanford receiver OK. • Starting a La Crosse receiver breaks the flow. • WiscNet identified problem router. • Vendor support engaged. • Discovered rpd restart sufficient to fix. • Reoccurs every 2 months.

  14. La Crosse / WiscNet … • When failing • Upstream interface on router gets set to unreasonable value. • Sender continues to send data in encapsulated PIM-register messages. • Router never sends register-stop messages.

  15. La Crosse / WiscNet • Problem has survived router chassis upgrade. • No solution as yet.

  16. U. Michigan / Merit … • Discovered after CENIC problem solved. • Small delay in setup for Michigan to Stanford. • Varies between 0 and 60 sec. • Similar behavior for Milwaukee to Stanford. • Does not appear to be in CENIC?

  17. U. Michigan / Merit … • Presence of other receivers seems to change the setup delay. • Merit engaged in isolating problem. • No solution as yet.

  18. U. Michigan / Merit • Discovered Jan ‘06 using AccessGrid. • Traffic from Stanford to MCBI/Merit starts correctly but stops after 208 seconds. • When stopped IPLSng shows as pruned. • Merit identified problem with a switch in Chicago not allowing streams to setup correctly. • Problem resolved with OS upgrade.

  19. Diagnostic Help • Debugging strategies • Tools • Monitoring

More Related