480 likes | 565 Views
Distributed Computing with DAFFIE. Glenn Bresnahan Boston University Scientific Computing and Visualization (http://scv.bu.edu/DAFFIE/Earlab-Oct02). Agenda. DAFFIE overview DAFFIE live example DAFFIE distributed computing components Earlab “leaky” example
E N D
Distributed Computing with DAFFIE Glenn Bresnahan Boston University Scientific Computing and Visualization (http://scv.bu.edu/DAFFIE/Earlab-Oct02)
Agenda • DAFFIE overview • DAFFIE live example • DAFFIE distributed computing components • Earlab “leaky” example • Advanced display demonstrations (CGL) Scientific Computing and Visualization
What is DAFFIE? DistributedApplicationsFrameworkForImmersiveEnvironments System for easily creating distributed and shared virtual environments Authored by Glenn Bresnahan, Erik Brisson, Robert Putnam Scientific Computing and Visualization
Background & Motivation • Distributed/remote visualization (~1992) • CM to networked SGI workstations • IWay (1995) • NSF (vBNS) Connections Program (1996) • Alliance computing and visualization grid (NSF PACI) (1997) • ImmersaDesk • Alliance ‘98 Scientific Computing and Visualization
Distributed Application Issues • Analysis of application communication requirements • Sensitivity to bandwidth • Sensitivity to latency • Testing and debugging • Performance measurement and analysis • Robustness • Interoperability of software tools • Management and scheduling of resources in multiple administrative domains • Hardware bottlenecks • Scalability of applications • Security and privacy Scientific Computing and Visualization
The VR Experience - Immersion • Real-time, realistic 3D graphics • Stereopsis • Wide (complete) field of view • Viewer based point-of-view • Navigability • Natural interaction modes • Realistic audio Scientific Computing and Visualization
Distributed VR • Multi-person, multi-site shared virtual space • Navigation • Avatar representation • Telephony Scientific Computing and Visualization
DAFFIE Distributed VR Model • Objective • State consistency • Minimize and/or hide latency • Persistence • Equipotent clients • Network visible state • Message (event) based • Message classes • Delivery service classes • Globally sequenced, reliable delivery • Message class subscription Scientific Computing and Visualization
DAFFIE Components • Messaging system (events) • VR (upper layer) protocol • Client applications (agents) FOR MORE INFO... http://scv.bu.edu/DAFFIE Scientific Computing and Visualization
DAFFIE Application Clients • Viewer • Sound server • Telephony generator • Sound player • Behavior agents • Audio/video generators/processors • Object generators/manipulators Scientific Computing and Visualization
DAFFIE Environment telephony behavior agent viewer behavior agent sound player event server behavior agent sound player telephony viewer Scientific Computing and Visualization
DAFFIE Viewer • SGI Performer based • Mono or stereo display • Workstation, CAVE/Immersadesk, HMD • Interactive and server (video) modes • Six-degree of freedom head-tracking • Six-degree of freedom navigation • Texture-mapped streaming video Scientific Computing and Visualization
DAFFIE Viewer • Simple "script-based" world description • Geometric models • Animation • Sounds • Many 3D model formats • Interaction by picking and/or proximity • Local control of time-sensitive animation and sound • Global update of avatar motion and object state Scientific Computing and Visualization
DAFFIE Sound Server • Telephony generator • Digitize microphone input • Broadcasts "packetized" audio stream • Echo suppression Scientific Computing and Visualization
DAFFIE Sound Server • Sound player input • Multiple sound sources • Real-time audio streams (e.g. telephony) • Prerecorded object sound files • Audio attachment objects (3D location) • Avatar position in virtual space • Listener position in physical space • Sound player output • Multi-channel localized audio signals Scientific Computing and Visualization
DAFFIE Agents • Control object behavior • Computational modules • Proximity triggers • Portals • Device interfaces • Kinematic sculptures, robotics • Audio and video sourcing (e.g. camera) • Audio and video processing Scientific Computing and Visualization
DAFFIE Live Example Telephony gen. Telephony generator Sound player Sound player Navigation client event server Renderer (viewer) Render server Proximity detector Video broadcaster Starfish Video director Chair Video player Skater Butterfly Scientific Computing and Visualization
DAFFIE Messaging System • Application protocol • Client services • Client application library (API) • Message router (“event server”) Scientific Computing and Visualization
Message Routing • Message router (“event server”) • Multiple delivery service classes • Globally sequenced, reliable delivery • Client management • Message subscription • Multiprocessor, high bandwidth • Hierarchical servers • Proxy routers • Fan-in/fan-out • Tunneling • Direct routing for P2P Scientific Computing and Visualization
DAFFIE Message routing client client client proxy client client event server client client proxy client client Scientific Computing and Visualization
Message Transports • TCP/IP • Single channel or striped • Shared memory • Myrinet • IP Multicast Scientific Computing and Visualization
Application Architecture App thread App thread App thread … App thread Services (timing, streaming, file sharing) Event Messaging Message Transport Scientific Computing and Visualization
DAFFIE VR Protocol • Compound objects • Geometry • Transformation • Audio hook • Video texture map • Component ownership/object locking • Request ownership • Broadcast changes • Release ownership • Ownership transfer • Real and effective ownership Scientific Computing and Visualization
Defining Upper Layer Protocol • Components • Event type ID (#define in a .h) • Data layout (compile-time) (C struct in a .h) • Data layout (run-time) (event_define) • Needed for canonical byte ordering • Types statically (compile time) defined • Protocol compiler Scientific Computing and Visualization
Messaging API • Multithreaded, non-blocking • Dynamic join and leave • Point-to-point and broadcast messages • Polled or callback (active messages) • Subscription based receive • Low level services • Registration, timing, synchronization, query Scientific Computing and Visualization
Service Discovery • Client registration/identification • Class (Viewer, Audio, Video, Agent) • Species (video: source, broadcaster, sink, processor) • Instance (video: stream-name) • ID • Hostname • Client location services • Service request/acknowledgement Scientific Computing and Visualization
Client Services • Streaming • Data chunking • Bandwidth management • Dynamic tuning • File sharing • File servers • Conditional upload/download Scientific Computing and Visualization
Job Initiation & Monitoring • Rely on external authentication and job initiation (e.g. ssh, Globus) • Launch scripts (ad hoc) • cgl-video; visible.bu.edu:vgrab -compress -process; -:vswirl -steps 9999999 -np 4; -:vbroadcast; .:vdirect -object poolanim_grp ; • Agent initiation daemon • Logging daemon Scientific Computing and Visualization
Client Monitoring • Dynamic clients • JOIN, EXIT, IDENTIFICATION events • Event server monitoring • Heart beat • Latency monitor • Congestion control • Statistic generation • Client monitoring • Process level monitoring • Congestion management • Remote process termination Scientific Computing and Visualization
Leaky Example: Headers typedef struct { EVENT_HEAD head; char filename[200]; } EVENT_GETPARAM; typedef struct { EVENT_HEAD head; int logger, frame_size, sample_rate); } EVENT_START; typedef struct { EVENT_HEAD head; char msg[1000]; } EVENT_LOGINFO; Scientific Computing and Visualization
Leaky Example: Headers typedef struct { EVENT_HEAD head; int client; int size; float pressure[1]; } EVENT_ADVANCE; Scientific Computing and Visualization
Leaky Example: Headers #define ET_START 1001 #define ET_GETPARAM 1002 #define ET_LOGINFO 1003 #define ET_ADVANCE 1004 #define ET_STOP 1005 event_define(ET_START, “start”, 1, 3, EVENT_DATA_INT); event_define(ET_GETPARAM, “getparam”, 1, 200, EVENT_DATA_CHAR); event_define(ET_ADVANCE, “advance”, 3, 1, EVENT_DATA_INT, 1, EVENT_DATA_INT, 0, EVENT_DATA_FLOAT); Scientific Computing and Visualization
Leaky Example: Initialization id = event_join(“earlab.bu.edu”, &nc); printf(“Joined as client id %d of %d\n”, id, nc); event_register(“integrator”, “leaky”, “1.0”); event_receive_type(ET_GETPARAM, ET_START, ET_ADVANCE, ET_STOP); event_callback(ET_START, do_start); event_callback(ET_GETPARAM, do_getparam); … Scientific Computing and Visualization
Leaky Example: File Service GotParamFile = 0; efs_init(); efs_chdir_receive(“/tmp/earlab”); Scientific Computing and Visualization
Leaky Example: GetParameters do_getparams(EVENT_GETPARM *egp) { efs_download_cond(egp->filename, egp->filename, xfer_done, 0); return(1); } xfer_done(int reqid, int fsid, inr rc, char *file, int tag) { char path[1000]; strcpy(path, “/tmp/earlab/”); strcat(path, file); GotParamFile = 1; Leaky_GetParameters(path); ); Scientific Computing and Visualization
Leaky Example: Start do_start(EVENT_START *est) { Logger = est->logger; while (!GotParamFile) event_sleep(100); Leaky_Start(egp->frame_size, egp->sample_rate); return(1); } Scientific Computing and Visualization
Leaky Example: LogInfo void LogInfo(char *msg) { EVENT_LOGINFO einfo; strcpy(einfo->msg, msg); event_send(Logger, ET_LOGINFO, &einfo, sizeof(einfo)); } Scientific Computing and Visualization
Leaky Example: Advance do_advance(EVENT_ADVANCE *ein) { int bytes; EVENT_ADVANCE *eout; bytes = sizeof(EVENT_ADVANCE) + (ein->size-1)*sizeof(float); eout = (EVENT_ADVANCE *) malloc(bytes); Leaky_Advance(&ein->pressure, &eout->pressure); eout->client = event_clientid(); eout->size = ein->size; event_send(ein->client, ET_ADVANCE, eout, bytes); free(ein); return(1); } Scientific Computing and Visualization
Demonstrations • Computer Graphics Lab (Room 203) • ImmersaDesk • Display Wall Scientific Computing and Visualization
VR/AG Bridging • Interconnect VR and AG spaces • VR image display over AG • Bridge audio in both directions • Provide interactivity for AG participants • AG navigation • AG interaction tools • Couple external a/v to both AG and VR Scientific Computing and Visualization
VR/AG Bridging Boston Cyberarts Festival SC Global FOR MORE INFO... http://scv.bu.edu/hipart/GoGrid/ Scientific Computing and Visualization
VR/AG Bridging AG node view renderer video bridge navigation Multicast DAFFIE audio renderer AG audio AG node viewer Scientific Computing and Visualization
Tiled Display Wall • Benefits • Low cost • Scalable • Multi-use • Passive stereo • Construction • Linux cluster display driver • Inexpensive LCD projectors • Cheap polarizing filters Scientific Computing and Visualization
Tiled Display Wall • One Linux node for every 1 or 2 projectors • One additional “control” or “head” node • Two projectors per tile • Linear polarizing filters • Polarization-preserving projection screen • Custom fabricated projector mounts • 6 DoF adjusters Scientific Computing and Visualization
Tiled Display Wall • Prototype configuration • 2x2 stereo array (8 projectors) • 4 Linux display nodes + one control node • Fast Ethernet interconnect • Final configuration • 4x3 stereo array (2 x 9.4 Mpixel) • 24 Linux display nodes + one control node • 52 Linux compute nodes • Myrinet interconnect Scientific Computing and Visualization
Tiled Display Wall • Software • WireGL/Chromium • VTK • Performer + local synchronization • Movie player (several) Scientific Computing and Visualization
Tiled Display Wall • Issues • Software • Alignment • Custom projector mounts • Computerized motor control • Imaging transformation warps • Autocalibration • Edge blending • Opto-mechanical • Software Scientific Computing and Visualization
Access Grid Overview • Suite of commodity hardware and software • Open source, Open protocol • Multicast • Tiled projection wall • Approximately 90 installations world-wide (7 countries) FOR MORE INFO... http://www-fp.mcs.anl.gov/fl/accessgrid/ http://scv.bu.edu/accessgrid Scientific Computing and Visualization