1 / 14

ANF Detection Network

ANF Detection Network. Dave Foster dfoster@asascience.com Guy de Wardener gdewardener@asascience.com. Resources. Confluence page: https://confluence.oceanobservatories.org/display/CIDev/ANF-SQLstream+Integration ANF Detection Network code: https://github.com/daf/anf-detection-network.

sheera
Download Presentation

ANF Detection Network

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ANF Detection Network Dave Foster dfoster@asascience.com Guy de Wardenergdewardener@asascience.com

  2. Resources • Confluence page: https://confluence.oceanobservatories.org/display/CIDev/ANF-SQLstream+Integration • ANF Detection Network code: https://github.com/daf/anf-detection-network

  3. Introduction / Goals • Using elastic processing (cloud computing) to operate on seismic data • Drive development and test ION (https://confluence.oceanobservatories.org/display/CIDev/Home) • Data from Array Network Facility (http://anf.ucsd.edu/) • Uses SQLstream (http://sqlstream.com/) for processing for real time processing

  4. (con’t) • Continuation of POC/Prototype (Aug 2009, https://confluence.oceanobservatories.org/display/CIDev/SQLstream+Prototype)

  5. Architecture

  6. (con’t) • ANF Application Controller • Part of four op units in a deployment • Listens for instrument announcements, partitions those instruments into AMQP queues, requests SQLstream worker op units be created • ANF Application Agent • Runs on SQLstream worker op unit • Manages SQLstream instances, communicates status with Application Controller

  7. AppAgent • Installs/configures SQLstream on demand from AppController • Allows us to use common base image • More than one SQLstream instance (listening to different queues) can be run on a worker op unit • Downsides: time spent transferring installer from S3, time spent installing SQLstream

  8. SQLstream • Listens to AMQP queue assigned to it by App Controller • Transmits detection events over AMQP topic “anf.detections”

  9. Deployment • Uses CEI developed tools cloudinit.d, epumgmt • cloudinit.d: https://github.com/nimbusproject/cloudinit.d • epumgmt: https://github.com/nimbusproject/epumgmt • Actual startup commands: • cloudinitd boot launch-plans/anf/main.conf -n anf -v -v -f /tmp/anf.log • epumgmt.sh -a find-workers -n anf --wholerun • epumgmt.sh -a logfetch -n anf --wholerun

  10. Demo • Live system • 30 stations • Captured Data • 10 stations, showing events

  11. Metrics Instrument Announce ~ 7:20 startup

  12. (con’t) SQLstream running 4:58 36s 12.5s 44.47s 175.40s (2:55)

  13. Future Goals • Data Preservation • Detections, possibly input events • Service Hardening • Distributed, stateless processes • Replacement for instrument announcement • True load balancing • Need more metrics with live data

  14. (con’t) • Monitoring • Graphical dashboard showing SQLstream op units, queues/bindings, consumption rates

More Related