Roy Williams Andrew Drake, Matthew Graham, Ashish Mahabal California Institute of Technology

1. Skyalert and Event Processing Roy Williams Andrew Drake, Matthew Graham, Ashish Mahabal California Institute of Technology

2. This talk defines these • VOEvent: Int’l standard representation of data for astronomical transient as XML • VOEventNet: A Protocol for pushing VOEvents, and its nodes • Skyalert: Software for managing VOEvents: web, subscriptions, repository

3. VOEvent Key Concepts • IVORN – unique identifier for event • Instance of a Stream that also has IVORN • How is it resolved? • Who • author’s provenance • What • data model designed by author • May have URLs … permanence, calibration? • WhereWhen • targeting in spacetime • Citation • related events • “followup", "supersedes" or "retraction”

4. VOEventLibhttp://lib.skyalert.org/VOEventLib/ • Python interface to VOEvent • Code built automatically from schema • With ‘utility’ package Vutil v = VOEvent.VOEvent(role='test', ivorn='ivo://testing#111') author = Author(contactName=['Donald Duck']) v.set_Who(Who(Author=author)) what = What() what.add_Param(Param(name='apple', value='123')) what.add_Param(Param(name='orange', value='124')) v.set_What(what) s = stringVOEvent(v) print s

5. VOEventNet Key Concepts • Broker • Broadcaster of received events • May know which subscriber wants what • Subscriber • Connects to Broker • Long-term connection • Publisher • Scientifically responsible • Works with a broker • May sign events with PGP

6. VOEventNeta network of independent nodes exchanging VOEventswith the VOEvent Transport Protocol (VTP) CBAT AAVSO CRTS1,2,3 CSS NEO Skyalertrunning Comet LOFAR and 4piSky running Comet NASA events running GCN/TAN DC3.com running Dakota

7. VOEventNet defined http://arxiv.org/abs/1206.0236

8. VOEventNet Patterns subscribe publish translate annotate relay translate registry query repository

9. SkyalertOpen sourced VOEventNet node Translate Push messages Push actions Web interface Web services Publish Query VOEventNet Translate External annotator repository Many types of events can be homogenized and interoperated Supported by US National Science Foundation OCI-0915473

10. SkyalertAnnotation • Annotation can be: • Newer observations or calibration • Archive search for matches • Machine-learning ‘decision’ • Follow-up observation • Comment / Circular Annotation Publish Annotation VOEventNet How can annotations be queued and clouded in case of high event rate?

11. Two lists of events skyalert.org ‘role=observation’ (astrophysics) events only XMPP and email translations betelgeuse.ligo.caltech.edu All ‘events’ from five high-energy missionsincluding ‘utility’ events Received by VTP

12. http://www.skyalert.org/streams/ Skyalert Stream Registry... will be VO registry

13. Skyalert Key Concepts • Stream • Event semantics defined by author • What does Burst_SOD mean in the Fermi stream? • Like a data dictionary, a schema, common metadata • Portfolio • Event + associated data • As collection of VOEvents with the observation first • Query • Define what is ‘interesting’ • For Past or Future events • Language, as XML, SQL tables (flex and fixed), KW dict, triple store??

14. Concept: Streamas Event TemplateEvent author defines parameters and meaning in advance. stream event made byrobot system at night made byperson in daytime

15. SWIFT portfolio Notices from the satellite Circulars from the humans

16. Event DatabasesTechnology and Language Evaluations • Stardog • SELECT ?s WHERE {?s voe:hasParam ?p . ?p voe:hasName "…"^^xsd:string . ?p voe:hasValue ?v . FILTER (?v < …)} • MySQL fixed: with a separate table for each stream • select id from … where … < … • MongoDB (NoSQL): ivorn, param name, value • db.test.find({"param":…, "value":{"$lt":…}}) • MySQL flex: with a params table • select id from params where name = "…" and value < … • eXist native XML • for $event in /voe:VOEvent where $event/What//Param[@name = '...']/@value < "..." return <id>{$event/@ivorn}</id> Speed

17. Skyalert web services from django.utils import simplejson import urllib sk = http://skyalert.org # CRTS and CRTSCircular["First"]["eventClass"] == "Supernova" query = "285" jtableurl = "%s/events/jtable/%s/" % (sk, query) while 1: jsontable = urllib.urlopen(jtableurl).read() pytable = simplejson.loads(jsontable) list = pytable["list"] print "Found %d portfolios" % len(list) for pf in list: ivorn = pf[1] dict = {} dict['ivorn'] = ivorn dict['JSON'] = 'on' params = urllib.urlencode(dict) resolveurl = "%s/resolve/" % sk jsondetail = urllib.urlopen(resolveurl, params).read() pydetail = simplejson.loads(jsondetail) # ivorn of the primary event of the portfolio should be here crtsevent = pydetail["portfolio"][ivorn] mag = crtsevent["FirstDetectionparams"]["magnitude"] tok = ivorn.split('#') localivorn = tok[1] date = crtsevent[""]["ISOtime"] print localivorn, date, mag jtableurl = jsonresult["next"] if jtableurl == "finished": break Magnitudes of CRTS supernovae 1107030071174105491 2011-07-03T08:36:01 18.935400 1107030091124153498 2011-07-03T08:01:30 18.756399 1107031180794128821 2011-07-03T04:39:47 20.169800 1107031070764142675 2011-07-03T04:07:39 18.166100 1107021180094118167 2011-07-02T10:46:06 18.601400 1107021041264109182 2011-07-02T09:31:06 15.879600 1107011150104107901 2011-07-01T10:47:36 18.279699 1107011120104119909 2011-07-01T10:46:15 17.874800 1107010040834139996 2011-07-01T06:17:32 18.483601 1107011040674116917 2011-07-01T04:13:08 19.630899

18. Who are the subscribers? • What size telescope / Data mining / Citizens • Is it the same as the “GCN followup” gang? • How can we set up a clearinghouse of information? • How to build “simulated event stream” for LSST? • So prototypes can be tested • Remember average rate and burst rate • What are annotation services for LSST? • Same as “common queries” • What is the security requirement? • In testing, commissioning, observing • What language for alerts and history queries • Triples, SQL, noSQL, XML, ...