1 / 28

sMAP 2.0: Integrating and Managing Physical Data code.google /p/ smap -data

sMAP 2.0: Integrating and Managing Physical Data http:// code.google.com /p/ smap -data. Stephen Dawson-Haggerty Computer Science Division , University of California, Berkeley stevedh@eecs.berkeley.edu. sMAP Redesign. Design iteration based on 2 years’ experience Case studies

aldona
Download Presentation

sMAP 2.0: Integrating and Managing Physical Data code.google /p/ smap -data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. sMAP 2.0: Integrating and Managing Physical Datahttp://code.google.com/p/smap-data Stephen Dawson-Haggerty Computer Science Division, University of California, Berkeley stevedh@eecs.berkeley.edu

  2. sMAP Redesign • Design iteration based on 2 years’ experience • Case studies • Building management system integration • Residential monitoring • Retrofit system • Repository integration • Transient instrumentation • Ecosystem • Real-time • Slicing • Analysis Local Winter Retreat 2012

  3. Services Communicating with sMAP Many deployments share common infrastructure 6lowpan networks sMAP sMAP RS-458 bus control web models mgmt sMAP BacNET/IP Archiver RDBMS TSDB Points of decoupling Local Winter Retreat 2012

  4. sMAPPrinciples • Decoupled, multi-site design • Instruments, archive, and front-ends can all cross administrative boundaries • Common representation of Timeseries • Between all parties • Import all data and metadata • Don’t be tied to particular schemas • Allow for slicing and dicing • Fix common pain points once Local Winter Retreat 2012

  5. Why Not an Existing System? • We are built on existing systems • postgresql, django, twisted, bdb • Many existing systems are vertical stovepipes or proprietary • OSISoft, BMSs, BI software • Open source software is still struggling to take timeseries seriously • OpenTSDB, Flume/FlumeBase • PubSubHub, pachube • Be modular and efficient and share code in an open ecosystem • Decoupled architecture allows independent development, optimization Local Winter Retreat 2012

  6. sMAP Source Cases Local Winter Retreat 2012

  7. Case 1: Building Management System • Sutardja-Dai (SDH) Hall • Managed with Siemens Apogee System • Shared serial bus, polled by master • Large number of points, high data volume • Similarly complex data relationships Local Winter Retreat 2012

  8. SDH Apogee System • 1358 control settings • Set points, Relays (lights, pumps, etc), Schedules • 2291 meters/sensors • Power (building, floor, lights, chiller, pumps,etc) • Current, voltage, apparent, real, reactive, peak • Temp (rooms, chilled water, hot water) • Air volume • Alarms, Errors • 2165 control outputs • Dampers, valves, min/max flow, fan speed, PID parameters • 72 other Local Winter Retreat 2012

  9. Case 2: Residential Plug-Loads Dense plug-load metering using wireless mesh networks Goal: understand the use patterns of appliances • Deployment methodology for inventory and stratified sampling of loads • Takeaways: unreliability of home internet, NATed operation (With LBNL MELS group) Local Winter Retreat 2012 Edge Router

  10. Network Complexity & Reachability Local Winter Retreat 2012

  11. (Re)design Takeaways • Integrating heterogeneous communication substrates is [still] a must • Was a goal of sMAP 1 • Better support is possible • Intermittent connectivity is unavoidable • Don’t lose data when not necessary • Design for reliability • Existing repositories are unavoidable • Efficient import/export of data (same as a trickle load) • Well known requirement of existing BI/decision support products • Accept metadata from existing systems when available Local Winter Retreat 2012

  12. / # list resource under URI root [GET] /data # list sense points under resource data [GET] / [sense_point] # select a sense point [GET] /meter # meters provide this service [GET] / [channel] # a particular channel [GET] /reading # meter reading [GET] /format # calibration and units [GET/POST] /parameter # sampling parameter [GET/POST] /profile # history of readings [GET] sMAP 1 Overview • Represent units, sampling parameters of the underlying device, for each channel • Expose over HTTP Local Winter Retreat 2012

  13. Change 1: Represent Timeseries • Objects are Timeseries and Collections • Timeseries are durably identified • A collection of structured key-value pairs, data • Data are (time, value[, seqno]) vectors • Collections group timeseries with shared metadata { "/" : { "Contents" : ["sensor0"], “Metadata” : { “SourceName” : “Example sMAP Source” }, }, "/sensor0" : { "Contents” :["channel1"] }, "/sensor0/channel0" : { "uuid" : "a7f63910-ddc6-11e0-8ab9-13c4da852bbc", "Readings" : [ [1315890624000, 12.5 ] ] } } collections timeseires Local Winter Retreat 2012

  14. Change 2: Simple Metadata • Each stream can be “tagged” with key-value pairs • Easy to tag related collections of points • Natural way to represent “subset of” relation Extra/DentElement: elt-A Extra/Circuit: 1 Extra/System: lighting Extra/ServiceDetail: lighting floors 0-4 Properties/ReadingType: double Properties/Timezone: America/Los_Angeles Properties/UnitofMeasure: kW Instrument/Manufacturer: Dent Industries Instrument/Model: PowerScout 18 Instrument/SamplingPeriod: 20 Location/Campus: UCB Location/Building: Cory Hall Extra/Phase: ABC /true_power /elt-A /ABC / gpe Local Winter Retreat 2012

  15. OSS sMAP Source Implementation • Implementation of new design • Focus on ease of writing new source • Reliable delivery with local buffering • Multiple configuration options for adding metadata, setting up drivers • Growing driver library • Documentation, tutorials, etc. • Focus on minimum effort to add new source Temperature/ PAR/TSR sMAP Dent circuit meter sMAP http://code.google.com/p/smap-data/ Local Winter Retreat 2012

  16. Example Teaser caiso.ini [/] uuid= dd6a4ae4-b17c-11e0-9d53-0026bb56ec92 Metadata/SourceName = CA ISO price and consumption feed [/demand] type = smap.drivers.caiso.CaIsoDriver [/oakland] type = smap.drivers.caiso_price.CaIsoPrice # this is passed to the driver Location = OAKLAND_1_N001 Local Winter Retreat 2012

  17. Infrastructure Local Winter Retreat 2012

  18. sMAP Ecosystem Architecture Local Winter Retreat 2012

  19. Services Communicating with sMAP Many deployments share common infrastructure 6lowpan networks sMAP sMAP RS-458 bus control web models mgmt sMAP BacNET/IP Archiver RDBMS TSDB Points of decoupling Local Winter Retreat 2012

  20. Archiver • Application server functionality on top of back-end databases • Add data and metadata from sMAP sources • Query stream data and metadata • Enforce security policy /add/KEY # add new data to system /republish # data firehose /api/query # look up timeseries by tags /api/[data,next,prev] # retrieve timeseries data /api/tags # return tagsets Local Winter Retreat 2012

  21. High-performance Storage Service • Low-latency, high speed access to archived readings • 5 billion readings, 3.5 bytes/reading on disk Time-series Interface Bucketing Compression Storage mapper RPC readingdb Key-Value Store SQL Page Cache Lock Manager Storage Alloc. MySQL Local Winter Retreat 2012

  22. Archiver Query Language • Column-like store with SQL-like interface select distinct Metadata/SourceName Tag names look like columns Archiver Query Language Query Execute on postgres Transformation to set operations Security check addition Local Winter Retreat 2012

  23. Archiver Query Language Examples query > select distinct Metadata/SourceName SELECT DISTINCT m.tagval FROM metadata2 m, stream s, subscription sub WHERE s.id = m.stream_id AND s.subscription_id = sub.id AND tagname = 'Metadata/SourceName' AND (sub.public OR sub.key = 'jNiUiSNvb2A4ZCWrbqJMcMCblvcwosStiV71’) query > select distinct Metadata/SourceName where Metadata/Instrument/Manufacturer ~ '^Dent' SELECT DISTINCT m.tagval FROM metadata2 m, stream s WHERE stream_id IN ( (SELECT stream_id AS cnt FROM metadata2 mi, subscription sub, stream si WHERE ((tagname = 'Metadata/Instrument/Manufacturer' AND tagval~ E'^Dent')) AND (sub.public OR sub.key= 'jNiUiSNvb2A4ZCWrbqJMcMCblvcwosStiV71') AND mi.stream_id= si.id AND si.subscription_id = sub.id)) AND tagname= 'Metadata/SourceName' AND s.id= m.stream_id Local Winter Retreat 2012

  24. powerdb2: Plotting and Exploration Local Winter Retreat 2012

  25. slicr: Tags Generate Multiple Views [ { tag : "Metadata/SourceName", restrict: "has Metadata/Extra/EndUse"}, { tag: "Metadata/Extra/EndUse"}, { tag: "Metadata/Extra/Category", defaultSubStream: "Properties/UnitofMeasure = 'mW'", seriesLabel:["Metadata/Location/Room", "Metadata/Extra/Load"]}, { tag: "Metadata/Extra/ProductType", defaultSubStream: "Properties/UnitofMeasure = 'mW'", seriesLabel:["Metadata/Location/Room", "Metadata/Extra/Load"]}, { tag: "Metadata/Instrument/PartNumber", defaultSubStream: "Properties/UnitofMeasure = 'mW'", seriesLabel:["Metadata/Instrument/PartNumber", "Metadata/Location/Room", "Metadata/Extra/Load”]}, "Properties/UnitofMeasure” ] Local Winter Retreat 2012

  26. Status • Nearly all sMAP1 sources moved • Dent + Verismodbuselectric • Siemens + Johnson BACnet • pQube, campus obviuselectric + steam • Vaisala weather, wunderground, NOAA, NWS • Steam, condensate • BPA Hydro, CAISO price + consumption • ACme, Dust, Labjack, TED • Sweet new sources • Real-time resampling, campus baselining • Applications, too: • Data closet PUE • Lighting • Dashboards galore • Lots of undergrad involvement, projects in the works • Entire system released: readingdb, smap-data, powerdb2 Local Winter Retreat 2012

  27. Adoption & The Future • CBE site • Shared backend • Store and organize BMS data from study participants • Mix of some realtime with mostly loaded data • LBNL site • Full separate install • CIEE Energy and Climate Research Innovation Seed Fund Grant • Energy data application contest (thanks, Quanta!) Local Winter Retreat 2012

  28. http://new.openbms.org http://code.google.com/p/smap-data/ DEMO & Questions Local Winter Retreat 2012

More Related