1 / 37

IS4: A Storage System for Organizing Physical Data

IS4: A Storage System for Organizing Physical Data. Jorge Ortiz University of California, Berkeley. Talk overview. Integrated sensor stream storage system (IS4) Recap from last retreat Physical data Representation of: Physical devices Measurements Environments Management challenges

lilka
Download Presentation

IS4: A Storage System for Organizing Physical Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. IS4:A Storage System forOrganizing Physical Data Jorge Ortiz University of California, Berkeley

  2. Talk overview • Integrated sensor stream storage system (IS4) • Recap from last retreat • Physical data • Representation of: • Physical devices • Measurements • Environments • Management challenges • Current features and future work

  3. Multi-scale Energy Network Wind Modeling IS4 Gen-to-Building Gen-to-Grid Facility-to-Building Building-to-Grid Building-to-Grid Facility-to-Building uGrid-to-Grid Facility-to-Building Grid Storage-to-Building Building MR-to-Building Demand Response Machine Room Load Following Temperature Maintenance Supply Following Plug Loads Web Server Instrumentation Models Grid OS Instrumentation Instrumentation Lighting CompressorScheduling Web App Logic Instrumentation Models Models Models DB/Storage Facilities Routing/Control Control Building OS Load Balancer/Scheduler Fridge as a model Thermal storage system Power-AwareCluster Manager Controls

  4. SCADA SystemsSupervisory Control and Data Acquisition + SODA4R787__ART + Data

  5. Problems with SCADA • No systematic organization of data andmetadata • Metadata embedded in UI • Difficult to find context information • Incomplete data and metadata • Mostly 0’s • Missing type descriptions, location information

  6. Capturing building layout Chiller Pump Zone Chiller Vent Vent Pump SF EF AHU

  7. Entity relationships • Building • Systems • Spaces/Zones • Structural relationship • Sensors/Actuators • Inside systems • Inside zones/spaces • Types System Chiller Space Pump Zone Chiller Vent Vent Pump SF EF AHU

  8. Building entity schema

  9. Relevant queries • List temperature sensors on 4th floor of Soda hall • SELECT id FROM zone_sensors WHERE building=‘Soda’ and floor=4 AND id LIKE ‘%ART’ • Get all temperature sensor data in the last day on the 4th floor of Soda • SELECT * FROM sensor_data WHERE id=[list of sensors] AND timestamp >= ’ 2009-12-16 00:00:00’ and timestamp<=now()

  10. System objectives • Physical data storage repository • Append only, forever • Management of data and metadata • Changes occur over time • Physical-data queries • Historical and real time • Security • Secure access to data streams, distillates, metadata

  11. What is physical data? • Data collected from the environment • Why is it different from other kinds of data? • Data is produced by sensing devices • Devices are placed in the physical environment • Placement matters • Deployment changes over time • Devices, configuration, environment Data, metadata relationship-management is challenging

  12. Sensing device metadata • Make • Dent SmartLogger • Model • CTLogger TOU-CT • url • http://www.dentinstruments.com/media/PDF/SMARTware2009_Manual.pdf • Make • ACme • Model • Revision 1 • url • http://smote.cs.berkeley.edu:8000/tracenv/wiki/ACME

  13. Object schema { “type“:"object_stream", “$schema”:”object_stream_schema.json”, “device_name":“dentMeter123", "model":“CTLogger TOU-CT", "desc":“dent meter”, “url”:” http://www.dentinstruments.com/media/PDF/SMARTware2009_Manual.pdf” } instance { "description":"object_stream_schema", "type":"object", "properties": { “type":{"type":"string","optional":false, "options":[{"value":"object_stream"}]}, "device_name":{"type":"string", "optional":"false"}, "model":{"type":"string","optional":false}, "desc":{"type":"string","optional":true}, “url":{"type":"string",”format”:”url”,"optional":true } } schema

  14. Communication information • Network address or URL • Function calls, parameter description • Functional schema or url documentation http://coryacme.cs.berkeley.edu/data/33 2001:252:0:1::2008:6 and 2001:252:0:1::2008:8

  15. Logic schema { “type“:“logic", “$schema”:”logic_stream_schema.json”, “address”:”http://is4/resource”, “function”:[{ “function_name”:”a”, “opertation”:”GET”, “parameters”:[{“field”, “name”}] } } instance { "description":“logic_stream_schema", "type":"object", "properties": { “type":{"type":"string","optional":false, "options":[{"value":“logic"}]}, “$schema”:”logic_stream_schema.json”, “address”:{“type”:”string”, “format”:[“string”, “ipv4”, “ipv6”, “url”], “optional”:”false”}, “functions”:{“type”:”array”, “options”:[{ “function_name”:”{“type”:”string”}, “operation”:{“type”:”string”, “optional”:”true”}, “parameters”:{“type”:”array”, “options”:[”object”, “string”, “$ref”]} } } } schema

  16. Sensor context information Vibration Humidity Temperature Pressure

  17. Electric load tree+ Electrical Load Tree Physical CT mains power monitoring Circuit/breaker panel level power monitoring ACme: plug load energy monitor and controller + Slide courtesy of Xiaofan Jiang Humidity Temperature PAR/TSR Vibration

  18. Context description • General description • Google JSON GeoCoding object • http://code.google.com/apis/maps/documentation/javascript/v2/services.html#Geocoding_Object • IS4 context graph • Schema that expresses contextual objects and relationships as a graph

  19. IS4 Context graph schema Context Node: { "label":{"type":"string", "optional":false}, "name":{"type":"string", "optional":false}, "type":{"type":"string","options":[{"value":"context_node"}], "optional":false}, "description":{"type":"string", "optional":false}, "properties": { "AssociatedDevices":{ "type":"array", "options":[{"value":"string","format":"UUID"}], "optional":true }, "parents":{ "type":"array", "options":[{"value":"string", "properties":{"$ref":"#.properties.cnid"}}], "optional":true }, "children":{ "type":"array", "options":[{"value":"string", "properties":{"$ref":"#.properties.cnid"}}], "optional":true }, "cnid": { "type":[“string"] "description":"local node identifier", "optional":false }}} Context Edge {… "label":{"type":"string", "optional":false, "description":"String with spaces"}, "name":{"type":"string", "optional":false, "description":"String without space"}, … "sourceNode":{"type":[“string"], "description":"local node identifier", "optional":false}, "destinationNode":{"type":[“string"], "description":"local node identifier", "optional":false}, "AssociatedDevices":{ "type":"array", "options":[{"value":"string","format":"UUID"}], "optional":true }} schema

  20. Cory hall load tree Load Tree Image Graphical representation

  21. Context state snapshots Time t1 Time t2

  22. Queries • What floor consumes the most power per month? • Send alert if the air conditioner in room 465 consumes more than Y kW in a 24 hour period • Send alert if independent zones become dependent • What is the most common state of the air/water flow graph?

  23. Expressing physical-data queries • Semantic graph + temporal data • Physical and logical relationships • Data streams associated with graphical elements • Easily express queries across both data types • Efficient storage • Efficient access

  24. IS4 Current Status • Data publishers identified by unique id (and IS4 resource URI). • Publishing process includes object and logic JSON objects as input • Context data managed separately • Publisher may belong to multiple contexts • Data and metadata stream explicitly bound

  25. Current Architecture Publish CT mains power monitoring Subscribe HTTP/REST Subscribe Manager Publish Manager Proxy Manager Storage Context Manager ACme: plug load energy monitor and controller Humidity PAR/TSR Temperature Vibration

  26. IS4 Interface Overview / #root – status information [GET] /is4 # a particular channel [GET] /info # statistical system information [GET] /publish # publishing resource [GET/PUT] /all # list of all publishers [GET] /id # sub-children [GET] /<id> # id of publisher [GET/POST] /name # name alias for this id [GET] /mysubs # list/add to pub subscriptions [GET/PUT/DELETE] /metadata # returns publisher-associate metadata [GET] POST requests supply JSON objects as arguments: POST: http://is4.cs.berkeley.edu/is4/publish/id/123456/mysubs?username=jortiz&pw=8832749823{“SubscribeTo":[13241324,234213,45456745], “UnsubscribeFrom":[45432,567365,21465765,2435786] } Username and pw hash Publisher Id

  27. Ongoing work • Security • Token-based security • Tokens encapsulate access rights to resources • Resources referenced by IS4 URI • Permissions: GET, PUT, POST, DELETE • GET = read • PUT = create • POST = update • DELETE = delete • Query interface • Semantic graph + timeseries

  28. Future Work • Incorporate security model into current implementation • Move the data store into Amazon cloud • 14 Dents, 1 min period = ~15 MB/hour  130 GB/year (uncompressed) • Data processing functionality • Interpolation, extrapolation, other time series processing • JSON streaming query engine

  29. Summary • Current building management systems poorly handle physical data • Physical data is tightly bound metadata and context • IS4 provides schemas to express metadata • IS4 implements mechanisms to manage data/metadata binding

  30. Thank you • More information • http://smote.cs.berkeley.edu:8000/tracenv/wiki/is4 • Jorge Ortiz • jortiz@cs.berkeley.edu • Questions?

  31. Extra Slides

  32. Snapshots of physical information • Time t1 • Time t2

  33. Publishers and Subscribers

  34. Open design issues • Doesn’t PowerMeter already do (some) of this? • They also deal with the data stream; context is not used • How different from pubsubhubbub? • Explicit context management

  35. Interpreting data across contextual changes • All the data is timeseries • Real time and historical querying • User can be notified if context changes have occurred over query-time interval

  36. General notes • Talk about what I presented at the last retreat • Include previous work • Include a summary slide at the end • Update the opening slide (missing sponsors) • Look at Stephen’s slides and figure out how to incorporate them • You have to make connections between the two • Ask Gil for his slides to see how to segway these into that

  37. General notes (2)

More Related