Online Piece-wise Linear Approximation of Numerical Streams with Precision Guarantees

1. Online Piece-wise Linear Approximation of Numerical Streams with Precision Guarantees Hazem Elmeleegy Purdue University Ahmed Elmagarmid (Purdue) Emmanuel Cecchet (UMass) Walid Aref (Purdue) Willy Zwaenepoel (EPFL)

2. Outline • Introduction • Swing & Slide Filters • Experiments • Conclusion

3. Application Scenario • Some Common Applications: • Cluster Monitoring • Sensor Networks • Stock Market Transmitter Receiver

4. The Problem • Goal • Minimize amount of transmitted data • Saves bandwidth • Saves storage (at the receiver side) • Saves battery life (esp. for sensor networks) • Using piece-wise linear approximation • Assumptions • Receiver can tolerate: • A bounded error for each data point received (max error = e) • A maximum lag behind the transmitter • Terminology • We refer to any algorithm to solve this problem as a filtering technique, or simply a filter

5. 9e 8e 7e 6e Value 5e 4e 3e 2e e t1 t2 t3 t4 t5 Time Existing Techniques • Cache Filter • The transmitter caches the last transmitted value. • A new value is transmitted only if it is more than e away from the cached value. • Piece-wise constant approximation x3 x2 x4 x1 x5

6. 9e 8e 7e 6e Value 5e 4e 3e 2e e t1 t2 t3 t4 t5 Time Existing Techniques • Cache Filter • The transmitter caches the last transmitted value. • A new value is transmitted only if it is more than e away from the cached value. • Piece-wise constant approximation x3 x2 x4 x1 x5

7. 9e 8e 7e 6e Value 5e 4e 3e 2e e t1 t2 t3 t4 t5 Time Existing Techniques • Linear Filter • The transmitter maintains a line segment that can approximate the last observed data points. • The line segment is updated only when a new data point falls more than e away from the maintained line. x3 x2 x4 x1 x5

8. Outline • Introduction • Swing & Slide Filters • Experiments • Conclusion

9. Swing and Slide Filters • Key Idea • Maintain a set of candidate line segments at any given time • Postpone the selection decision as late as possible to accommodate more points

10. 9e 8e 7e 6e Value 5e 4e 3e 2e e t1 t2 t3 t4 t5 Time Swing Filter • Connected line segments • Complexity • Maintains upper and lower segments only • O(1) space and time complexity • Lag • If max lag is exceeded, switch to linear filter • Correctness • Proof of correctness in the paper x3 x2 x4 x1 x5

11. 9e 8e 7e 6e Value 5e 4e 3e 2e e t1 t2 t3 t4 t5 Time Slide Filter x3 x2 x1

12. 9e 8e 7e 6e Value 5e 4e 3e 2e e t1 t2 t3 t4 t5 Time Slide Filter x3 x2 x4 x1

13. 9e 8e 7e 6e Value 5e 4e 3e 2e e t1 t2 t3 t4 t5 Time Slide Filter • Optimization #1 • Connect line segments whenever possible • Optimization #2 • Do not maintain all the data points currently being approximated • Maintain their convex hull only • Complexity • O(h) space and time complexity • h is the number of data points on the convex hull --- very small in practice • Lag • If max lag is exceeded, switch to linear filter • Correctness • Proof of correctness in the paper x3 x2 x4 x1 x5

14. Outline • Introduction • Swing & Slide Filters • Experiments • Conclusion

15. Compression Ratios for the Sea Temperature Signal Sea Surface Temperature e

16. Effect of Signal Behavior (Degree of Monotonicity) Synthetic Signal: Random walk with probability p to increase and (1-p) to decrease.

17. Overhead for the Sea Temperature Signal e

18. Outline • Introduction • Swing & Slide Filters • Experiments • Conclusion

19. Conclusion • We introduced two new filtering techniques: the swing and slide filters • They have significantly higher compression ratios compared to earlier techniques, especially the slide filter • They have a small overhead, and hence are suitable for overhead-sensitive applications

20. Thank you • Questions?