1 / 16

A New Approach to File System Cache Writeback of Application Data

A New Approach to File System Cache Writeback of Application Data. Sorin Faibish – EMC Distinguished Engineer P. Bixby, J. Forecast, P. Armangau and S. Pawar EMC USD Advanced Development. SYSTOR 2010, May 24-26, 2010, Haifa, Israel. Outline. Motivation: changes in servers technology

ivo
Download Presentation

A New Approach to File System Cache Writeback of Application Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. A New Approach to File System Cache Writeback of Application Data Sorin Faibish – EMC Distinguished Engineer P. Bixby, J. Forecast, P. Armangau and S. Pawar EMC USD Advanced Development SYSTOR 2010, May 24-26, 2010, Haifa, Israel

  2. Outline • Motivation: changes in servers technology • Cache writeback problem statement • Monitoring behavior of application data flush • Cache writeback as a closed loop system • Current cache writeback methods are obsolete • I/O “slow down” problem • New algorithms for cache writeback • Simulation results of new algorithms • Experimental results of a real NFS server • Summary and conclusions • Future work and extension to Linux FS

  3. Motivation: changes in servers technology • Large numbers of cores in CPUs – more computing power • Large cheaper memory caches – cached data very large • Very large disk drives – but modest increase in disk throughput • Application data I/O increased much faster – but require constant flush to disk • Cache writeback is used to smooth bursty I/O traffic to disk • Conclusion: cache writeback of large amounts of application data is slower

  4. Cache writeback problem statement • I/O speeds increase forcing caching large amounts of dirty pages at servers to hide disk latency • Large number of clients access servers increasing burstiness of disk I/O and need for cache • Large caches of the FS and servers allow longer retention • Cache writeback flush is based on cache fullness metrics • Flush to disk is done at maximum speed when cache full leaving no room for additional I/Os • As long as cache is full I/Os will have to wait for empty cache pages availability – I/O “stoppage” • Result application performance is lower than disk performance

  5. Monitoring behavior of application data flush • Understanding the problem: • Instrument kernel to measure cache Dirty Pages dynamics • Monitor the behavior of DP in Buffer Cache • Run benchmark multi-client application

  6. Cache writeback as a closed loop system • Application controls the flush using I/O commit based on application cache state • DP in cache are difference between incoming I/O and DP flushed to disk • Goal is to keep difference/error zero • The error loop is closed as application send commits after each I/O • Cache Writeback is controlled by application • Flush to disk based on state of fullness of the Buffer Cache • Cache control mechanism ensure cache availability for new I/Os • DP in cache like water in tank • Water level is controlled by cache manager to prevent overflow • No relation between application I/O arrival and when the I/O is flush to disk • Result in large delays between I/O creation and I/O on disk – open loop • Cache writeback is controlled by algorithm

  7. Current cache writeback methods • Trickle flush of DPs • Flush based on proportion of incoming application I/Os (rate based) • Use low priority to reduce CPU consumption • Background task with low efficiency • Used only to reduce memory pressures • Cannot address high bursts of I/O • Watermark based flush of DPs • Inspired from database and transactional applications • Cache writeback triggered by number/proportion of DP in the cache • There is no prediction of high I/O bursts – disadvantage for multi-clients • Flush is done at maximum disk speed to reduce latency • Close to incoming I/O rate for small caches – flush often • Inefficient for very large caches • Interfere with metadata and read operations

  8. Current cache writeback deficiency • Watermark based flush of DPs is similar a non-linear saturation effect in the cache closed loop • Introduces oscillations in the DP behavior due to the saturation • The oscillation introduces additional I/O latencies to the disk latencies • Creates burstiness to the disk I/O – reduce aggregate performance

  9. I/O “slow down” problem • Application data flush require FS MD updates to same disks • Flush is triggered when high watermark threshold is crossed • Watermark based flushes cannot throttle the I/O speed as it is an ultimate resort before kernel crash on starvation • Additional I/Os are slowed down until the MD is flushed for the new arriving I/Os • Even if NVRAM is used the DP need to be removed from cache to make room for additional I/Os • Application I/Os latency increases until the cache is freed – “slow down” • In worst cases the latency is so high that resemble to a I/O stoppage • If additional burst of I/Os on other new clients there is no room to put I/Os and new I/Os will wait until the watermark goes under low watermark - stoppage

  10. New algorithms for cache writeback • Trying to address deficiency of current cache writeback methods • Inspired from control system and signal processing theory • Use adaptive control and machine learning methods • Utilize better modern HW characteristics • The goals of the solution are: • Reduce the I/O slowdown limited only by maximum disk I/O throughput • Reduce to minimum disk I/O burstiness and • Maximize aggregate I/O performance of the system (benchmark) • Same algorithms apply to network as well as local FSs • All the algorithms can be used for application DPs and MD DPs flush

  11. New algorithms for cache writeback (cont.) • We present and simulate only 5 algorithms (more were considered): • Modified Trickle Flush – improved version of trickle by changing priority and use more CPU • Fixed Interval Algorithm – use a goal as target of number of DPs similar to watermark methods but compensate better for bursts of I/O (semi-throttling) by pacing the flush to disk • Variable Interval Algorithm – use an adaptive control scheme that adapt the time interval based on the change in DP during previous interval similar to trickle but with faster adaptation in response to I/O bursts • Quantum Flush – use the idea of lowest retention of DP in cache similar to watermark based methods but adapt flush speed proportional to number of new I/Os in the previous sample time • Rate of Change Proportional Algorithm – flushes DPs proportional to the first derivative of the number of DPs using fixed interval and a forgetting factor proportional to difference between I/O rate and maximum disk throughput: c = R * (t - ti ) + W * μ μ = α * (B – R) / B

  12. Simulation results of new algorithms • Selection of best algorithm by: • Optimal behavior to unexpected bursts of I/Os • Flush best matching the rate of change in DPs in the cache (minimum DP level) • Minimize I/O slow down to clients (reduce I/O average latency) • Rate of change based algorithm with forgetting factor was best

  13. Experimental results of a real NFS server • We implemented the Modified Trickle and Rate Proportional algorithms on the Celerra NAS server • Used SPEC sfs2008 benchmark and measured the number of DP in cache with 4 msec resolution • Experimental results show some I/O slowdown using the MT algorithm resulting in 92K NFS iops (diagrams sampled at same 55K NFS iops level) • The Rate Proportional algorithm show much shorter I/O slow down time resulting in 110.6K NFS iops

  14. Summary and conclusions • Discussed new algorithms and paradigm to address the cache writeback in modern FS and servers • Discussed how the new algorithm can reduce the impact of bursts of application I/Os to the aggregate I/O performance otherwise bounded by the maximum disk speeds • We show how current cache writeback algorithms create I/O slowdown at I/O speeds that are lower than disk speed but changing rapidly • We presented reduced number of algorithms that are presented in the literature explaining their deficiencies • We discuss several new algorithms and show simulation results that allowed us to select the best algorithm for experimentation • We presented experimental results for 2 algorithms and show that Rate Proportional is the best algorithm based on the given criteria of success • Finally we discuss how these algorithms can be used for MD and DP on any file system network or local

  15. Future work and extension to Linux FS • Investigation of additional algorithms inspired from signal processing of non-linear signals that address oscillatory behavior • Address similar behavior for cache writeback of local file systems including ext3, ReiserFS and ext4 in Linux OS (a discussion at next Linux workshop) • Linux FS developers are aware of this behavior and currently work to instrument the Linux kernel with same measurement tools as we used • We are also looking to use machine learning in order to be able to compensate for very fast I/O rate changes that will allow to optimize application performance for very large number of clients • Additional work is needed to find algorithms that will allow the maximum application performance equal the maximum aggregate disk performance • We are also looking to instrument NFS clients’ kernel to allow us evaluate the I/O slow down and tune the flush algorithm to reduce the slow down effect to zero • More work is needed to extend this study to MD and find new MD specific flushing methods

More Related