6.4 Data and File Replication

1. 6.4 Data and File Replication Neha Purohit

2. Why replicate • Performance • Reliability • Resource sharing • Network resource saving

3. Challenge Transparency Replication Concurrent control Failure recovery Serialization

4. Atomicity • In database systems, atomicity is one of the ACID transaction properties. An atomic transaction is a series of database operations which either all occur, or all do not occur[1]. • All or nothing

5. Atomicity • In DFS (Distributed File System), replicated objects (data or file) should follow atomicity rules, i.e., all copies should be updated (synchronously or asynchronously) or none.

6. Goal One-copy serializability: The effect of transactions performed by clients on replicated objects should be the same as if they had been performed one at a time on a single set of objects.[2]

7. Architecture[3]

8. Read operations [3] • Read-one-primary, FSA only read from a primary RM, consistency • Read-one, FSA may read from any RM, concurrency • Read-quorum, FSA must read from a quorum of RMs to decide the currency of data

9. Write Operations[3] • Write-one-primary, only write to primary RM, primary RM update all other RMs • Write-all, update to all RMs • Write-all- available, write to all functioning RMs. Faulty RM need to be synched before bring online.

10. Write Operations • Write-quorum, update to a predefined quorum of RMs • Write-gossip, update to any RM and lazily propagated to other RMs

11. Read one primary, write one primary • Other RMs are backups of primary RM • No concurrency • Easy serialized • Simple to implement • Achieve one-copy serializability • Primary RM is performance bottleneck

12. Read one, Write all • Provides concurrency • Concurrency control protocol needed to ensure consistency (serialization) • Achieve one-copy serializability • Difficult to implement (there will be failed TM to block any updates)

13. Read one, Write all available • Variation of Read one, Write all • May not guarantee one-copy serializability • Issue of loss conflict in transactions

14. Read quorum, Write quorum • Version number attached to replicated object • Highest version numbered object is the latest object in read. • Write operation advances version by 1 • Write quorum > half of all object copies • Write quorum+read quorum > all object copies

15. Gossip Update • Applicable for frequent read, less update situations • Increased performance • Typical read one, write gossip • Use timestamp

16. Basic Gossip Update • Used for overwrite • Three operations, read, update, gossip arrive • Read, if TSfsa<=TSrm, RM has recent data, return it, otherwise wait for gossip, or try other RM • Update, if Tsfsa>TSrm, update. Update TSrm send gossip. Otherwise, process based on application, perform update or reject • Gossip arrive, update RM if gossip carries new updates.

17. Causal Order Gossip Protocol[3] • Used for read-modify • In a fixed RM configuration • Using vector timestamps • Using buffer to keep the order

18. Windows Server 2008[4] • Support DFS • “State based, multi master” scheduled replication • Use namespace for transparent file sharing • Use Remote Differential Compression to propagate change only to save bandwidth

19. FUTURE WORK IN FILE REPLICATION

20. DISTRIBUTED FILE SYSTEMS: STATE OF THE ART • GFS: Google File System –Google –C/C++ • HDFS: Hadoop Distributed File System –Yahoo –Java, Open Source • Sector: Distributed Storage System –University of Illinois at Chicago –C++, Open Source

21. FILE SYSTEMS OVERVIEW • System that permanently stores data • Usually layered on top of a lower-level physical storage medium • Divided into logical units called “files” –Addressable by a filename (“foo.txt”) –Usually supports hierarchical nesting (directories) • A file path joins file & directory names into a relative or absolute address to identify a file (“/home/aaron/foo.txt”)

22. SHARED/PARALLEL/DISTRIBUTED FILE SYSTEMS • Support access to files on remote servers • Must support concurrency –Make varying guarantees about locking, who “wins” with concurrent writes, etc... –Must gracefully handle dropped connections • Can offer support for replication and local caching • Different implementations sit in different places on complexity/feature scale

23. SECTOR AND SPHERE • Sector: Distributed Storage System • Sphere: Run-time middleware that supports simplified distributed data processing. • Open source software, GPL, written in C++. • Started since 2006, current version 1.18 • http://sector.sf.net

24. Sector –Brief Definition Sector is an open source data cloud model. Its Assumptions: Presence of a high bandwidth data link among the racks in a Data Center and also among different Data Centers. Also, that individual applications may have to process large streams of data and produce equally large output streams.

25. SECTOR: DISTRIBUTED STORAGE SYSTEMS

26. SECTOR: DISTRIBUTED STORAGE SYSTEMS • Sector stores files on the native/local file system of each slave node. • Sector does not split files into blocks –Pro: simple/robust, suitable for wide area –Con: file size limit • Sector uses replications for better reliability and availability • The master node maintains the file system metadata. No permanent metadata is needed. • Topology aware

27. SECTOR:WRITE/READ •Write is exclusive •Replicas are updated in a chained manner: the client updates one replica, and then this replica updates another, and so on. All replicas are updated upon the completion of a Write operation. •Read: different replicas can serve different clients at the same time. Nearest replica to the client is chosen whenever possible.

28. SECTOR: TOOLS AND API •Supported file system operation: ls, stat, mv, cp, mkdir, rm, upload, download –Wild card characters supported •System monitoring: sysinfo. •C++ API: list, stat, move, copy, mkdir, remove, open, close, read, write, sysinfo.

29. Sector: Architecture Sector manages its data with the help of the following components. Security Server : Authenticates the clients and the slave servers Master Server : Contains File Meta Data, Schedules the work among the slave servers. Slave Servers: Contains datasets divided amongst them in the form of Linux files.

30. Sphere: A Brief Definition Sphere is a programming model built over the Sector architecture of data cloud. It falls under the Single instruction Multiple Data Category from Flynn’s taxonomy.

31. Sphere : Computing Model Sphere identifies the individual records in a file with the help of index files. Each record or a bunch of records are treated as independent data entities that can be processed in Parallel by different slave nodes. Similar to data parallelism. These slave nodes are managed by Sphere Processing Engines(SPE) and the SPE are scheduled by Sphere. Multiple stages of SPE can be coupled together.

32. MAP REDUCE Similarity to Map Reduce Sphere model is quite similar to Map Reduce because both of them deal with data parallelism and allow coupling of at least two stages of processing elements. Map Reduce shuffles the data among the slave nodes in the second stage whereas Sphere allows the output of the first stage to be distributed among different processing nodes of the second stage.

33. References [1] Wikipedia; http://en.wikipedia.org/wiki/Atomicity [2] M. T. Harandi;J. Hou (modified: I. Gupta);"Transactions with Replication";http://www.crhc.uiuc.edu/~nhv/428/slides/repl-trans.ppt [3] Randy Chow,Theodore Johnson, “Distributed Operating Systems & Algorithms”, 1998 [4] "Overview of the Distributed File System Solution in Microsoft Windows Server 2003 R2";http://technet2.microsoft.com/WindowsServer/en/library/d3afe6ee-3083-4950-a093-8ab748651b761033.mspx?mfr=true [5] "Distributed File System Replication: Frequently Asked Questions";http://technet2.microsoft.com/WindowsServer/en/library/f9b98a0f-c1ae-4a9f-9724-80c679596e6b1033.mspx?mfr=true

34. [6] http://code.google.com/edu/parallel/dsd-tutorial.html [7] http://code.google.com/edu/parallel/mapreduce-tutorial.html [8]http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en/us/papers/gfs-sosp2003.pdf [9]http://arxiv.org/ftp/arxiv/papers/0809/0809.1181.pdf