A Design of User-Level Distributed Shared Memory

1. A Design of User-Level Distributed Shared Memory Progress Report Zhi Zhai Feng Shen Computer Science and Engineering University of Notre Dame Oct. 27, 2009

2. Outline • Part I: General Ideas • Part II: Related Work • Part III: Implementation • Client/Server Processes • C/S Page Tables • Page Fault Handler • Consistency mechanism • Part IV: Accomplished Work • Part V: Future work

3. General Ideas DSM Characteristics: • Physically: distributed memory • Logically: a single shared address space P1 P2 P3 Pn-1 M1 M2 M3 Mn-1 Software DSM Layer

4. Structure of DSM … CPU CPU CPU MemoryBus Memory DSM Hardware Network + Simpler abstraction + Possibly better performance: Larger memory space - no need to do paging on disk + Process migration simplified - one process can easily be moved to a different machine since they all share the same address space • Long shared memory access can be a bottleneck!

5. Related Work Models and Main Features: • IVY (Yale) - Divided Space: Shared & Private space • Mirage (UCLA) - Time Interval d : Avoid page thrashing • TreadMarks (Rice) - Lazy Release Consistency : Improve efficiency • SAM (Stanford)

6. Sample Operation connect Get Addr. connect Fetch Page

7. Implementation • Server Process and Client Process • Server Page Table and Client Page Table • Page Fault Handler • Consistency Mechanism

8. Client/Server Process Client Process

9. C/S Process • Listening new requests coming in • Server Process Listening Thread Page Table Thread • Page Table Management • Client n C/S Communi-cation • Processing requests from client 1 • Client 2

10. C/S Page Table • Server Page Table • Client Data • Client IDs, IP addresses • Page Info for all • Setting the number of pages/frames • Owners / Prot bits/ frame mappings (all) • Does not care underlying storage on each client

11. C/S Page Table • Client Page Table • Storage Info • Pointer to the physical memory • Address space of the Virtual Memory • Page Info for local pages • Self-owned pages • Cached pages • Owners / Prot bits / frame mappings (local) • Does not care pages not visited

12. Page Fault Handler • Fetch the IP address and frame # • Clone the demanded page • Update Prot bits • Executing operations A B C… • Writing back dirty pages (writing) • Restore Prot bits

13. Consistency Mechanism • Single Writer / Multi-Readers • Snap-shot, one writing allowed • Page Modification • Two folded role of local frames • Two reads should return the same value • Occurrences • Writing operations • Page replacement • Block modifications to the pages being used • Variable: use_counts (>0? Wait or redo: modify OK ) • Fcntl lock (modifications suspended)

14. Accomplished work Server Client Ready (bima.helios.nd.edu) Client 1 (chopin.helios.nd.edu) Communicating Client 2 (mozart.helios.nd.edu)

15. Future Work • Page Fault Handler Implementation • Testing Plan • Inspired by JUMP (Univ. of Hong Kong) • Similar Mechanism: File locks to keep consistency • Source Code Available • Relatively New: 2001 • Comparing the performance of the same application on: • DSM vs. Single Machine • Different settings • Different DSM Systems

16. Thank You!

17. Appendix

18. Algorithms Implementation • Central Server Algorithm • Migration Algorithm • Read-Replication Algorithm • Full-Replication Algorithm Non Replicated Replicated Central Full Replication Non Migrated Migrated Migration Read Replication

19. Consistency Model • Strict Consistency • Causal Consistency • Weak Consistency • Release Consistency

20. Granularity • Granularity: size of the shared memory unit • Large page size: + less overhead incurred due to page size - greater chance for contention to access a page by many processes. • Smaller page sizes: + less apt to cause contention (reduce the likelihood of false sharing) - Higher Overhead

21. C/S Page Table • Server Page Table Entries • npages, nframes • nframes – set by the server owner • Npages – decided by # clients connected • client_addresses • Be accessed when page fault occured • Full_page_mappings • Recorded the frame # each page is located in • Full_page_bits • Indicate the usage status of each page

22. C/S Page Table • Client Page Table Entries • Client_id (int) • Assigned by the server • nframes, npages (int) • Constant values configured on the server • Physmem • Actual allocated physical address / frame spaces • PROT_READ|PROT_WRITE • Virtmem • Virtual memory address range • PROT_NONE, MAP_NONRESEARVE

23. C/S Page Table • Local_Page_mapping • Page # -> frame # they are loaded in • local_page_bits (PROT_NONE for unknown pages) • PROT_NONE, PROT_WRITE, PROT_READ, PROT_READ|PROT_WRITE • local_page_owners • To separate the pages owned by local client and the pages loaded from other clients • Use_status (int) • Indicate if the owned page s are being cached or written by other clients • Page_fault_handler

24. Page fault handler • Reading Attempts • Send PAGE_REQ to server • Fetch the corresponding client address and the frame # • Modify the server page table • page_bits -> PROT_READ • Clone the page from page owner to the local frame x • Modify the local page table • Page_mapping -> framex • Page_owner -> remote client ID • Page _bits -> PROT_READ

25. Memory Consistency • Writing • Snapshot • The client processes who have cloned the page to local memory will not see the change until being notified after the writing completes • One concurrent writer only • The server page table bits will be set as PROT_READ|PROT_WRITE, write requests to the same page will be delayed until the writing program exits

26. Memory Consistency • Page Replacement • Two consecutive read on the page should return the same value if no writing is requested • The local frame being read or written by other clients will not be replaced • Use_status • == 0, no other clients are using this page • > 0, the number of clients who are reading or writing this page