Region-Based Software Distributed Shared Memory

1. Region-Based Software Distributed Shared Memory Song Li, Yu Lin, and Michael Walker CS 656 -- Operating Systems May 1, 2000

2. Outline • Motivation • Problems • DSM Models • Proposed Solution • Conclusion

3. Motivation • Share distributed resources to increase computing power • First Solution: Tightly-coupled system • Problem: Central memory bus • Alternative: Loosely-coupled system • New Problem: Need IPC in distributed systems

4. Motivation: RPC • Solution: Remote Procedure Calls (RPC) • Snd/Rcv protocol • Pass-by-value • OS Assignment 1 • Problems with RPC • Explicit awareness of communication • Marshalling complex data structures is hard

5. Motivation: DSM • Solution: Distributed Shared Memory (DSM) [Li86--Yale] • Idea: Nonresident pages are fetched from network • Illusion of tightly-coupled system in a loosely-coupled one • Abstraction on top of message-passing model • Programmers use memory-accessing paradigm

6. DSM: Potential Benefits • Shared access to memory • Avoids Von-Neumann bottleneck • Familiar abstraction • Pass-by-reference • Spreads communication load over time • Provides more memory than is locally available

7. Problems • Who handles remote access? • What should be shared? • Cache, cache, cache…but how? • Page Replacement & Thrashing • More than one way to solve the problems...

8. DSM Models: Decisions • Hardware: MMU controls message passing, data migration, and caching • Software: Control managed by OS or library • Page-based shared memory versus shared variable versus Object • Cache consistency models

9. Proposed Solution: Overview • Software-based model using libraries • Variable-size regions -- similar to segments • Every granularity has its advantages/drawbacks • Make the region size flexible, provide reasonable defaults (sound familiar?) • Multiple reader, single writer (MRSW) caching support

10. Solution: User Interface • r_handle sm_malloc(size); • int sm_regionat(r_handle, attrib); • int sm_read(r_handle, offset, buf, size); • int sm_write(r_handle, offset, buf, size); • int sm_regiondt(r_handle); • int sm_free(r_handle);

11. Client n Client 1 Client 2 Client Table Client Table Client Table Cache Cache Cache Manager Master Table Shared Memory Shared Memory Shared Memory Provider 1 Provider 2 Provider n Solution: Mode of Operation ... ...

12. Client 2 Client n Client 1 Client Table Client Table Client Table Cache Cache Cache Manager Master Table Shared Memory Shared Memory Shared Memory Provider 1 Provider n Provider 2 Solution: Mode of Operation ... (1) Allocate memory ...

13. Client 2 Client n Client 1 Client Table Client Table Client Table Cache Cache Cache Manager Master Table Shared Memory Shared Memory Shared Memory Provider 1 Provider n Provider 2 Solution: Mode of Operation ... (2) Query table & attach region ...

14. Client 2 Client n Client 1 Client Table Client Table Client Table Cache Cache Cache Manager Master Table Shared Memory Shared Memory Shared Memory Provider 1 Provider n Provider 2 Solution: Mode of Operation ... (3) Send region info to client ...

15. Client 2 Client n Client 1 Client Table Client Table Client Table Cache Cache Cache Manager Master Table Shared Memory Shared Memory Shared Memory Provider 1 Provider n Provider 2 Solution: Mode of Operation ... (4) Client communicates with provider ...

16. Client 2 Client n Client 1 Client Table Client Table Client Table Cache Cache Cache Manager Master Table Shared Memory Shared Memory Shared Memory Provider 1 Provider n Provider 2 Solution: Mode of Operation ... (5) Client uses cached copy ...

17. Client 2 Client n Client 1 Client Table Client Table Client Table Cache Cache Cache Manager Master Table Shared Memory Shared Memory Shared Memory Provider 1 Provider n Provider 2 Solution: Mode of Operation ... (6) Client detaches and frees memory ...

18. Client 2 Client n Client 1 Client Table Client Table Client Table Cache Cache Cache Manager Master Table Shared Memory Shared Memory Shared Memory Provider 1 Provider n Provider 2 Solution: Mode of Operation ... (7) Memory is freed ...

19. Solution: MRSW Caching Model • Replication is good for reads, so allow MR • No communication overhead • Writer requires cache invalidation, so restrict to SW • MW is too complicated

20. Client 2 Client n Client Table Client Table Cache Cache Manager Master Table Shared Memory Shared Memory Shared Memory Provider n Provider 1 Provider 2 Solution: MRSW Caching Model Readers Client 1 ... Client Table Cache (1) Multiple readers ...

21. Client 2 Client n Client Table Client Table Cache Cache Manager Master Table Shared Memory Shared Memory Shared Memory Provider 2 Provider n Provider 1 Solution: MRSW Caching Model Readers Writer Client 1 ... ? Client Table Cache (2) Write request ...

22. Client 2 Client n Client Table Client Table Cache Cache Manager Master Table Shared Memory Shared Memory Shared Memory Provider 2 Provider n Provider 1 Solution: MRSW Caching Model Readers Writer Client 1 ... Client Table Cache (3) Cache invalidation to selected clients only ...

23. Client 2 Client n Client Table Client Table Cache Cache Manager Master Table Shared Memory Shared Memory Shared Memory Provider 2 Provider n Provider 1 Solution: MRSW Caching Model Readers Writer Client 1 ... Client Table Cache (4) Single writer, multiple readers ...

24. Conclusion: Current Status • Basic client/manager model is implemented

25. Conclusion: Remaining Work • Page replacement strategy: LRU • Region protection mechanism • Replicated managers for better performance and fault tolerance • Evaluation...

26. Conclusion: Evaluation • Find distributed apps to use DSM model • Performance evaluation: • Test apps with DSM model and message-passing model • Desired results: DSM performance is better