CS510 Concurrent Systems

1. CS510 Concurrent Systems “What is RCU, Fundamentally?” By Paul McKenney and Jonathan Walpole Daniel Mansour (Slides adapted from Professor Walpole’s)

2. Review of Concurrency • The Story So Far • Critical Sections • Shared global objects • Blocking/Non-Blocking • Reader/Writers • Reads are “safer” • Writes are Fewer • Hardware/Compiler Optimizations • Operations are not always in order • Memory Reclamation • Deleted Objects can still be read

3. Read-Copy Update (RCU) • Read/Writes • Do not allow reads while writes are occurring • RCU solves this by allowing multiple versions • Insertion • Publish-Subscribe Mechanism • Deletion • Wait for Readers to complete • Non-Blocking

4. Publish-Subscribe Mechanism 1 struct foo { 2 int a; 3 int b; 4 int c; 5 }; 6 struct foo *gp = NULL; 7 8 /* . . . */ 9 10 p = kmalloc(sizeof(*p), GFP_KERNEL); 11 p->a = 1; 12 p->b = 2; 13 p->c = 3; 14 gp = p;

5. Publish-Subscribe Mechanism 1 struct foo { 2 int a; 3 int b; 4 int c; 5 }; 6 struct foo *gp = NULL; 7 8 /* . . . */ 9 10 p = kmalloc(sizeof(*p), GFP_KERNEL); 11 gp= p; 12 p->a = 1; 13 p->b = 2; 14 p->c = 3; Reordered, concurrent readers would get uninitialized values

6. Publish-Subscribe Mechanism • Publish the Object • rcu_assign_pointer 1 p->a = 1; 2 p->b = 2; 3 p->c = 3; 4 rcu_assign_pointer(gp, p);

7. Publish-Subscribe Mechanism • Readers • Ordering can still be an issue • p->a can be read before p in some compiler optimizations 1 p = gp; 2 if (p != NULL) { 3 do_something_with(p->a, p->b, p-c); 4 }

8. Publish-Subscribe Mechanism • Subscribe to the object • rcu_dereference • Implements any necessary memory barriers 1 rcu_read_lock() 2 p = rcu_dereference(gp) 3 if (p != NULL { 4 do_something_with(p->a, p->b, p->c); 5 } 6 rcu_read_unlock();

9. Implementation • Doubly Linked Lists • list_head • Head links to tail with prev and vice versa

10. Implementation - list_head • Publishing 1 struct foo { 2 structlist_head list; 3 int a; 4 int b; 5 int c; 6 }; 7 LIST_HEAD(head); 8 9 /* . . . */ 10 11 p = kmalloc(sizeof(*p), GFP_KERNEL); 12 p->a = 1; 13 p->b = 2; 14 p->c = 3; 15 list_add_rcu(&p->list, &head); Still needs to be protected

11. Implementation - list_head • Subscribing 1 rcu_read_lock(); 2 list_for_each_entry_rcu(p, head, list) { 3 do_something_with(p->a, p->b, p->c); 4 } 5 rcu_read_unlock();

12. Implementation - hlist_head/hlist_node • Doubly Linked Lists • hlist_head/hlist_node • Linear list

13. Implementation - hlist_head/hlist_node • Publishing 1 struct foo { 2 structhlist_node *list; 3 int a; 4 int b; 5 int c; 6 }; 7 HLIST_HEAD(head); 8 9 /* . . . */ 10 11 p = kmalloc(sizeof(*p), GFP_KERNEL); 12 p->a = 1; 13 p->b = 2; 14 p->c = 3; 15 hlist_add_head_rcu(&p->list, &head);

14. Implementation - list_head • Subscribing 1 rcu_read_lock(); 2 hlist_for_each_entry_rcu(p, q, head, list) { 3 do_something_with(p->a, p->b, p->c); 4 } 5 rcu_read_unlock(); Iteration requires checking for null Instead of p = head

15. Reclaiming Memory • Waiting • rcu_read_lock and rcu_read_unlock • Must not block or sleep

16. Reclaiming Memory • Basic Process • Alter the data (replace/delete an element) • Wait for all pre-existing RCU reads to completely finish • synchronize_rcu • New RCU reads have no way to access removed element • Reclaim/Recycle memory

17. Reclaiming Memory • Example 1 struct foo { 2 structlist_head list; 3 int a; 4 int b; 5 int c; 6 }; 7 LIST_HEAD(head); 8 9 /* . . . */ 10 11 p = search(head, key); 12 if (p == NULL) { 13 /* Take appropriate action, unlock, and return. */ 14 } 15 q = kmalloc(sizeof(*p), GFP_KERNEL); 16 *q = *p; 17 q->b = 2; 18 q->c = 3; 19 list_replace_rcu(&p->list, &q->list); 20 synchronize_rcu(); 21 kfree(p);

18. Reclaiming Memory • synchronize_rcu • rcu_read_lock and rcu_read_unlock do nothing on non-CONFIG-PREEMPT kernels • Recall that code in rcu_read blocks can not sleep or block, preventing context switches • When a context switch occurs, we can deduce that the read has completed • synchronize_rcu waits for each cpu to context switch conceptually: 1 for_each_online_cpu(cpu) 2 run_on(cpu);

19. Maintaining Multiple Versions - Deletion • During deletion • Initial State 1 p = search(head, key); 2 if (p != NULL) { 3 list_del_rcu(&p->list); 4 synchronize_rcu(); 5 kfree(p); 6 }

20. Maintaining Multiple Versions - Deletion • list_del_rcu(&p->list) • p has been deleted, but other readers can still be accessing it

21. Maintaining Multiple Versions - Deletion • synchronize_rcu() • kfree(p)

22. Maintaining Multiple Versions - Replacement • During replacement • Initial State 1 q = kmalloc(sizeof(*p), GFP_KERNEL); 2 *q = *p; 3 q->b = 2; 4 q->c = 3; 5 list_replace_rcu(&p->list, &q->list); 6 synchronize_rcu(); 7 kfree(p);

23. Maintaining Multiple Versions - Replacement • kmalloc()

24. Maintaining Multiple Versions - Replacement • *q = *p

25. Maintaining Multiple Versions - Replacement • q->b = 2; q->c = 3

26. Maintaining Multiple Versions - Replacement • list_replace_rcu • There are now two versions of the list, but both are well formed.

27. Maintaining Multiple Versions - Replacement • synchronize_rcu • kfree()

28. Issues • Very much a read/write implementation, not ideally suited for rapidly updated objects • Responsibility on the programmer to make sure that no sleeping/blocking occurs in rcu_read_locks • RCU updaters can still slow readers via cache invalidation