Recitation 1: Intro, Revision Control, Project 1 Hints

1. Vinay Chaudhary, Michael Ashley-Rollman Most borrowed from Dave Anderson CMU Computer Science Spring 2007 Recitation 1: Intro, Revision Control, Project 1 Hints

2. Overview • Recitation Mechanics • Thoughts & Advice • Makefiles • Source Control • Project Hints • Partners?

3. Recitation Mechanics • 1) These are your recitations. • We’re here to help • We’ve got a schedule. It’s flexible. • Ask questions, make comments, … • 1 part lecture, 1 part “public office hours” (homework questions? Sure! Project questions? Great!) • 2) These aren’t the final answers • Recitations culled from our experience, other faculty, friends in industry, books, etc. • We’re always looking for better ideas/tools/practices/etc. If you have some, please share.

4. Thoughts & Advice

5. Systems Software • Low-level (projects in C) • Designed to run forever • Handle every possible error condition • Manage resources carefully • Must be secure! • The Internet is not a friendly place • Based on documented protocols

6. Quite different from 213 • Project size: 1000s of lines vs 100s. • Project duration: 5 weeks vs 1-2 weeks • Partners vs. solo developer • Result: • You can’t keep the “state” for 441 projects in your head at all times. It’s too big! • Requires one step more care in development.

7. Software EngineeringFor Systems Hackers • Goals and foundations: • 1-5 developers • Context: Systems code, but not too much is systems specific • Very low overhead and up-front cost • Like to see benefits w/in a single project

8. Our Philosophy • Your time is valuable and limited • Some things are fun: • Design, initial coding, a working project • Some things are less fun: • Agonizing debugging, bad project grades, spending 10x longer than you thought • Use techniques that minimize time and maximize fun vs less fun.

9. Partly-free lunch • Techniques take a bit of time to learn • E.g., make files, revision control software (today) • But they will pay off! • Some techniques take a bit more up-front time • E.g., writing good log messages, thinking about design, good debugging capabilities • But they make the rest of the project more predictable and reduce the uncertainty of failing in the last day. • (And they save debugging time).

10. Your job • Ask yourself: “Could I be doing this in a more efficient way?” • Typing “gcc –g –Wall foo.c bar.c baz.c”vs typing “make” • Debugging: “Have I seen this bug before? What caused it? How could I avoid it?” • Be reflective; strive to learn & improve.

11. In Practice: Algorithms • Most systems programs need: • Hashes, linked lists • Searching and sorting • For many, that’s it. • (Databases are different) • Given this, • What would a good, lazy programmer do?

12. Don’t write it twice • Hashes/lists: Have a nice implementation that you reuse. • We suggest either the ones from • “The Practice of Programming” • Or rip them out of the BSD kernel • This is perfectly acceptable in 441 • Sorting: Don’t write at all! • C library “qsort” (heap, merge…)

13. Don’t prematurely optimize • If it ain’t slow, don’t break it • Keep your programs simple • Easier to write • Easier to debug • But make it easy to change implementation details • Modularity! (Later lecture)

14. Optimizing your time • Sorting 3 numbers: Do it by hand • Sorting 3000 numbers: Do it in ruby • Sorting 300,000,000,000 numbers: Write some serious code • Mental calculation • Time spent doing task • Time spent automating/optimizing • Will you have to do this again?

15. Resources • Some great books: • The Pragmatic Programmer • The Practice of Programming • Writing Solid Code • Recitation notes: • On the course website

16. Makefiles

17. Makefiles – Example1 smtpverify: verf.c smtp.c gcc -Wall -Werror verf.c smtp.c -o smtpverify -lresolv smtpsend: send.c smtp.c gcc -Wall -Werror send.c smtp.c -o smtpsend -lresolv clean: rm -f *~ smtpverify smtpsend

18. Makefiles – Example2 VERFOBJS = verf.o smtp.o SENDOBJS = send.o smtp.o LIBS = -lresolv CC = gcc CFLAGS = -Wall -Werror .PHONY: all clean all: smtpverify smtpsend clean: rm -f *.o *~ smtpverify smtpsend cscope.out output.ps smtpverify: $(VERFOBJS) $(CC) $(CFLAGS) $(VERFOBJS) -o smtpverify $(LIBS) smtpsend: $(SENDOBJS) $(CC) $(CFLAGS) $(SENDOBJS) -o smtpsend $(LIBS)

19. Source Control

20. Revision Control • Before you write a line of code… • Use subversion/CVS/etc. • Provides access to all old versions of your code • No more “cp file.c file.c.2006-01-01-1059am-oh-god-please-let-this-work”

21. What is revision control? • A repository that stores each version • You explicitly “check out” and “check in” code and changes. • 597 bark:~/tmp> svn checkout https://moo.cmcl.cs.cmu.edu/svn/systems-se A systems-se/related.tex A systems-se/acks.tex A systems-se/tinylang.tex A systems-se/emacs.tex ….

22. Why do I want it? • Super-undo: Go to arbitrary versions • rm –rf your source tree? No problem! • Tracking changes /“why did this break?” • Concurrent development • Snapshots • Turning in the assignment: just make a snapshot when you want, and we’ll grade that. You can keep developing afterwords. • Useful, e.g., for optimization contest, or for making sure you have something working.

23. The repository • Master copy of code is separate from what you work on • You canhave multipleworking copieschecked out.(So can your partner) Repository Your working copy Laptop working copy Partner working copy

24. Check out and commit • Explicitly synchronize with the repository Repository Checkout / Update Commit Your working copy

25. Every revision is available Current version Older Version 1 Older Version 2 First Version

26. And you can see what changed Revision control lets you note (and then see) what you changed: > svn log gtcd.cc r986 | ntolia | 2006-08-01 17:13:38 -0400 (Tue, 01 Aug 2006) | 6 lines This allows the sp to get rid of chunks early before a transfer is complete. Useful when a file is requested in-order and the file size > mem cache size And makes it easy to go back to other versions: --------------------------------------------------------------------- r987 | ntolia | 2006-08-02 13:16:21 -0400 (Wed, 02 Aug 2006) | 1 line After much thought, I am reverting the last patch. We will need to revisit the issue when we think about DOT on storage-limited clients

27. Concurrent Development • Each person checks out a copy • Both can work at the same time without (much) fear of clobbering the other • Changes only visible on commit/update • What if both people edit the same file and commit it?

28. Concurrent edits Ann File v1 Drew Both check out file Edits (drew-1) Different part of same file Edits (ann-1) commit File v2 Update Successfully merged

29. Concurrent edits Ann File v1 Drew Both check out file Edits (drew-1) Overlap with ann-1 Edits (ann-1) commit File v2 Update (CONFLICT)

30. Resolving Conflicts • Subversion will give you 3 files: • The original with conflict markers • The version you were editing • The latest version in the repository • You can: • Keep your changes, discarding others • Toss your changes • Manually resolve • After finishing resolving, type svn resolved

31. Branches • Multiple paths of development, e.g. • Release 1.0 only gets security patches • “Development” branch gets everything • “tags” or “snapshots” • Save a good known state. E.g., for handing in. • Issue of merging

32. Subversion • svn checkout (co)https://moo.cmcl.cs.cmu.edu/441/.. • svn commit • svn update • svn add • svn mkdir • svn copy: create a branch or snapshot • svn diff: See difference between versionsby default: between what you started on and where you are now

33. Reverting • To remove all changes from a file and sync to the latest repository version, type svn revert. • To revert a file to a previous repository version (after you've checked it in), type svn merge -r HEAD:old_version <filename> then svn commit

34. Brief walkthrough > svn checkout https://moo.cmcl.cs.cmu.edu/svn/systems-se A systems-se/acks.tex … > cd systems-se > echo “new file” >> test.txt > svn add test.txt A test.txt > svn commit [svn will open an editor for log message] Adding test.txt Transmitting file data .. Committed revision 21.

35. Thoughts on Revision Control • Update, make, test, then commit • Update out of habit before you start editing • Merge often • Commit format changes separately • Check svn diff before committing • Try not to break the checked in copy • Invasive changes? Maybe a branch • Don’t use svn lock • Avoid conflicts by good decomposition (modularity) and out-of-band coordination

36. Go forth and revise! • Revision control will save you untold pain • Most people I know have accidentally nuked files or entire directories • Logs and diffs very useful for finding bugs • Much better way to coordinate with partners (but useful on your own! I use it for almost everything) • Very small investment to learn • Try it on your own! • Read the SVN book online for more info: http://svnbook.red-bean.com/

37. Project Hints

38. Project 1: Update • smtpverify and smtpsend now take an smtp-server argument • Allows forwarding mail through a machine • Ie: using courseweb.sp.cs.cmu.edu to send mail to andrew.cmu.edu • The project webpage is updated to reflect this

39. Project 1 Hints : DNS • The trickiest part may be getting the mail server ip address. • DNS contains multiple records • Address (A) • Mail Exchange (MX) • Name Servers (NS) • Etc • Queries return A records by default, but we need MX records • Can’t simply use gethostbyname() • Can use resolver

40. Project 1 Hints : DNS • int res_search(const char *dname, int class, int type, unsigned char *answer, int anslen); • dname is the domain name to lookup • class is usually C_IN (internet) • type is desired record (T_A, T_MX, T_NS, etc) • answer is not straight forward • Parse with ns_initparse, ns_parserr, ns_rr_rdata, ns_get32

41. Project 1 Hints : DNS • Use resolver (man resolver). Very poorly documented • Some documentation here: • http://www.unix.org.ua/orelly/networking/dnsbind/ch14_02.htm • http://www.experts-exchange.com/Programming/Programming_Languages/C/Q_20799912.html • Look at the arpa/nameser.h (/usr/include/arpa) file for definitions. • Try dig for testing out dns lookups (man dig)

42. Project 1 Hints : Other • telnet to port 25 on courseweb.sp.cs.cmu.edu and experiment • Make sure command messages end with \r\n. • The actual email should end with \r\n.\r\n • Be careful with TCP streams. • Calling tcp send/recv does not always guarantee sending/receiving the entire packet. Try putting it in a loop. • Once read is in a loop, there needs to be some way of deciding when to stop reading, can’t just block forever • Start early • Have fun

43. Project 1 Hints: Websites • http://www.greenend.org.uk/rjk/2000/05/21/smtp-replies.html • http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol • http://www.faqs.org/rfcs/rfc2821.html • http://www.faqs.org/rfcs/rfc821.html • http://www.unix.org.ua/orelly/networking/dnsbind/ch14_02.htm • http://www.experts-exchange.com/Programming/Programming_Languages/C/Q_20799912.html

44. Partners