40 Bosses, 3000 Users, 20 Projects (and a side order of fries)… Managing Computers for Academics

1. 40 Bosses, 3000 Users, 20 Projects (and a side order of fries)…Managing Computers for Academics David Parter, University of Wisconsin (chair) David N. Blank-Edelman, Northeastern University. Jim Duncan, Penn State University Christopher Thorpe, Harvard University

2. Site Reviews Funding Staffing User Support Problem Reports External Relationships Sources of Conflict Decision Making Security Changing Technology Favorite Parts of Site Least Favorite Parts A Typical Day Top 10 List Overview • Questions and Answers

3. Northeastern University CCS • 1 Building with pod in research center • Soon to be all switched 10/100 Ethernet • UNIX boxen (~2000 accounts) • ~125 SunOS Sparc’s/SunTerms, Solaris UltraSparc’s • ~40 Digital UNIX Alphas • ~40 MacOS (~180-200 users) • ~50 Pentium’s running NT 4.0 (1200 Users) • Misc.: Win95 laptops, BeBoxen, ATM, Auspex, dialup pool, peripherals

4. Penn State Applied Research Lab • 1000 users in 4 buildings on campus, multiple remote locations • 850-1300 computers • Funded primarily by the US Navy to perform basic and applied research for the Navy, contribute to the education and research goals of the university, and transfer technology to government and industrial sectors • Act as network manager and principal systems administrator • Lab management and decision-making happen mostly at the level of department managers and division heads

5. Univ. of Wisconsin CS Dept. • Mission: support instructional, research and administrative computing in the department • 6,000 users in database • 350 Unix Desktop Systems, 72 Unix Instructional Workstations • Full Support: solaris, solaris X86, Digital Unix • Partial Support: linux, irix, aix, hpux... • 75 NT Desktop Workstations, 100 NT Instructional Workstations • Kerberos V and AFS on all platforms

6. Harvard EECS Department • 200 active users: faculty, grad students, researchers, staff • 200+ machines: HP/UX, BSDi, Digital UNIX, SunOS, Solaris, NetApp • Funding: Research grant • Support: Single professional sysadmin supervises students and researchers

7. Funding • budget lines for systems staff and maintenance only, generally don’t charge researchers for maintenance, money for client machines from ACAC/donations, most researchers fund their own equipment, real hole is in money for server and network upgrades • Funded primarily by the US Navy and many other clients • 1/3 College "Instructional”, 1/3 College "Research”, 1/3 Facilities Chargeback paid by research projects ($1800/workstation/year, billed semi-annually) • All funding comes from a substantial grant to six professors for the "Ubiquitous Information" project

8. Staffing • 4 FTE (hiring!), 1 Co-op, 5-15 student volunteers w/some serious responsibilities (“The Crew”) • Full-time staff in our group, but many part-time and untrained student and staff sysadmins in other departments • 8 FTE Professional Staff, 2 Research Assistants (1/2-time Grad Students), 17 Undergrads (6 FTE) • One full-time professional sysadmin, 5+ paid students, many unpaid but "clueful” graduate students also help

9. User Support • Web, Overview/UNIX tools class, clue sheets/FAQ’s • Help desk in development, req in use by group co-workers only,web pages provide information to users • Web Pages, UNIX “Orientation” sessions at start of semester • “student slave” creates and manages a documentation website, most users are fairly proficient (researchers in computer science) so few require substantial assistance

10. Problem Resolution • Coincidentally, req (as much as possible) • Req internally by the CSN group, other problems reported via hostmater alias and telephone, resolution through many channels (usually informally and without complete documentation) • Req • Small site served well by email to administrator and staff, reply all for escalation or completion of problems

11. Task Management • Req dispatch, staff meetings, crew surprises • Tasks are assigned by subject areas and availability • Weekly software staff coordination meeting • Weekly “ice cream” session and lab whiteboard

12. External Relationships • Own everything from campus net drop in. DAC • ARL participates in many university-wide projects, provides some security advising or consulting to other PSU departments • Work with Computing Center on some issues, we own network/routers/etc., cooperate & serve on committees • Network managed by Faculty of Arts and Sciences network group, autonomous for all other services (e.g. DNS/mail), share some services with other groups (e.g. backups/news)

13. Sources of Conflict • DAC, power boundary conditions w/faculty • Disagreement on quantity or type of services occurs frequently and usually goes unsolved • Computing Center (sometimes), poor communication with faculty, ambiguous direction from faculty, lack of policy guidance • Parallel responsibilities for the department's support system, sometimes unclear who is responsible (e.g. backups/license servers)

14. Decision Making • Us, Dean, Faculty, Students • Typical management tree, except that I make many decisions that most can't understand • Facilities Committee, weekly staff meeting w/Facilities Chair, staff consensus • Professional system administrator is The Boss, but relies on input from faculty, grad students and student staff, P.I. veto power

15. Security • Standard defense measures (tcpwrap, crack,etc.), switched ports, SSH/OTP, espionage • Planning to provide KRB/KDC for entire lab, incident handling still largely ad-hoc, level of security can vary widely from machine to machine • Kerberos V and SSH, phase out clear text passwords (KPOP, ktelnet...), “unsup” net, “tripkerb” • Less emphasis on security than a commercial site, monitor for suspicious network activity and modification of important files, a 486 with console logins only syslogs everything , SSH & some Kerberos

16. Changing Technology • Assume change, push hard to be sane about it • Discover and compensate for unexpected unannounced changes, attempt to guide lab to solutions for problems-to-be • Deal with surprises, try to standardize hardware, 99% solution for software installation • Professors often get new software/hardware that we've never had to deal with, negotiating admin/location/etc. can be challenging

17. Least Favorite Parts of Site • Dependency & archeology • High turnover and offered pay makes it hard to find and keep good sysadmins, difficult to accomplish many obvious tasks • Surprises, lack of policy, lack of respect • Student staff often doesn't complete large projects well: academic responsibilities often get in the way

18. Favorite Parts of Site • Creative possibilities (innovation expected), management that “gets it” and sane policies, • Work environment, co-workers, variety, fiber backbone • teaching students, making improvements, one facility for entire department • Giving grad students and trusted student slaves root privileges allows 24/7 uptime with a single 9-5 professional administrator

19. A Typical Day • A Day in the Life of David Blank-Edelman • dnb@ccs.neu.edu • A Day in the Life of Jim Duncan • jim.duncan@psu.edu • A Day in the Life of David Parter • dparter@cs.wisc.edu • A Day in the Life of Christopher Thorpe • cat@eecs.harvard.edu

20. David Blank-Edelman’s Top 10 ReasonsTo Be/Not Be in Academia • Potentially less stress • Research • No marketing department • Educational discounts and donations • AOL/CIS/Prodigy is not your ISP

21. David Blank-Edelman’s Top 10 ReasonsTo Be/Not Be in Academia • UNIX • We don’t serve “The Almighty Buck” • Freedom of Speech • “That Professor” • Dress code

22. Questions and Answers