Prototype for a Windows Remote Desktop Web System Utilizing AFS

1. Prototype for a Windows Remote Desktop Web System Utilizing AFS By Rodney M. Dyer A Presentation to the AFS & Kerberos Best Practices Workshop 2005 Carnegie Mellon University June 20-24, 2005 Lee College of Engineering

2. Who Am I • Windows System Programmer for the William States Lee College of Engineering at the University of North Carolina at Charlotte. • I design, program, and perform service work for the college’s Mosaic PC Network Group. Typical jack of all trades, master of few. • AFS user since 1992. Lee College of Engineering

3. The Lee College of Engineering Mosaic Computing Environment • Provides desktop computing platforms for all of the college faculty, staff, and students. • Approximately 950 fully managed and secured machines. Of those, around 500 are PCs running Windows XP, while the rest are Sun Workstations and servers running Solaris. Soon to offer Linux workstation platform as well. • Uses Kerberos authentication with AFS file system for data storage. • 100+ available PC applications, and 150+ Sun Solaris applications. • Fully scripted build, update, and patching system via remote management. Lee College of Engineering

4. Typical Mosaic Student Lab Lee College of Engineering

5. What was Mosaic missing? • A method to provide Internet and home based connectivity to the Mosaic XP platform beyond XServer, web based, or client-server application based file transfers. • A familiar interface to the on campus systems with the same available application base. • A way for the users on campus data and settings to follow them elsewhere. • Non-intrusive support for laptops instead of complete laptop conversions, giving users applications to install, or creating laptop support (cost) centers. • A method to provide non-Mosaic on campus systems access to Mosaic. Eg, independent PCs used for research. This is not altogether different from the laptop support problem. Lee College of Engineering

6. Available Solutions • Citrix • Microsoft Windows Terminal Server • XP Remote Desktop • VNC Derivatives • Others Lee College of Engineering

7. Citrix • Can be more expensive in terms of cost and management; generally requires the setup of higher-end dedicated servers. • The expectation that one, or a few servers will serve many users (multi-user sessions). • Many corporate installations of Citrix only utilize application sharing to limit the number of processes the user can start. The Mosaic group would like to export the entire desktop which allows the user full control. • Runs on Windows 200x Server requiring extra licenses. • Requires a hands-on management approach since Windows Server doesn’t fit our existing Windows XP client desktop automation strategy. • Would require another dedicated position to manage. • Multi-user operation doesn’t fit well with scientific and engineering applications as they require extra CPU and memory resources than standard office packages; problems with applications data colliding on the local file system. Lee College of Engineering

8. Microsoft Windows Terminal Server • Not so expensive since we get large academic discounts. • Requires TS client access licenses. • The same core issues as with Citrix. The biggest being that our existing XP build and application base isn’t compatible with multi-user scenarios. Lee College of Engineering

9. VNC • Already installed for Mosaic administrative and help desk functions. • Generally slower than MS RDP and Citrix ICA. • A simple frame grabber of the console display (no deep integration). • One username and password; no Kerberos authentication. • No disk drive, printer, serial port, or sound redirection. • Communication beyond initial logon not encrypted without additional tunneling setup. • Some versions of VNC do have better features but with added cost. Lee College of Engineering

10. XP Remote Desktop • Zero cost to our group, we get Windows XP for free based on volume license agreement. • Server is built into XP Pro; most XP users have the client built in. • Immediately compatible with existing build, update, and patch management systems of existing Mosaic XP platform; looks just like any other Mosaic XP desktop machine. • No multi-user local file system application data collision problems. • All CPU and memory resources dedicated to a single user. • Update or reboot single RD machine without effect to other users. • Communication is encrypted, and uses Windows standard logon. Lee College of Engineering

11. Proposal Create a new Mosaic service consisting of a web site and related systems that allows access to XP remote desktop enabled workstations. Call the new system “MosaicAnywhere”. Lee College of Engineering

12. MosaicAnywhere Design • Simple and maintainable. • Web site uses standard HTML. • Use our existing Apache web server running on top of Solaris and AFS. • Use low end rack mount servers for RD machines with possible use of existing lab machines during nights, weekends, and holidays. • A service to monitor and control the user session on the RD enabled machine. • A service to “probe” the RD machines to determine availability. • A CGI script to check the availability data and generate html for the web site. Lee College of Engineering

13. MosaicAnywhere Remote Desktop System Components • Mosaic XP built workstations with RD enabled. • A single Mosaic XP workstation used as a RD availability probe server. • Apache web server with access to AFS tree. • RDSessiond – RD user session manager service. This service runs on each RD machine. • RDProbed – Service to monitor each remote desktop workstation for availability. This service runs on the probe server. • RDCgi – Creates RD system status list web page upon http request. Lee College of Engineering

14. MosaicAnywhere use of AFS • All web page content will be stored in AFS. • RD systems list will be stored in the AFS web site as a simple flat text file RDLIST.TXT. • System availability data created by “RDProbed” service will be stored in AFS for the web site to access. • User “session locks” will be stored in AFS to prevent simultaneous logon to multiple RD machines*. * Single logon session locks and their creation are outside the scope of this presentation. Only a brief overview will be discussed. Lee College of Engineering

15. The AFS Client Cache • A local file used to store recently accessed files from AFS so that if they are re-read within a given time period, and the AFS file data hasn’t changed, the client will not re-request the data again from the AFS file server. This reduces network bandwidth and decreases the response time for requested AFS files. • File data is accessed in “chunks” (also known as blocks) of a specified byte count from the AFS server. An OS file handle is allocated for each chunk, so the total number of handles required would be calculated by dividing the chunk size into the cache size. • The number of files that can be placed in the cache is specified by a setting called the “status cache entries”. Status entries store information about the files in the cache. There is a trade-off between cache size and status entries. If you are requesting small files from AFS, you should increase the number of status entries available for the cache. • Cache size, location on disk, chunk size, and the number of status entries are all configurable. • The AFS client will retain the cached contents in the cache file if the AFS service is shutdown and restarted. This is known as a “persistent cache”. • In case of problems, the “cmdebug” utility can be used to examine information about the current state of the cache. Lee College of Engineering

16. AFS Callback Mechanism • Used to provide consistency between a file in the local client AFS cache, and the true file that exists on the server. • A promise by the file server to contact the client when the file on the server has been changed. • The callback promise has a limited life time that depends on the number of users accessing the file on the server. • For read-only volumes a callback is outdated (broken) when the replicated volume is released. • The commands "fs flush" and "fs flushvol" are used to manually force the AFS service to re-read and re-cache the server file upon access. Lee College of Engineering

17. Callback Specifics • The following code snippet from the OpenAFS source tree reveals the current time-out quantification values for callbacks… From: src/afs/callback.c /* Time to live for call backs depends upon number of users of the file. * TimeOuts is indexed by this number/8 (using TimeOut macro). Times * in this table are for the workstation; server timeouts, add * ServerBias */ static int TimeOuts[] = { /* Note: don't make the first entry larger than 4 hours (see above) */ 4 * 60 * 60, /* 0-7 users */ 1 * 60 * 60, /* 8-15 users */ 30 * 60, /* 16-23 users */ 15 * 60, /* 24-31 users */ 15 * 60, /* 32-39 users */ 10 * 60, /* 40-47 users */ 10 * 60, /* 48-55 users */ 10 * 60, /* 56-63 users */ }; /* Anything more: MinTimeOut */ Lee College of Engineering

18. MosaicAnywhere Probe Server Callback Scenario • Upon startup, the probe server creates a RD status file for each remote desktop machine in the AFS MosaicAnywhere web site. • When the status of a RD machine changes, the status file in AFS will be changed. • Since the status files are only written when the status changes, only the local probe machine AFS cache will be used until then and so network traffic is reduced. Lee College of Engineering

19. MosaicAnywhere Web Server Callback Scenario • The RD web page CGI script reads each probe server status file for each RD machine upon http request. • The RD status files only change when the probe server writes the change so*… • The web server will generally always read from the local AFS cache reducing network traffic. For frequent user requests of a web server this is very important. * And this is the main point being made by this presentation.  Lee College of Engineering

20. Responsibilities of RDSessiond • Used to manage the RD user session. • Keeps track of session time and idle time. • Provides session time extensions when possible. • Provides interface for a user RDTimer GUI to indicate how much session time the user has left on the RD machine. • Provides popup dialogs that inform user of idle time, logoff warning time, extension time, and logoff. • Removes user session lock preventing multi-workstation logon. Lee College of Engineering

21. Responsibilities of RDProbed • Read list of RD machines from RDLIST.TXT. • If a RD machine RDP file doesn’t exist in the web site, it creates one from a default.rdp file. • Ping each RD machine to determine if responding on the network. • If ping ok, perform quser.exe* to determine system availability. • Read previous status state from AFS website status file. • If status changed, then write new status to website. • Perform these operations every 14 seconds per machine. * A utility to determine who is logged on a terminal server. Lee College of Engineering

22. Responsibilities of RDCgi • Read list of RD machines from RDLIST.TXT. • For each RD machine, read its status from the status file. • Generate HTML for IFRAME in main Index.html. • Generated HTML is a simple table with an image and text representing the RD status. • Both the html image and text are hyperlinks to an RDP file for a particular system. • Detects whether browser is IE or non-IE, and points hyperlinks to the appropriate RDP file, either UNICODE, or ASCII. Lee College of Engineering

23. Lee College of Engineering

24. MosaicAnywhere Web Site Lee College of Engineering

25. Possible MosaicAnywhere Icons • The “Probe Error” only occurs if the RDProbed service cannot understand the output of quser.exe. • The “CGI Error” only occurs if the RDCgi web server script cannot understand the RD machine state file value. Lee College of Engineering

26. RDTimer GUI Application This application is the MosaicAnywhere session time progress indicator. It starts up under the users own account when they logon. It simply reads the value of the session time from the RDSessiond service and provides a visual representation of the remaining time. The RDTimer auto-adjusts when session time extensions are added. Lee College of Engineering

27. Scaling Issues • The probe server performs an RPC call using quser.exe to check for RD availability. A separate thread is executed to perform quser for each machine and is blocked until quser returns. This process eats the probe server CPU, memory, and network bandwidth. • Multiple probe servers could be implemented. • Attaching the probe server and a set of RD machines to one switched hub is a good idea. • The web site RD icons can be reduced in size for larger numbers of machines. However, there is a limiting factor due to visual representation and page design. In this case it would probably be better to break out the machine status list to its own window. • Practical AFS limits. • Use of OS virtualization software to expand RD machine count? Lee College of Engineering

28. Some Rare Problems • The remote desktop server will immediately disconnect a user just after logging on. The quser.exe utility shows the user as being logged on, however they have no processes running, and a reset session is unable to log them out. The local administrator is even prevented from logging on. The only way out of this dilemma is to kill the lowest csrss.exe PID, which will reboot the machine. • The rdsessiond service is unable to remove a users session lock file because it cannot access AFS. We have a semi-permanent workaround for this issue. • Rarely the AFS logon authenticator (afslogon.dll) will not execute during logon. This is not an AFS bug, it is a Windows remote desktop network provider execution issue. • Paged pool memory problem. Lee College of Engineering

29. Summary • For minimizing network traffic, you should update data files stored in AFS only if they change relatively infrequently (obvious), but you can go one step further by only updating the data in AFS if the new data is different than the old data. • Since the AFS client has a built in caching system with callbacks, you don’t need to double-cache your data to determine whether a change has taken place. Just use the local AFS cache. Open for read, determine if the new data is different, if so then write. • Certainly this view of using AFS to minimize network traffic is only specific to a few specialized applications, such as the one presented in this talk, however understanding this may help with future design decisions. Lee College of Engineering

30. Commentary • In many ways, for retrieving data that a client application may need, AFS can be viewed as a simplified directory, like LDAP, since it has a global namespace. Unlike a directory service, AFS can store entire files, since that is its intended purpose (something LDAP wasn’t designed for). With the ability of local caching plus file replication to multiple read-only sites, AFS can appear to be a nice alternate “directory service” solution. This is a viewpoint that I would like to popularize as missing from the mindset of most large organizations with, and especially without AFS infrastructure. • The Mosaic group has, so far, never needed a dedicated directory because we have used AFS as a directory. Our Mosaic group’s Oracle database has been configured to write data files into AFS so that our many in-house client utilities can pickup on this data. A similar method was used for the MosaicAnywhere website RDCgi script in this presentation. This saves us the need to “directory enable” our clients. Again, whether this is advantageous or not depends solely on the applications intended purpose and its “fit” into the bigger solution. Use a directory service where a directory is required, otherwise why not use AFS? Lee College of Engineering

31. MosaicAnywhere Demo with Questions and Answers E_mail: rmdyer_at_uncc.edu Web: http://www.coe.uncc.edu/~rmdyer A special thanks goes to Jeffrey Altman of Secure-Endpoints for prodding me into this presentation and especially for the excessive work that he has put into the OpenAFS Windows client this past year. Thanks Jeff! Lee College of Engineering