690 likes | 765 Views
Welcome to this TechNet Event. We would like to bring your attention to the key elements of the TechNet programme; the central information and community resource for IT professionals in the UK: FREE bi-weekly technical newsletter FREE regular technical events hosted across the UK
E N D
Welcome to this TechNet Event We would like to bring your attention to the key elements of the TechNet programme; the central information and community resource for IT professionals in the UK: • FREE bi-weekly technical newsletter • FREE regular technical events hosted across the UK • FREE weekly UK & US led technical webcasts • FREE comprehensive technical web site • Monthly CD / DVD subscription with the latest technical tools & resources • FREE quarterly technical magazine To subscribe to the newsletter or just to find out more, please visit www.microsoft.com/uk/technet or speak to a Microsoft representative during the break
Terminal Services Server-Centric Computing Now and in the Future
Welcome • Agenda 09:00 – 09:30 Registration 09:30 – 11:30 Supporting Terminal Services 11:30 – 11:45 Coffee 11:45 – 12:45 Terminal Services Roadmap • Facilities Phones off Toilets in lobby
The PSS View: Supporting Terminal Services Sasha Loncarevic Critical Problem Resolution EMEA GTSC Platforms Support
Agenda • Microsoft GTSC Support Processes • Support Statistics for Terminal Server • Common Issues • Licensing • Scalability • Profiles • Printing • Summary • Questions
Support Processes in EMEA GTSC • WinSE (sustained Engineering) • Part of Product Group • Instrumentation • Close collaboration with developers and PMs • Developers and WinSE are on aliases too • Support Specialist • Incident database, KB (external, partner and draft), Solution Objects (internal) • International/Local Aliases • Collaboration and mentoring • Escalate to CPR/EE • CPR/EE • In-depth troubleshooting • Source code & Build Environment • Debugging and Instrumentation • Request for Collaboration/Hotfix with WinSE
Support Statistics • Terminal Services calls are categorised in two main areas • Licensing • All others • Many sub-categories also coded • by engineer when closing the case • e.g. “\Printer Redirection\Driver Comatibility\Misconfiguration” • Results are analysed by Volume and Labour Time (Minutes per Incident) • Used to drive product development and documentation
TS Other TS Licensing Non TS Support in 2004 • Windows 2000 • ~10% of total Windows Server volume • ~5% of total Windows Server labour • Licensing accounts for half of volume, a third of labour • Windows 2003 • ~15% of total Windows Server volume • ~7% of total Windows Server labour • Licensing accounts for 75% of volume and labour • Terminal Services Licensing is in the top 10 by labour Windows 2000 Windows 2000 Windows 2003 Windows 2003 Volume Labour Volume Labour
Support Trends • Volume increasing for Windows 2003 • Terminal Services improvements in Windows 2003 driving deployment • Decreasing MPI for Windows 2000 and Windows 2003 • Bugs exposed by modern scenarios being ironed out • Building a Hotfix stockpile • Symptoms matched quickly • Up to a third of non-Licensing calls resolved with an existing Hotfix
EMEA Virtual Team Experience • Licensing • Connections denied, or licenses not tracked • Scaling • Memory limitations • Old 3rd party drivers • Deadlocks and hangs • Profiles • Incomplete profile upload/download • Printing • Printers not mapped or jobs not printed • Application compatibility • Lose profile settings
Licensing - Overview • No licensing required in grace period • 90 days Windows 2000 • 120 days Windows 2003 • Grace ends when Licensing Service is discovered • First contact obtains certificates to check CALs
Windows Terminal Services Server running Licensing Service The Grace Period
Finding the License Service • Discovery is done • Windows 2003 • In the background every hour until an in-site License Service is found • Windows 2000 • In the background every 15 minutes until a License Service is discovered thereafter every 2 hours • Or whenever a License Service is required • Upgrade or issue CAL • Secure Licensing option in Windows 2003 • Configured via Group Policy • Local “Terminal Server Computer” group
Finding the License Service • Registry override • Multiple servers in Windows 2003 • Check AD for site licence object • Created during install Licensing Service If Enterprise mode selected User modifiable (add servers) • Contact each and cache (EnterpriseMulti) • Check each DC (in site for Windows 2003) • Contact each and cache (DomainMulti) • Broadcast
License Service Discovered • Each device needs a CAL • Except with Internet Connector (Windows 2000) or Per User mode (Windows 2003) • Upon first connect, client offers a unique hardware ID • Terminal Server gets CAL from Licensing Service • CAL is uniquely identified by client hardware ID and stored in Jet db • CAL passed back to client and stored in its registry • Client offers CAL at subsequent connections • Terminal Services validates the CAL • No CAL and no License Service available = no session • License Server issues temporary CALs until Activated
Windows Terminal Services License Service Not Activated Terminal Services Licensing Service
Activating the License Service • User Interface for connection to Clearing House (CA) • Web Browser, telephone, fax, Internet • Activated Windows 2003 License Service issues • Builtin CAL for Windows 2000/Windows XP clients • Windows 2000 Terminal Services only • No Expiry, Unlimited pool, A02-5.00-EX • Temporary CALs for Windows 2000 or Windows 2003 • 90 days expiry, not renewable, A02-5.00-S or A02-5.02-S • Full CAL for other clients (if CAL pak installed) • 52-89 day expiry, A02-5.00-S or A02-5.02-S • Temporary CAL issued at first connect, marked after logon, and upgraded to full CAL on next connect • Prevent Denial of Service, and stealing of CALs
Windows Terminal Services Activated License Service Terminal Services Licensing Service
Additional Considerations • Lost license is ‘found’ via hardware ID • Some thin clients do not store license • Citrix caches licenses on Terminal Servers • Metaframe XP replicates to all farm servers • Deleting entire Licensing store (registry) on client causes new hardware ID generation • Client renews CAL 7 days before expiry • Expired CALs returned to pool every 24 hours
Problems • Multiple CALs for one client name • Cannot connect • Cannot discover License Service • Cannot upgrade temporary CAL • Backing up, moving License DB
Troubleshooting • Eventlog (MPS Reports) • Resource Kit • LSREPORT • LSVIEW • TSCTST • PSS Tools • LSDIAG • TSTST • Checked version of LSERVER.EXE • TermSrv tracing & additional instrumentation
Troubleshooting Connection • Also for Remote Desktop • Ping target • telnet target 3389 • Check firewall if no connect but can ping • Delete local license & retry
Recommendations • Implement backup LS (no licenses) • Not ideal for certain cases e.g. Metaframe 1.8 • Policy module is complex, some cases result in no connection • Enterprise LS if possible, manual edit • Verify before deployment • TSCTST • LSVIEW • LSAdmin • Review KBs and Whitepapers 822134 - The function of Terminal Server CALs in Server 2003 823313 - Server 2003 TS licensing issues and requirements
Scalability • Number of user sessions limited • Poor Performance • Errors during certain operations • New applications or sessions fail • Session or Server Hangs • How many sessions can a server host?
Scaling – Windows 2000 • Tested in conjunction with NEC • Tests use classes of worker • Data Entry Dedicated (3Mb), Data Entry (4Mb), Knowledge (9Mb), Structured (10Mb) • Knowledge worker types a page in Word, responds to an email, creates a small spreadsheet, uses IE • Data Entry Worker connects to and uses SQL via simple app http://www.microsoft.com/windows2000/techinfo/administration/terminal/tscaling.asp
Scaling - Windows 2000 (NEC) • 500MHz procs used – now Xeon 3.2GHz with HT not uncommon • Windows 2003 uses HT better; Windows 2000 suffers slightly • Scripted: No indication of ‘usability’
Scaling - Windows 2003 • From product documentation on Microsoft.com • Users classed as Light (10Mb), Power (21Mb) and Structured (10-21Mb) • Recommended RAM = 128 + user RAM as above • CPU & RAM scale linearly… • Depends on bus technology etc • No users/CPU recommendation • Test, test, test
Scaling – Windows 2003 • Scalability Whitepaper • Tests use more modern hardware • Two worker classes • Data Entry Worker (3.5 Mb) • Knowledge Worker (9.5Mb) http://www.microsoft.com/windowsserver2003/techinfo/overview/tsscaling.mspx
Windows 2000 vs Windows 2003 • When kernel address space limits, Windows 2003 is much better • Otherwise, Windows 2003 still better • MM/OS design improved • 64bit exciting • Hardware will limit • Until recent CPU performance improvements, 64bit did not offer much
Limiters • Obvious : Hardware • CPU, Disk, RAM, Network • Less obvious : Software and Operating System • Address space limitations (2Gb Kernel, 2Gb User) • Redirector and other driver design • Registry size/contention • Applications • Polling (keyboard or otherwise) • Not multi-user friendly design • …most of these can be tuned
Common Problems • Poor performance • Hardware limitations • Registry contention • Old client technology (UI needs accelerator) • Logon and/or network file access slow (>15secs) • Redirector current commands, Server workitems • KB232476 • Missing icons, user32.dll errors • Desktop Heap or session pool • Application or session start failures • memory space limitations • PagedPool 160Mb-380Mb (Windows 2000 registry hives) • NonPagedPool 128Mb, double in Windows 2003
MemoryMap ffffffff ffbe0000 Non-paged poolSystem PTEs eb000000 Paged Pool (160Mb) 4Gb e1000000 System Cache Win32k & GDI drivers 8Mb Kernel Mode c1000000 Process page tables etc c0000000 2Gb Session View (desktop heaps) 20Mb Xtra PTE/cache User Mode a4000000 Win32k space, session space 0k a0000000 Session Pool 16Mb Kernel code + initial Non paged pool 80000000
Troubleshooting with Perfmon • CPU • >75% usage • Q length>1 sustained • Process • Identify spinning or leaking process • Private Bytes or Handle Count growing • Memory • Page Faults/s >10000 • Available memory <10Mb • Disk Q Length >1 sustained • Paged Pool, Non-Paged limits
Troubleshooting with Perfmon • Physical Disk • Q Length > 1 • Thrashing due to paging, fragmentation • Redirector (on Terminal Server) • current commands>20 • Server Work Queues (on File Server) • Available WorkItems <10 • Resources dynamically allocated • Workitems < MaxWorkItems • Connections created asynchronously
Demo • Performance Monitor • Redirector Current Commands • Server Workitems
Other Troubleshooting • Network trace analysis • Repeated packets • Large RoundTripTime • User-mode Debugging • Kernel Debugging (forced dump or live) • Find deadlocked threads • Examine memory usage • OEM tools • dheapmon.exe, userdump.exe
Recommendations • Test scalability • Roboclient in Resource Kit • Performance Monitor • Sweetspot may be 4-way CPU with 2Gb • If task takes 10% longer than on empty server, server is reaching saturation • Run User Acceptance Tests under desired load • Scale out not up • Address space limitations, until 64 bit • RAM and disk are cheap • <=4Gb, no PAE • Pagefile & OS on separate/multiple spindles/controller
Recommendations • Rethink folder redirection & run apps locally • Set up and test Memory Dump • Problems are more complex since hardware allows greater loading • Collect and store baseline performance data
Profiles – Overview • User configuration, settings and files • Per machine persistence • Cached locally in c:\documents and settings\username • Optionally roaming, mandatory or temporary • PKI implications • Stored on file server for roaming • Separate profiles available for Terminal Services (not at console)
Establishing Profile Location • WINLOGON/MSGINA handles logon & obtains normal profile path from user account • Set via UI, TSPROF, or scriptable Windows 2003 WTSADMIN • If not at console then check GPO settings • Force local • Override path (append with %username%), not for mandatory • Otherwise TS profile path retrieved via SAM API • RPC over SMB has firewall implications • Log event & optionally deny logon if SAM calls fail • If profile path is still blank, use default profile • “\\logonserver\Netlogon\Default User” or • local “\documents and settings\Default User”
Profile Load • Userenv has 10 threads to copy profile • Each file is copied to a prfxxx.tmp, target is deleted & temp file renamed to target • ntuser.dat and usrclass.dat (Classes key, non-roaming) loaded into HKEY_CURRENT_USER • User policy applied • In foreground i.e. before desktop • background (90+- 30 minutes) • Loopback, if configured
Profile Unload • Unload ntuser.dat, usrclass.dat • Retry 60 times, once per second • Copy files (as per load) to roaming store • Remove cached copy if DeleteRoamingCache enabled • TermSRV waits 180s for completion of session close
Profile Problems • Profile load/unload failures • Registry key in HKCU in use (unload) • File copy fails (file in use or other error) • Usually AntiVirus • Redirector timing • Profile path is blank or not collected • Password change code defect in Windows 2003 (KB833409) • >=SP4 provides single retry with DC rediscover • SAM 2048 handle limit (Windows 2000 DC)
Profile Problems • XForestLogon (Windows 2003, Win2k >=SP4) • Prevent roaming profile and policy from foreign domain • Profiles shared between NT4, Windows 2000, Windows 2003 and Windows XP • ProtectedStorage one time migration
Troubleshooting Profile Problems • Eventviewer • Userenv events in Application log • userenv debug logging • userenvdebuglevel = 0x10002 (HKLM\...\Winlogon) • Make userenv.bak read-only • UPHCLEAN • Identifies process holding registry key • SPOOLSV.EXE usually implies printer driver • Inspect HKEY_USERS • Each HKCU is loaded as user SID
Demo • UPHCLEAN • Profile unload
Recommendations • Use UPHCLEAN • Monitor event log for problems • Force local profile or use mandatory where possible • Monitor disk space and cached profiles • DeleteRoamingCache has overhead • Keep profiles small when using roaming • Hybrid profile solution – beware PKI
Redirected Printing • Seamless experience • Default printer on client is default in session • Print on Word in session, job comes out of local printer • Enabled by server, per user, or by client • At connect time, client printers are mapped into user’s Terminal session • Jobs printed in session are spooled/rendered on Terminal Server • Raw data conveyed to client