Impact of Cloning and Virtualization on Active Directory Domain Services

2. SESSION CODE: SVR412 Pete Calvert pete@sage-work.com @erucsbo Impact of Cloning and Virtualization on Active Directory Domain Services (c) 2011 Microsoft. All rights reserved.

3. Session Objectives and Takeaways • Session Objective(s): • Convey the technical challenges surrounding Windows & Active Directory in a virtual world • logistical and other valid concerns beyond scope for now • Highlight fundamental Windows & Active Directory concepts & assumptions • identity, replication, time, etc. • Provide an understanding of the risks stemming from virtualization • Key Takeaways: • Improved comprehension of… • core Active Directory and Windows components impacted by cloning & virtualization • best practices when virtualizing DCs and domain members • what qualifies as “successfully cloning a Windows machine” and what doesn’t (c) 2011 Microsoft. All rights reserved.

4. Windows Concepts Machine Identities

5. Computer-identity comprises… • Name • Stored locally, suffixed with a $ • IP address • Network identifier • Name/IP information stored in DNS • SIDs • What are these? • What is that for?

6. Protocol Documentation – Glossary: An identifier for security principals in Windows that is used to identify an account or a group Conceptually, a SID is composed of three parts: aSID prefix revision (1 = revision 1) + an identifier authority (5 = NT Authority) an account-authorityportion (typically the domain’s SID) principals created in the same domain share the same prefix and authority-portion an integer uniquely representing an identity relative to the account-authority commonly known as the relative identifier(RID) a 30-bit address-space (~1 billion principals per domain-lifetime) S-1-5-21-2000478354-492864223-854245397-19221 What is a SID?

7. SID assignment • Machine SIDs • How is it assigned? See [MS-SAMR], section 3.1.1.9.2 • How many individual SIDs does a computer serving as a domain-member have? • Domain SID • Where does that come from? • SID Usage • Authorization • Deployment Scenarios • Lets walk through a few potential usage scenarios that, at first glance, may appear perfectly acceptable…

8. Deployment Scenarios Pay close attention; this gets tricky…

9. Start with a domain joined machine named M1 Clone it and boot-up the clone (e.g. copy its VHD) Can the clones co-exist? What about if we “offline” unjoin the clone, rename it to M2 and join it (M2) back to the domain And now? Scenario 1

10. Scenario 2 2k8r2.VHD Windows Server 7) PEACH\Administrator logs on to a PEACH domain-member and tries to map a drive to:\\luigi.princess.peach.com\Gameboy What happens? 1)A template VHD file that is used to deploy new Windows servers is copied 4) Another copy is made of the template VHD. It is renamed to LUIGI & joined to the PRINCESS domain 2) The cloned VM is renamed & promoted to a DC creating the PEACH domain 6) The PEACH\Administrator is added as a member of CHILD\SuperMarioBros 5) CHILD\SuperMarioBros is granted READ/WRITE access to the Gameboy share on LUIGI 3) A child domain (PRINCESS) is promoted from a clean OS-install in a branch office

11. Setup a machine M1 Clone M1 to get M2 Promote both in different domains in different forests Result: 2 domains share the same SID space Establish trust between the 2 domains/forests What happens? Scenario 3 M1 & M2 promoted as first DCs in two forests M1 is cloned  M2 Trust? Forest1.com SID: S-10 Computer: M1 SID: S-10 Forest2.com SID: S-10 Computer: M2 SID: S-10

12. Create domain from machine M1 (dom1.lab) Install a new machine M2 Clone M2 to get new machine: M3 Promote M2 as a replica in dom1.lab Join M3 to dom1.lab domain hosted by M1 and M2 Anything wrong here? Scenario 4

13. Windows Concepts Active Directory Replication

14. Update Sequence Numbers (USN) • What’s a USN? • 64 Bit QWORD • Logical clock, per DC (USNs are local to a DC) • Never re-used and SHOULD NEVER rollback • When are USNs assigned? • (i.e. when does the clock tick?) • Assigned to new objects / update transaction • if transaction is aborted  USN skipped, remains unused • Independent from system time

15. Update Sequence Numbers (USN)

16. Object usnCreated=4711 ObjectusnChanged =4711 Property Value USN Version# Timestamp Originating GUID Orig. USN P1: Value 4711 1 <time> DS1 4711 P2: Value 4711 1 <time> 4711 DS1 P3: Value 4711 1 <time> 4711 DS1 P4: Value 4711 1 <time> DS1 4711 Object creation & metadata DS1 • Add new user on DS1 • DS1 USN increases to 4711 • DS1 object metadata below USN: 4710 USN: 4711

17. Object usnCreated =2052 Object usnChanged =2052 Property Value USN Version# Timestamp Originating GUID Orig. USN P1: Value 2052 1 <time> DS1 4711 P2: Value 2052 1 <time> 4711 DS1 P3: Value 2052 1 <time> 4711 DS1 P4: Value 2052 1 <time> DS1 4711 Object replication & metadata DS2 DS1 • User replicated to DS2 • DS2 USN increases to 2052 • DS2 object metadata below USN: 2052 USN: 2051 USN: 4711

18. High Watermark vector table • Table per NC per DC • Maintains • replication partners using DC’s DC-GUID • highest known USN from last replication • Used to detect recent changes on replication partners • so that DCs only replicate that which changed since the last replication cycle

19. DC GUID Highest known USN DS1 GUID 4711 DS3 GUID 1217 High Watermark vector table DS1 USN:4711 • DS4’s high-watermark vector • assumes that DS1 and DS3 are its replication partners DS4 DS2 USN: 3388 USN: 2052 DS3 USN:1217

20. Database identity • Domain Controllers are machines with machine identities • Name, SID • Domain Controllers host a database with an identity • Invocation ID, stored on NTDS Settings Object • When is it assigned/updated? • Usage of the invocation ID • Replication metadata (UTD Vector)

21. Up-To-Dateness (UTD) vector table • Table per NC per DC • Used to detect updates already received via another replication route • Maintains • originating DC’s invocation ID • highest originating USN • timestamp of last successful replication cycle • Which DCs have an entry in UTD vectors?

22. Replication timestamp Invocation ID Highest originating USN 12:02.31 DS1 GUID 4691 12:02.29 DS2 GUID 2052 12:02.36 DS3 GUID 1216 Up-To-Dateness (UTD) vector table DS1 USN: 4711 • DS4’s up-to-dateness vector • assumes that DS1, DS2 and DS3 have all originated writes against the partition DS4 DS2 USN: 3388 USN: 2052 DS3 USN: 1217

23. Making the UTD vector “up-to-date” • DC2 initiates replication from DC1 • DC1 determines what changes to send: • Local USN higher than the one stored by DC2 in its high watermark table • Originating USN higher than values in the UTD vector stored by DC2 • At the end of replication: • Increase DC2’s high watermark for DC1 to new DC1’s highest local USN • DC2’s UTD vector becomes the max-merge of DC1 and DC2’s UTD vectors

24. Lingering Objects • An object on DC1 is lingering if: • It is not present on DC2 that fully hosts the same NC • It is not “about to” be garbage collected • The creation of that object is not part of any upcoming replication cycle • in other words, USNcreated on DC1 is lower than highest exchanged USN - as stored in High Watermark Vector for DC1 on DC2 • Detection happens when DC2 receives from DC1 an update or deletion event for the object. • Events 1388, 1988 • The fact that an object is lingering doesn’t necessarily make it “wrong”

25. USN rollback • What is a USN rollback? • corresponds to the situation where a USN which had previously been allocated to an update gets re-used • Such a phenomenon breaks the strongest assumption made in our replication algorithm • Detection: • DC2’s UTD vector indicates that it has replicated all originating updates from DC1 up to USN X1 • Next time DC2 pulls updates from DC1, DC1 “thinks” that its highest originating USN is X2<X1. • Since DC1 realizes that it has previously sent out udpates with higher USN than what it’s currently using, it quarantines itself • Event 2095

26. USN rollback USN rollback detected

27. USN bubbles… how a USN rollback can turn really bad USN rollback detected USN rollback NOTdetected!

28. Improper Backup/Restore • What can go wrong with an improper backup/restore? • Summary of a real-world case: • 2500 users not able to log on • users having access to resources they should not have access to anymore • schema mismatches after Schema Master rolled back • Exchange server failing • RID pool allocated twice after RID master rolled back

29. Application – Backup/Restore • Resetting the invocation ID • Use supported backup/restore solutions • VSS writers, whether in Windows backup or 3rd party solutions • Last resort option… (and not formally tested) • before you apply the snapshot, disable the network adapters on the VM • apply the snapshot • set registry value Database Restored from Backup = 1 • reboot • verify that the DC has a new invocation ID • re-enable network adapters

30. Application – P2V migration • Is it enough to reset the invocation ID on the newly created Virtual DC? • Online or offline P2V? • Lab creation via P2V • What happens if various DCs are P2V’d at different times and placed in test network? • Recommendations: • Use P2V in SCVMM, it has a few checks in place • Reset the invocation ID • Do not place physical and P2V’d VM on same network… ever!

31. Application – RODCs • Virtualization of RODCs • Can I take snapshots of RODCs and use them? • Mostly but with various ramifications, e.g. • lastLogon and other logon-statistics-attributes written only locally on RODC • Can I clone RODCs in a branch site? • No

32. Miscellaneous Considerations TimeSync, Security, Performance, Going all virtual, etc.

33. If you have followed our existing guidance… we’ve changed our minds  documentation changes are on the way (or already published) Windows Time Service has a well-defined algorithm for time synchronization within a domain (Domain Hierarchy) let it do its thing and ensure the HyperVisor participates in the same timesync hierarchy minimizes/eliminates large deltas in time Are we suggesting you disable Virtual Machine Integration Services completely? no… absolutely NOT! Virtual Machine Integration Services are still needed, e.g. while the VM is booting or in the midst of other VM-specific operations such as Resume Instead, disable the VMIC timesync provider in the guest KEY: HKLM\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\ VALUE: [REG_DWORD] VMICTimeProvider: 0 (NOTE: that’s a zero) Time Synchronization

34. Hosts of domain controllers should be handled with same care as the DCs they host Possible EoP from host administrator to Domain/Enterprise Admin As possible, reduce attack surface on host Server Core A guest DC has admin privileges over domain members, including Hyper hosts, if joined to the domain Possibility: make the host a DC Security considerations

35. In testing conducted in a W2K8 Hyper-V environment Virtual DCs perform at about 90% compared to physical DCs Is that still true? No, virtualization technologies improve. We’re now almost at par assuming, of course, that the host isn’t running too many VMs Performance considerations

36. Key: Avoid single points of failures Same messaging for the past 10 years Do not place all your DCs on the same host we have seen this Diversify host’s hardware if possible oftentimes, this is simply not realistic, but it remains optimal nonetheless Maintain 1-2 physical DCs per domain? as above Going all virtual – a good idea?

37. Disk Write Caching (FUA) Disk write caching setting on guest is honored by the host Machines running hot Host running 5 VMs gets (too) hot and shuts down VMs Antivirus Runs on the host, “locks” VM files (cannot boot) KB 961804 Snapshots and host’s disk space What if a snapshot takes up the whole disk? What if snapshot files improperly deleted? Others

38. Recap

39. Recap • Cloning non Domain Controllers? • Perhaps, risks for 3rd-party software remain an unknown quantity • Best Practice: SYSPREP instead • Cloning Domain Controllers? • ABSOLUTELY NOT! • What if it’s the only DC in the entire forest? Still a concern: • it won’t naturally replicate • What happens to apps that understand the replication fabric, etc. • HyperV host snapshotting on Domain Controllers guests • Writeable: practically guarantees a USN rollback situation • RODCs: perhaps… but untested  the risks are undetermined • TimeSync in virtualized environments • Disable the VMIC timesync provider within the guest

40. Enrol in Microsoft Virtual Academy Today Why Enroll, other than it being free? The MVA helps improve your IT skill set and advance your career with a free, easy to access training portal that allows you to learn at your own pace, focusing on Microsoft technologies. • What Do I get for enrolment? • Free training to make you become the Cloud-Hero in my Organization • Help mastering your Training Path and get the recognition • Connect with other IT Pros and discuss The Cloud Where do I Enrol? www.microsoftvirtualacademy.com Then tell us what you think. TellTheDean@microsoft.com

41. Resources • www.msteched.com/Australia • Sessions On-Demand & Community • www.microsoft.com/australia/learning • Microsoft Certification & Training Resources • http:// technet.microsoft.com/en-au • Resources for IT Professionals • http://msdn.microsoft.com/en-au • Resources for Developers (c) 2011 Microsoft. All rights reserved.

42. © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION. (c) 2011 Microsoft. All rights reserved.