Ahead of Time Dynamic Translation

1. Ahead of Time Dynamic Translation PreJit/NGEN by any other name George Bosworth Microsoft MRE’04 March 21, 2004

2. Dynamic Translation • What is it? – a compiler (not) • What does it do? – produce good fast code (hopefully) • When does it run? – Before the code is executed. i.e ahead of time. How much before is the subject of this talk

3. Outline • Code Generation and execution in the Common Language Runtime (CLR) • PreJit • What is it, why do it, at what cost • What is it really, and what is hard vs easy • Simple example • What the resultant code could look like

4. Common Language Runtime Frameworks Common Language Runtime Base Classes ExecutionSupport Security IL to native code compilers GC, stack walk, code manager Class loader and layout

5. Runtime Control FlowJust in Time Assembly ClassLoader First reference to type IL to nativecode compiler ExecutionSupport ManagedNativeCode First call to method CodeManagers SecuritySystem CPU

6. Execution Model VB VC ... Script NativeCode PreJit timeCode Gen IL Common Language Runtime “Econo”-JITCompiler Standard JITCompiler NativeCode

7. Compiling IL To Native • “Econo” JIT • Generates unoptimized native code • Code can be discarded and regenerated • “Standard” JIT • Generates optimized native code • Includes verification of IL code • PreJit code generation • Done at install time • Reduces start-up time • Native code has version checks and reverts to runtime JIT if they fail

8. Assemblies • Unit of deployment • One or more files, independent of packaging • Self-describing via metadata (“manifest”) • Versioning • Captured by compiler • Policy per-application as well as per-machine • Security boundary • Assemblies are granted permissions • Methods can demand proof that a permission has been granted to entire call chain • Mediate type import and export • Types named relative to assembly

9. What Happens at Compile Time? • Compiler reads any imported metadata and builds internal symbol table • Compiler resolves method overloading using language-specific type matching rules • Compiler adds any required coercions or casts • Compiler selects and records a precise member reference • Compiler provides object layout information through its choice of • Layout technique • Non-static fields • Virtual methods, with “new slot” vs “use existing slot” • Compiler emits metadata by merging incoming metadata with metadata it has generated • Compiler emits IL with metadata tokens

10. What Happens at Class Load Time?, I • A type reference occurs and is resolved either to an existing type or to an assembly reference. • The assembly is loaded and some validation checks occur. • The assembly is asked to resolve the type reference and a module is located • It is at this point that the security system runs inheritance permission checks

11. What Happens at Class Load Time?, II • If no managed native code is available, the IL module is loaded and preliminary validation checks occur. • The particular class is then validated and the CLR creates in-memory data structures representing the class, including thunks for use when a method is called the first time. • When a method on the class is called the first time, the code is JITted (and verified if needed).

12. What Happens at JIT Time? • If a member reference appears in the code, the type is first loaded (see below), which will also load classes from which it inherits. • If the type does not itself have the member, the inheritance chain is searched to see if it can be resolved higher up in the class hierarchy. • The member access check, including a possible security check, occurs at this time. • The JITter may insert calls to security code for declarative security checks in addition to those specified using the imperative security mechanism. • The code runs.

13. DEVELOPMENT DEPLOYMENT Source code PE header + EH Table + Native code (incl. object layout) public static void Main(String[] args ) { String usr; FileStream f; StreamWriter w; try { usr=Environment.GetEnvironmentVariable("USERNAME"); f=new FileStream(“C:\\test.txt",FileMode.Create); w=new StreamWriter(f); w.WriteLine(usr); w.Close(); } catch (Exception e){ Console.WriteLine("Exception:"+e.ToString()); } } Compiler public static void Main(String[] args ) { String usr; FileStream f; StreamWriter w; try { usr=Environment.GetEnvironmentVariable("USERNAME"); f=new FileStream(“C:\\test.txt",FileMode.Create); w=new StreamWriter(f); w.WriteLine(usr); w.Close(); } catch (Exception e){ Console.WriteLine("Exception:"+e.ToString()); } } app. directory, Win32 directory Services • Exception Native codeEH table Unmanaged (Traditional) Code EXECUTION

14. DEVELOPMENT DEPLOYMENT Source code Assembly PE header + MSIL + Metadata + EH Table PEVerify public static void Main(String[] args ) { String usr; FileStream f; StreamWriter w; try { usr=Environment.GetEnvironmentVariable("USERNAME"); f=new FileStream(“C:\\test.txt",FileMode.Create); w=new StreamWriter(f); w.WriteLine(usr); w.Close(); } catch (Exception e){ Console.WriteLine("Exception:"+e.ToString()); } } Compiler public static void Main(String[] args ) { String usr; FileStream f; StreamWriter w; try { usr=Environment.GetEnvironmentVariable("USERNAME"); f=new FileStream(“C:\\test.txt",FileMode.Create); w=new StreamWriter(f); w.WriteLine(usr); w.Close(); } catch (Exception e){ Console.WriteLine("Exception:"+e.ToString()); } } NGEN Evidence EXECUTION Host Policy Manager Assembly info Module + Class list AssemblyLoader GAC, app. directory, download cache Policy <?xml version="1.0" encoding="utf-8" ?> <configuration> <mscorlib> <security> <policy> <PolicyLevel version="1"> <CodeGroup class="UnionCodeGroup" version="1" PermissionSetName="Nothing" Name="All_Code" Description="Code group grants no permissions and forms the root of the code group tree."> <IMembershipCondition class="AllMembershipCondition" version="1"/> <CodeGroup class="UnionCodeGroup" version="1" PermissionSetName="FullTrust" Permission request Granted permissions (class) (assembly) CLR Services • GC • Exception • Class init • Security (method) ClassLoader JIT +verification Vtable +Class info Native code+ GC table Managed Code Execution

15. PreJit: What Is It?An MSIL-to-Native Compiler (Not) • Pre-compile, pre-load, pre-layout • Validate at execution time • May factor in processor type, other installed assemblies, etc. • If invalid, use the normal execution path • Goal: faster start-up • Because of smaller start-up working set • Usually as part of assembly installation ngen.exe, ships in Visual Studio® .Net

16. What Happens at PreJit Time? • Record version information about any dependents • Classes are laid out • Internal runtime data structures allocated • Generate managed native code, verifying as needed • Emit a specially formed PE file (.exe, .dll) in the code cache and associate it with the input IL assembly

17. DEVELOPMENT Source code Assembly PE header + MSIL + Metadata + EH Table public static void Main(String[] args ) { String usr; FileStream f; StreamWriter w; try { usr=Environment.GetEnvironmentVariable("USERNAME"); f=new FileStream(“C:\\test.txt",FileMode.Create); w=new StreamWriter(f); w.WriteLine(usr); w.Close(); } catch (Exception e){ Console.WriteLine("Exception:"+e.ToString()); } } Compiler public static void Main(String[] args ) { String usr; FileStream f; StreamWriter w; try { usr=Environment.GetEnvironmentVariable("USERNAME"); f=new FileStream(“C:\\test.txt",FileMode.Create); w=new StreamWriter(f); w.WriteLine(usr); w.Close(); } catch (Exception e){ Console.WriteLine("Exception:"+e.ToString()); } } Evidence EXECUTION Host Policy Manager Assembly info Module + Class list Policy <?xml version="1.0" encoding="utf-8" ?> <configuration> <mscorlib> <security> <policy> <PolicyLevel version="1"> <CodeGroup class="UnionCodeGroup" version="1" PermissionSetName="Nothing" Name="All_Code" Description="Code group grants no permissions and forms the root of the code group tree."> <IMembershipCondition class="AllMembershipCondition" version="1"/> <CodeGroup class="UnionCodeGroup" version="1" PermissionSetName="FullTrust" Permission request Granted permissions (class) (assembly) CLR Services • GC • Exception • Class init • Security (method) ClassLoader JIT +verification Vtable +Class info Native code+ GC table NGen Code Creation DEPLOYMENT PEVerify NGEN AssemblyLoader GAC, app. directory, download cache

18. Host Policy Manager Assembly info Module + Class list Prebuilt (check Version) AssemblyLoader Policy <?xml version="1.0" encoding="utf-8" ?> <configuration> <mscorlib> <security> <policy> <PolicyLevel version="1"> <CodeGroup class="UnionCodeGroup" version="1" PermissionSetName="Nothing" Name="All_Code" Description="Code group grants no permissions and forms the root of the code group tree."> <IMembershipCondition class="AllMembershipCondition" version="1"/> <CodeGroup class="UnionCodeGroup" version="1" PermissionSetName="FullTrust" Permission request Precomputedpermissions (check) Granted permissions (class) (assembly) CLR Services • GC • Exception • Class init • Security (method) ClassLoader Vtable +Class info Prebuilt Vtable +Class info Native code+ GC table NGen Code Execution DEPLOYMENT EXECUTION GAC, app. directory, download cache JIT +verification Fixups / Relocs (or JIT)

19. Deployment Model • Shipping and installation • Compile IL in build lab • Ship IL • PreJit during installation (optional) • Possible JIT during execution • Patching • Deliver new IL image • Repair native images (all depending assemblies)

20. Why Add Complexity of PreJit? • Performance • Remove expensive post-deployment work from app startup • Signature validation • Cross-component validity checks • Type safety verification • Optimization (distribution -> execution) • Code generation • Data structures • Packing • Disk-based images allow sharing data/code across processes and across machine runs

21. Performance Gains • Warm startup time • Up to 6x faster • Private pages • Reduced by as much as 2/3 • Startup working set • Reduced by as much as 1/2 • Basis • Whidbey (the upcoming release of the CLR) • Reference WinForm app with 100s of controls Caveats: your mileage may very and these are very preliminary based on pre-release builds and hence subject to change

22. Costs • Disk space • Typical prejit image is ~2-2.5x IL size • Stored in Native Image Cache (alongside GAC) • Installation time • Typically ~3s per MB of IL • 40s to prejit Framework assemblies • Steady state performance • Slowdown due to indirections in code • ASP.net • V1: PreJit 7-12% slower • Whidbey: PreJit 2-5% slower • Bytemark • Whidbey: same performance • Future: PreJit will have better performance • Recompilation

23. Scenarios • Critical: libraries • Loaded into multiple processes • Warm startup is the norm • Maximize shared pages • Used by OS like components • Important: large apps • Large percentage of code is app code • Large impact on machine • Benefit is primarily startup time • Important: shared apps • Terminal server • Shared pages are key

24. Scenarios • Questionable: small single instance apps and addins • ~1MB working set improvement • ~15% startup improvement • Relatively small amount of compilation overhead • Likely to be numerous • Recommendation: do not ngen unless heavily used • Not helpful: transient code • Reflection emit • “Run and (probably) throw away” code • Frequently updated apps (relative to execution) • Code under development (edit-compile-debug) • Assumption: some processes will be jitting some assemblies

25. PreJit Architectural Background • IL was designed as CLR deployment format • Verifiable type safety • Small • Self describing • Portable • PreJit image was designed as internal format • Designed for execution speed • Working set rather than disk size • Data structures paged off of disk & used directly • Highly dependent on CLR implementation • Data structure snapshots • Complex codegen requirements • High rate of churn over time • Specialized for particular scenario

26. Specialization and Recompilation • PreJit images are specialized translations of an IL assembly for a particular environment • Processor • Version of CLR • Versions of assembly dependencies • Development scenario (debug, profile, IBC instrument) • Other factors • Processor features • Security policy • Hosting hooks (SQL, IE,???) • AppDomain shareable (state access) • Profiling data • Different environments require different prejit images • Different machines may have different environments • Servicing can change CLR or assembly dependencies • Environmental changes can invalidate prejit images • App-level configuration can change requirements across apps • Recompilation isneeded to adjust available prejit images over time to adapt to a changing environment

27. Client code generation motivation • IL Distribution • Verifiable (limited trust) • Small • Self describing • Specialized native code • Reach • Processor • CLR Implementation • CLR Version • Platform (assemblies) Version • Recompilation Brittleness okay Rebuilt as needed

28. Why is specialization important? • Innovation • Managed APIs and implementation • CLR implementation • Compatibility • Application needs to deploy on multiple OS’s • OS update must retain application compatibility • Plus performance (of course)

29. Framework evolution & recompilation • We want to be able to recompile application native code as frameworks (platform assemblies) evolve • Flexible compatible update rules • Add public stuff • Change any non-public stuff • Rules based on semantics – no implementation driven constraints • Rearrange and add fields to public structures & classes • Rearrange and add virtual methods • Insert into inheritance heirarchy • Non-compatible changes are possible and handled relatively gracefully • Versionability is a fundamental assumption when designing managed APIs • Implementation inheritance • Each layer up the hierarchy may add fields, methods, virtuals • All are rolled into single flat structure • All can version independently from each other and from client • Typesafe extensibility • Public fields on a structure or class • Set can be extended as needed • Example: delegates on form, properties on window • Fine grained, version-resilient component interaction • Properties • Can protect access and future evolution with a function call but still get efficiency • Raises the bar for API quality • Uniform mechanisms with language support • Patterns can be enforced with tools

30. CLR evolution & recompilation • We want to be able to recompile native code as CLR evolves • Much of runtime implementation is “baked” into code • Code sequences • Helper calls, Dispatch sequences, Global variable access • Runtime data structures • Highly specialized, Slow to build from IL, Likely to change over time • Invariants • Garbage collection, Lazy initialization • Recent Changes • Interface/Virtual dispatch • Working set tuning • Single file loading • Generics • And on and on… • Future potential • More data structure tuning • Phoenix The CLR is a great vehicle for delivering improvements into Windows

31. Future Directions • Phoenix • Changes prejit equation • Higher compile cost • Better perf for prejit vs JIT • More cross-assembly analysis • Client compilation optimizations • Profile data feedback • Active base address management • Enable some additional specialization conditionally • Allow developer to dial conservative behavior vs. aggressive optimizations

32. Why it’s Not So Easylet me count the ways

33. Issue: Compilation Overhead • Installation cost • Recompilation cost • Cascaded low level changes have very high cost • Image churn can have secondary costs (OS optimizations, virus scan) • Goals: • Ensure compilation performance is well tuned • Minimize recompilation • Minimize factors which cause recompilation • Allow limited assembly servicing without cascading recompilation • Ensure that most low-level servicing falls in this limited category • Be smart about deferring compilation work • However, some compilation cost is fundamental

34. Issue: Managed Watson Dumps • Native code and CLR data structures are generated on user machine, not available in any shipped dlls • Much of this data is required to do a stack trace • IL<->Native mapping info is typically not available unless debug code is in use • Source mapping is described in terms of IL • Goals: • Leverage repeatability of codegen to optimize ngen image contents out of process dump • Result is faster dump collection and smaller dumps • Also leverage repeatability to regenerate IL<->native mapping info post mortem

35. Cross-assembly optimizations • These optimizations expose internal implementation details to other assemblies’ code generation • Examples • Method inlining • Constant value inlining • Persisted instance data created at compile time • Class constructors • Struct enregistration • Likely to be much more in Phoenix

36. NGen Fixup Blobs • Descriptive reference to something in another module • Blobs are grouped by referenced module? • Module level table describes ranges of blobs • Each table entry has an assembly ref token + file ref token • Could roughly follow metadata signature byte encoding? • Byte sized tags with fixed following values • Variable length encoded numbers & tokens • Treat blob is a type signature? • However, scope is in destination assembly • This avoids symbolic lookups, however introduces fragility if token numbers change • Other possible blob formats • MODULE (no args) • TYPE + type signature • METHOD + <vtable index> + containing type signature • FIELD + <field index> + containing type signature • STRING + <string token> • SIGNATURE + <signature token> • Blobs are read-only? & interned within a module? • Typically described with an RVA? Details Details Details

37. NGen Fixup Tables • Set of tables which provide fixup cells for code and data to reference • Header describes ranges of tables? • Table entry initially contains tagged RVA of fixup blob? • Table index implies type of fixup • Handle (runtime data structure) • Method call address • Static field address • Varargs signature • Others? • Side effects? Which ones? (e.g. type initializations?) • Entries are fixed up lazily? • All fixups are idempotent?, tag allows atomic update • Eager update is too expensive? • Fixup can fail? And they never end

38. But when it works it can work well An example (take with a grain of salt)

39. Example Assembly B (b.cs) public class InheritFromA : A {     static public int a1()     {     ...    } } public class B {     public static void b()     {         A a = new A();         A.a();         InheritFromA.a1();     } } Assembly A (a.cs) public class A {        static public int a()     {    ...    } }

40. Example: Naive b() has 4 fixups & requires fixup guarantee before execution { A a = new A(); 20002552 mov     ecx,[20006180]       // Fixup on A's method table pointer 20002558 call    CLRStub@200040e0     // Call to new 2000255d mov     ecx,eax 2000255f mov     eax,[20006190]       // Fixup on A.ctor 20002565 call    dword ptr [eax]      // Call to A.ctor A.a(); 20002567 mov     eax,[20006194]       // fixup on A.a 2000256d call    dword ptr [eax]      // Call to A.a InheritFromA.a1(); 2000256f call    dword ptr [20005cb8] // Indirect call to InheritFromA.a1 // (InheritFromA may need one time init) } 20002575 ret    

41. Example: In the Limit b() has no fixups and can be called directly { A a = new A(); 2000254a mov     ecx,0x6ab95b98      // Note no fixup on A's method table pointer 2000254f call    CLRStub@200040e0    // call to new 20002554 mov     ecx,eax 20002556 call    a!A..ctor()         // Note direct call to A.ctor A.a(); 2000255b call    a!A.a()             // Note direct call to A.a InheritFromA.a1(); 20002560 call    b!InheritFromA.a1() // Note direct call to // InheritFromA.a1  (InheritFromA // is pre-inited) } 20002565 ret

42. In Conclusion Ahead of Time Dynamic Translation (PreJit) • What is it? – a compiler (not) • What does it do? – produce good fast code (hopefully) • When does it run? – Before the code is executed. i.e ahead of time, (just a bit earlier than a JIT).

43. Bottom Line All that is left is compiler style optimizations Prejit makes sense in some scenarios, but not all