Dyalog’09

1. Dyalog’09

2. Technical Road Map ’09 Morten Kromberg Dyalog’09 – Princeton, NJ

3. Agenda • Introducing Version 12.1 • Demonstration of some Key Features • The Next Release Cycle • Version 13.0 • Conga 2.1, Encryption & Encoding Library • “APL#” - A new APL interpreter to complement Dyalog APL • Cirrus or Cumulo-Nimbus? • Eagle’s Eye View of Curves in the Road • Recommendations on preparing for the future Dyalog’09 - Princeton

4. Version 12.1 – Overview • It was supposed to be a “Year of Consolidation” • Let the new QA manager take charge • Build, Install & QA on every platform, every night • Find and Kill more bugs, Speed things up a bit • ... But we managed to sneak a few new things into v12.1  Dyalog’09 - Princeton

5. Version 12.1 – Overview • Faster on Single Core, Some Multi-Core Features • Editor Enhancements • Fast XML-to-Array Conversion Tool (⎕XML) • Microsoft.Net components embeddable in Dyalog GUI • User Command Processor for “Tool Delivery” • IPv6 Support with Conga v2.0 • Journaling Component Files: Protection against O/S Crashes • New Developer Tools: SAWS, MildServer, LoadData, GRAPL • Applications deployable as single .exe file (interpreter+ws) • Improved Keyboard and Terminal Support • I-Beam (⌶) allows experimental & undocumented features Dyalog’09 - Princeton

6. V12.1 Core Performance Dyalog’09 - Princeton

7. V12.1 Core Performance Dyalog’09 - Princeton

8. Multi-Core Support • Most Scalar Dyadic Functions on Floating-Point arguments will use multiple cores (configurable) • But not + and -: Dyalog’09 - Princeton

9. Parallel Each Operator • A user-defined operator called PEACH is included with v12.1. It uses Conga to orchestrate multiple cores. • [Demo] • Parallel Each on a single machine works well with algorithms that have a good CPU:Memory ratio • “Distributed Each” works for any kind of calculation if transfer or arguments and results not too expensive Dyalog’09 - Princeton

10. Performance Case Study • Visited company in Bruxelles doing “actuarial calculations” • V12.1 gave a 20% speedup “out of the box” • We found 20% more by doing “hot spot” analysis for a couple of hours • An 8 core machine (2xQuad) gave a 5x speedup using “PEACH” on the inner loop of the application • Summary: 7-8x faster. 1 Hour calculation takes <10 minutes. Customer is pleased . Dyalog’09 - Princeton

11. User Command Processor • Same idea as in APL*PLUS, but based on Unicode text files as source for user commands • A “User command” consists of a single Unicode Text file, all you need to do is drop it in a particular folder (no need to restart APL) • Makes it *VERY* easy to share development tools ]display 2 2↑3 3⍴'UCMDS' 'Rock!' ┌→────────────────┐ ↓ ┌→────┐ ┌→────┐ │ │ │UCMDS│ │Rock!│ │ │ └─────┘ └─────┘ │ │ ┌→────┐ ┌→────┐ │ │ │Rock!│ │UCMDS│ │ │ └─────┘ └─────┘ │ └∊────────────────┘ Dyalog’09 - Princeton

12. IDE Enhancements • The Editor has been completely rewritten to better support editing large classes and functions • Outlining, Collapsible Regions, “Refactoring” • John Daintree will show you all the new features after lunch! Dyalog’09 - Princeton

13. Fast XML Transformations • Converts XML text to (&from) an array:[;1] Level of nesting[;2] Element Name[;3] Element Data[;4] Attributes[;5] Node type (0=Empty Tag, 1=Text only, 3=Has Children, etc…) • [;4] not required when generating XML, [;5] ignored Dyalog’09 - Princeton

14. Integrated MS.Net GUI Dyalog APL

15. Version 12.1 – Overview • Faster on Single Core, Some Multi-Core Features • Editor Enhancements • Fast XML-to-Array Conversion Tool (⎕XML) • Microsoft.Net components embeddable in Dyalog GUI • User Command Processor for “Tool Delivery” • IPv6 Support with Conga v2.0 • Journaling Component Files: Protection against O/S Crashes • New Developer Tools: SAWS, MildServer, LoadData, GRAPL • Applications deployable as single .exe file (interpreter+ws) • Improved Keyboard and Terminal Support • I-Beam (⌶) allows experimental & undocumented features Dyalog’09 - Princeton

16. v12.1 Timeframe • Complete except for packaging and a rapidly diminishing number of bugs in the new editor • Target for General Availability on Windows+AIX is August/September 2009 (was “before” Dyalog’09) • Linux and PDA before year end • It isn’t too late to join the Beta-Test programme: Write to mkrom@dyalog.com and explain what you would like to test… • Monitor http://www.dyalog.com or twitter.com/dyalogapl for notifications Dyalog’09 - Princeton

17. Dyalog APL

18. The Next Release Cycle • Conga 2.1 • Version 13.0 Dyalog’09 - Princeton

19. Conga 2.1 • Support for More Certificate Types: • We’ve had help from Peter-Michael Hager • Encrypted Certificates • The Windows Certificate Store (allows use of SmartCards and other “Tokens” that interface with this) • General available early in 2010 Dyalog’09 - Princeton

20. Likely Features for ”v13” • More Performance and QA work • Continued focus on multi-core, cache optimization and other aspects of modern chip architectures • “Industrial Strength” local and distributed PEACH • Remove “2 billion element” limit from 64-bit version • New Data Types: • 128-bit Decimal Floating Point (IEEE-754-2008) for banking applications • Complex Numbers to attract educational users • Rational Numbers / Unlimited Precision Integers can wait • We have decided we do NOT NEED 64-bit Integers Dyalog’09 - Princeton

21. Likely Features for ”v13” • Improve and rationalise internal developer tools: • ⎕DMX: Diagnostic Message – eXtended • Provide more detailed information about errors • ⎕MONITOR: Measure CPU and Elapsed Time with higher precision, easier to use, handle recursion • ⎕STACK, ⎕NSI to return “refs” • ⎕NC, ⎕NL… Rationalise “Metadata” or “Reflection” tools: Dyalog’09 - Princeton

22. More Likely v13 Features • Support for Unicode source files built-in to interpreter • Load, Save and direct Edit of Unicode Text files • General-purpose Encryption Library (Hashing, Symmetrical and Asymmetrical encryption) • Target availability for v13 in “roughly 12 months” Dyalog’09 - Princeton

23. Other Potential Projects • SQAPL Server: Allow an APL application to appear to be a Relational Database Engine (with ODBC, ADO, ADO.NET and JDBC drivers) • (This will not be a free product due to 3rd party licensing) • Remote IDE for DyalogAPL on all platforms • “Mono” Bridge – use .Net Framework under Linux • Provide More Developer Tools – as part of the APL Library. Cultivate an “Open Source” code repository, Dyalog’09 - Princeton

24. ”APL #” • We have started a research project, to produce a completely new, ”Fully Managed” APL Interpreter • We will publish a specification and discuss it, first with major customers and then ”in public” • Will be fully integrated with Microsoft.Net – but the core language syntax will be independent of .Net Dyalog’09 - Princeton

25. APL # - Benefits • Applications written in APL# will be deployable in a ”secure” fashion over the internet, and in other scenarios where ”managed code” is a requirement or provides a significant advantage • For example, you will be able to write web applications with APL code executing in the browser to provide a ”rich client” experience Dyalog’09 - Princeton L

26. APL # - ”Likely Characteristics” • Full set of APL primitives, operators and other core language features • Designed as a tool for APL thinkers – but comfortable for software engineers • ONE BIG difference: No ”semiglobals” (a function cannot see the locals of its calling environment) • There WILL be true globals • And the equivalent of namespaces will provide a ”reasonable” migration path Dyalog’09 - Princeton

27. APL # - ”Likely Characteristics” • WILL PROBABLY NOT HAVE: • Auxiliary Processors, ⎕ML, ⎕OR, ⎕SM, ⎕RTL, ⎕SD, ⎕SR, ⎕NTIE, ⎕ARBIN, ⎕W*, ⎕AI, ⎕SVO, ⎕DQ, ⎕D.. • (Library functions for which a perfectly good equivalent exists in the .Net framework) • Saved workspaces will not be able to have an )SI stack • First versions may not be able )SAVE workspaces at all • WILL PROBABLY HAVE: • ⎕CT, ⎕IO (jury is still out), and a small set of carefully selected library functions, Component Files, ⎕XML (etc) which provide unique value to APL programmers Dyalog’09 - Princeton

28. ”Dyalog” vs ”APL #” • APL# is a complimentary product which allows the use of APL in places where it can not be deployed today • It is useful to organizations who need managed code, or use Visual Studio / C# or are otherwise heavily invested in Microsoft.Net • The rate of development of the core product will continue to accelerate • The two products will inter-operate seamlessly Dyalog’09 - Princeton

29. Dyalog → APL # ”Interop” • APL# assemblies can be called by Dyalog APL (as any other .NET assembly) • Dyalog and APL# will (eventually) be able to share Component Files and communicate (many) APL arrays via TCP/IP • But not saved workspaces • Selected New Language Features in APL# will be ported to Dyalog APL • Will allow the writing of code which can be used in both environments Dyalog’09 - Princeton

30. Timelines • APL# is still being designed, but seeC:\aplsharp\silverlight\aplsharp.html • We will start reviewing the design externally this winter (come to APL2010 to heckle) • A prototype may be available at APL’2010 – earliest ”General Availability” in 2011. • Pressure (and funding ) from customers could make us more ambitious about the features, but probably not the timing, of the first release Dyalog’09 - Princeton

31. Cirrus or Cumulo-Nimbus? • A good home for ”Vapourware”? • Dave Thomas is about to teach us how to recognize and use different forms of clouds… Dyalog’09 - Princeton

32. Amazon Web Services (AWS) • A ”mature” provider of ”web and computing infrastructure” (since 2006) • You provide a virtual machine image and Amazon ”scales it infinitely” at reasonable cost • We are investigating providing tools to support EC2 (Elastic Compute Cloud), S3 (Simple Storage Service) • Goal is to provide recipes, or pre-configured ”Amazon Machine Images” to build upon • We have a lot to learn, but hope to wrap our new knowledge up nicely for Dyalog users • No timeline... Talk to us if you have a potential app. Dyalog’09 - Princeton

33. Thoughts aboutthe Road Ahead Dyalog’09 - Princeton

34. Looking Back Very Briefly Version 11.0 (Autumn 2006) • Object Orientation – Classes • New Tool of Thought • Significantly Improved Integration with .Net • Source code management using standard tools • 64-bit Version - LARGE Workspaces • Core Language Extensions • A number of new primitives: ⍣ ⌷ ∧ ∨ Dyalog’09 - Princeton

35. Looking Back: v12.0 First Release (Win32) in February 2008 • Main Features: • Unicode Support • Journaling Component Files • Simplified, Secure Communications Toolkit (“Conga”) • Built-In Charting and Document Creation • Language Bar and On-Screen Keyboard Dyalog’09 - Princeton

36. Looking ”Back”: v12.1 October 2009, Key Features • Faster, more Robust • Multi-Core Support • Enhanced Editor • XML, Web Services • Tutorials and Tools – and more to come Dyalog’09 - Princeton

37. To Come ... • New “Commercial” and “Technical” Data Types • More Speed – Especially On Multiple Cores • Encryption and Other Web Enabling Tools • Managed Code Interpreter • More Ways to Run APL as a Service across Multiple Platforms (ODBC Data Source etc) Dyalog’09 - Princeton

38. Recommendations • 64-bit is Becoming Common, even on Windows • Good news: Porting to 64-bit requires few code changes (once all the other components you need are available in 64-bit) • Unicode • Please investigate what it will take to migrate • No hurry – but eventually, we will start adding new features in Unicode only (we’ll need more symbols ) • Multi (at least 2)-core machines are everywhere • Look at PEACH and other forms of ”multicore” splits Dyalog’09 - Princeton

39. Recommendations • To prepare for the Web or Cloud - think about addding: • Web Front Ends using ASP.NET, MildServer etc • Web Services using ASP.NET or SAWS • An ODBC or ADO frontend using the SQAPL Server? • Take a look at Source Code in Unicode Text Files • Easier sharing and management • Access to many excellent (often free) tools! Dyalog’09 - Princeton

40. Other Talks by Devt. Team... Dyalog’09 - Princeton

41. Conclusion • We’ve been keeping busy! • We’re starting to enjoy the benefits of being a significantly larger team. • We hope that you will enjoy some of the new Features - and Products - that are heading your way! • Thank You For Listening! Dyalog’09 - Princeton

42. Other Talks by Devt. Team... Dyalog’09 - Princeton