Dyalog’08

1. Dyalog’08

2. Technical Road Map ’08 Morten Kromberg Dyalog’08 - Elsinore

3. Looking Back: v11 Version 11.0 (Autumn 2006) • Object Orientation – Classes • New Tool of Thought • Significantly Improved Integration with .Net • Source code management using standard tools • 64-bit Version - LARGE Workspaces • Core Language Extensions • A number of new primitives: ⍣ ⌷ ∧ ∨ Dyalog’08 - Elsinore

4. Looking Back: v12.0 First Release (Win32) in February 2008 • Main Features: • Unicode Support • Journaling Component Files • Simplified, Secure Communications Toolkit (“Conga”) • Built-In Charting and Document Creation • Language Bar and On-Screen Keyboard • Release schedule: • Windows (32&64), Intel Linux, AIX/32 are available • AIX/64 will soon be released • Windows Mobile at year end • Solaris versions, other Unixes “on demand” Dyalog’08 - Elsinore

5. More About v12 (&v11): • Journaling Component Files • 16:00-16:30 Monday (Richard Smith & John Scholes) • Simplified, Secure Communications Toolkit (“Conga”) • 16:00-18:00 Tuesday (Morten Kromberg) • Built-In Charting and Document Creation • 16:00-18:00 Tuesday (Adrian Smith) • Source Code Management • 13:30-16:30 Wednesday (Dan Baronet) • Unicode Support • Workshop Sunday (Morten Kromberg) • Short Demo now! Dyalog’08 - Elsinore

6. Enhancements 1998-2008 Dyalog ’08

7. What’s Next? • Dyalog has developed at a breathtaking pace ... • Features for each release have often been researched, prototyped and released in a single cycle • This has made it hard to ensure quality and difficult to predict release dates • It is our goal to become a little more predictable – without losing momentum Dyalog’08 - Elsinore

8. Version 12.1 (2009) • Version 12.1 will contain few ”visible” changes – at least compared to recent releases • The theme is Enterprise Applications, but you don’t need to be an ”Enterprise” to benefit: • Focus on performance, scalability, quality/reliability - and making applications easier to deploy • Polishing off ”recent” features: • Better support for editing namespace and class scripts, • More work on Unicode-based source code management • In parallel, we will start research projects which may not appear until after 12.1. Dyalog’08 - Elsinore

9. ”Platform” Trends • Proprietary Unixes (AIX, Solaris, remember hp?) are ”fading” in favour of Linux • Windows remains the most important platform for our users at this point in time • 64-bit is becoming common in Unix/Linux • Even Windows users are sloooowly moving • Multi (at least 2)-core machines are everywhere • we MUST make it easy to harness them from APL Dyalog’08 - Elsinore

10. ”Application” Trends • Microsoft.Net is a strong unifying force • Open Source .Net (”Mono”) is now embedded with many Linux distributions • The differences between platforms are shrinking • Portability is as important as ever – and should be getting easier to achieve • ”Service Buses”, and open standards like WebServices, are supported on all platforms • Allows easy and secure integration of tools inside and outside of the .Net Framework Dyalog’08 - Elsinore

11. Platform Philosophy • Many of the most successful APL coders are very ”problem oriented”. They have no wish to recode to adapt to the ”IT flavour of the month”. • Rapid development... But also ULTRA long-term maintainability & portability is very important to us and to many of our customers. NO ”big bang” migrations. • APL code & data are binary compatible across platforms • Despite these constraints, good design has allowed better integration with modern platforms than competitors • Current trends (.Net/Mono) provide more, and better ways for us to achieve all of these goals Dyalog’08 - Elsinore

12. Current Research Projects • Exploiting multi-core architectures • Within the interpreter: For example, use several cores to execute a primitive function • Under control of the user (fork the application at suitable points, decided by the APL developer) • Support for writing Web Services in APL • Prototype is available, will be bundled with v12.1 • Tool to run a Dyalog app as a ”Windows Service” too Dyalog’08 - Elsinore

13. Likely Research Projects • Implement new Development Environment based on .Net/Mono • Would be easily portable to any platform where .Net or Mono is available (almost everywhere) • Can be connected to APL on a different machine / platform • Extend .Net bridge to allow APL developers to use Mono • Related to, but independant of the IDE project • Design and Implement Prototype of a Managed APL Interpreter • Based on the Microsoft.Net runtime • Would allow Dyalog components to run inside web browsers, etc • Language / interpreter research into using multiple cores • For example fuse trains of primitives into a single unit • File Acceleration Server (on the Back Burner) • Secure, Fast Component File System as a Service via TCP Dyalog’08 - Elsinore

14. New IDE and Managed APL • Managed interpreter would NOT be a replacement for the current Dyalog system: • Unlikely to be 100% upwards compatible, even at the core language level – due to framework constraints • Initially aimed at writing components to be embedded in web servers or other ”frameworks”, where managed code is a requirement • New IDE would work with both the current and the .Net interpreter • Interpreters would inter-operate as ”seamlessly” as possible • It should be easy to write and maintain systems composed of components based on both systems Dyalog’08 - Elsinore

15. Two Prongs - Always • We must make it easy to write components that can be embedded in enterprise development frameworks BUT • We must also constantly strive to make it easier for the lone entrepreneur or ”domain expert” to build a complete application without learning how to operate heavy machinery Dyalog’08 - Elsinore

16. Devt Tools • We will look at providing ”simpler” ways to use modern GUI and Web frameworks • Likely to be prototyped in APL, like the WebService framework • Should we invest in CausewayPro as a GUI design tool for ”simple” applications? • We will work on building (and collecting) general developer tools. For example, a tool for test-driven development. Dyalog’08 - Elsinore

17. v12.1 IDE Enhancements • Better support for editing classes • Make it easier to work with individual functions in a class • Some ideas prototyped - see John Daintree’s presentation right after lunch • Better access to .Net metadata • Evaluate and Autocomplete, and other ideas • (Also look at performance of .Net marshalling) Dyalog’08 - Elsinore

18. v12.1 Performance • Boolean Scans and Reductions • See Roger Hui at 14:15 today • Indexing and Indexed Assignment • Rewritten, primarily to reduce space consumption, but speed also improved for some cases • See Nic Delcros and John Scholes at 15:30 • Set membership and lookups • Will revisit algorithms for ∊ ⍳ ∩ ∪ • Have ideas for transpose and inner product • Experiments with using multiple cores to execute primitives on large arrays Dyalog’08 - Elsinore

19. Boolean Speedups (by Roger Hui) v ← randbool 6.4e6 + 1 ⍝ vector xw ← randbool 1000 100 64 ⍝ with word-multiple # of columns x8 ← randbool 1000 114 56 ⍝ with multiple-of-8 # of columns c5 ← randbool 1.28e6 5 ⍝ array with 5 columns c8 ← randbool 890 900 8 ⍝ array with 8 columns odd ← randbool 56638 113 ⍝ an odd number of columns (> 8) Dyalog’08 - Elsinore

20. Prioritizing Performance • We have ideas and some clear customer requests (like Indexing and Iota), but ... • We need your help to measure which primitives are important • Recently patched version 12.0 systems include a monitor which can be enabled using the startup parameter -Dm • The APL monitor registers CPU consumption broken down by primitive Dyalog’08 - Elsinore

21. APL Monitor Data • When started with –Dm, Dyalog will create a file called aplmon.csv at the end of the session: • With thanks to CONAC for permission to show this data Dyalog’08 - Elsinore

22. APLMON Summary Dyalog’08 - Elsinore

23. APLMON Drill Down Dyalog’08 - Elsinore

24. APLMON Summary • While ⎕MONITOR finds hot-spots in your application, APLMON finds them in the interpreter • If you send data collected while running time-critical application code, this will help us prioritise our tuning. • If your requirements don’t end up at the top of our list, you can join the ”sponsor a primitive” programme  • APLMON is available via DSS now. The analysis workspace will be available as a free download. • NB: Nice bar graphs in the session require the latest (Oct 7th) APL385 Unicode font (in the ”Conference Edition” folder on the memory stick). Dyalog’08 - Elsinore

25. Let APL Decide When to Fork • Experiment: Use multiple cores when executing scalar dyadic functions (arrays >32k elements) • Thanks to Norbert Jurkiewicz at Carlisle for testing on his quad core Windows server Dyalog’08 - Elsinore

26. Letting APL Decide... • Seems to work well on functions as expensive as floating-point ÷ • For ”cheaper” functions, we suspect that data cannot be delivered fast enough by the processors memory-management sub-system. Cores all run at 100% but don’t finish (much) faster than a single core would. • Looks interesting for large applications of functions like ÷!*⍟ on floating-point numbers. Some derived fns and idioms. Hashing. • And anything involving ⎕CT: ≤ < = > ≥ ⌈ ⌊ • We learned that measuring and understanding performance of multi-threaded applications is HARD. • Compiler-style data-flow analysis would help (turn a train of primitives into a unit), and is also HARD (but easier for dfns). Dyalog’08 - Elsinore

27. Let the User Decide to Fork • Using multiple cores within primitives is transparent to the APL programmer, so very easy to use. • However, forking and collecting on each primitive only works well in some cases. Synchronization and data loading/storing cause bottlenecks. • The average size of an array passed to a primitive is very small in most applications. • In many situations, better utilization of the processor is possible if the APL developer controls forking at a higher level. Dyalog’08 - Elsinore

28. A Notation for Forking • In addition to the implicit parallel nature of many primitives, Dyalog has two explicit forking mechanisms: • The each operator (¨): function¨data • A ”dot” to the right of an array of namespaces: ns_array.function data • In both cases, function is (logically) executed in parallel. • Due to current limitations in workspace management, Dyalog APL cannot actually make the calls in parallel. • However, imagine if we had a mechanism to generate namespaces outside the current workspace... Dyalog’08 - Elsinore

29. Demo of Model of Forking (remember to run with SHOW←1 and inspect the Demo function) nss←⎕NS¨ '' '' nss.mat←⊂¯1000+?500 500⍴2000 nss.(0⍕3 3↑mat+.×⌹mat) ⍝ show top left of mat times its inverse 1 0 0 1 0 0 0 1 0 0 1 0 0 0 1 0 0 1 (elapsed time: 18920 msec) remote←Y 2 ⍝ Y is the ”fork” function remote.mat←⊂¯1000+?500 500⍴2000 remote.(0⍕3 3↑mat+.×⌹mat) 1 0 0 1 0 0 0 1 0 0 1 0 0 0 1 0 0 1 (elapsed time: 11256 msec) Dyalog’08 - Elsinore

30. Forking Model • The right argument to Fork could be: • n: create empty namespaces • ref: copy an existing namespace or object instance • wsname: create a remote namespace from a saved workspace • The model is a function built on the Conga RPCServer example code • The APL model offers (almost) all the performance benefits, but not the elegance of interpreter support – which would put parallel execution ”at your fingertips” • We would like to explore the design thoroughly with you before implementing support in the language Dyalog’08 - Elsinore

31. Dyalog’08 Conference Edition • Most of Roger Hui’s work on boolean scans and reductions • New implementation of indexing (but not yet indexed assignment) • Parallel processing of ”large” scalar function calls, controlled by environment variables: • APL_MIN_PARALLEL (array size at which extra cores are used, default 32767) • APL_MAX_THREADS (number of cores to use, default 2) • The workspace DYALOG08 contains a function which shows off the features of the new interpreter • Warning: Do NOT use this interpreter for anything other than experiments. Enabling APLMON in it crashes in my build. Dyalog’08 - Elsinore

32. Dyalog’08 Conference Edition Dyalog’08 - Elsinore

33. Conclusion • The Release Notes for version 12.1 will be much thinner than the notes for v11.0 and v12.0 • It will be faster and stronger • It will look better, too (with GRAPL graphics and IDE enhancements). • And after that, things will get really interesting... Dyalog’08 - Elsinore

34. Other Talks by Devt. Team... Dyalog’08 - Elsinore