UNIX Portability

1. UNIX Portability Brad McCusker Transition Engineering and Consulting HP

2. Agenda • What is UNIX Portability (UP)? • UP features in V8.3 • Proposed future UP features • UP success story

3. What is UNIX Portability?

4. What is UNIX Portability? • An initiative to ease the porting of applications from UNIX and Linux to OpenVMS • New UNIX-like APIs and utilities • Integrated operating system support for UNIX features • Heavily utilized by HP and partners to develop layered products for OpenVMS • Used by HP to improve Java performance

5. Goals of UNIX Portability • Provide a full set of UNIX interfaces and tools within OpenVMS • In native, integrated fashion • No layered emulator (e.g. old “POSIX for OpenVMS” product) • No performance issues • No interoperability issues • Make the cost of porting to OpenVMS equal or comparable to porting from one “flavor” of UNIX to another (HP-UX, Solaris, AIX, Linux) • Retain native OpenVMS behavior as the default

6. Various terms – all mean the same… • UNIX Portability is the original term used to describe this effort • The effort quickly evolved to include: • Linux Interoperability • Linux Portability • Open Source Interoperability • Open Source Portability All these are part of UNIX Portability

7. Components of UNIX Portability • C Run-Time Library (C RTL) • GNV • Other components

8. C Run-Time Library (C RTL) • Library of APIs • Implementations of routines typically found on UNIX • Examples: printf(), open() • Open Group Base Specification system interfaces • Curses screen management functions • Socket routines (not documented in the C RTL Reference Manual) • Delivered in shareable image DECC$SHR.EXE (and DECC$SHR_EV56.EXE)

9. GNV • GNV = “GNU’s Not (Open)VMS” • UNIX-like shell (“bash”) • Utilities and tools • Examples: cp, ls, vi • Open Group Base Specification shells and utilities • OpenSource software • http://gnv.sourceforge.net

10. GNV – Current Status • GNV 2.1 Shipped with VMS 8.3 • Reportedly has problem with some configure scripts • Reportedly has installation problems • Reprotedly has removal problems • GNV 1.6 – found at http://h71000.www7.hp.com/portability/GNV.html

11. Other components • Java • Perl • POSIX threads (pthreads) • Ant • Python • Pkg-config • Base operating system and RMS changes

12. UP features in V8.3

13. UP features in V8.3 • C RTL features • GNV utilities • Other features

14. V8.3 C RTL Features • Symbolic links and POSIX pathname processing • Byte-range locking • Encryption routines • Other changes

15. What is a symbolic link? • A symbolic link is a directory entry that associates a name with a text string • The text string is interpreted as a POSIX pathname when accessed by certain services • It is implemented on OpenVMS as a file of organization SPECIAL and type SYMBOLIC_LINK • Symbolic links are also known as “Symlinks”

16. Example of symbolic link creation • New DCL qualifier /SYMLINK for CREATE • Example: $ sho def SYS$SYSDEVICE:[MCCUSKER.WORK] $ create/symlink="a/b.txt" link_to_b.txt $ dir/date link_to_b.txt Directory SYS$SYSDEVICE:[MCCUSKER.WORK] LINK_TO_B.TXT;1 -> a/b.txt 6-APR-2005 08:13:29.83 Total of 1 file. $

17. Example of symbolic link access • Assume file being referenced does not exist: $ type LINK_TO_B.TXT %TYPE-W-OPENIN, error opening SYS$SYSDEVICE:[MCCUSKER.WORK]LINK_TO_B.TXT;1 as input -RMS-E-ACC, ACP file access failed -SYSTEM-F-FILNOTACC, file not accessed on channel $ $ create/dir [.A] $ $ TYPE LINK_TO_B.TXT %TYPE-W-OPENIN, error opening SYS$SYSDEVICE:[MCCUSKER.WORK]LINK_TO_B.TXT;1 as input -RMS-E-ACC, ACP file access failed -SYSTEM-F-FILNOTACC, file not accessed on channel $

18. Example of symbolic link access (continued) • Create the missing file through the link: $type LINK_TO_B.TXT %TYPE-W-OPENIN, error opening SYS$SYSDEVICE:[MCCUSKER.WORK]LINK_TO_B.TXT;1 as input -RMS-E-ACC, ACP file access failed -SYSTEM-F-FILNOTACC, file not accessed on channel $ $ create link_to_b.txt This text is in file [.a]b.txt Exit $

19. Example of symbolic link access (continued) • Now we can type the file through the link: • In this example, RMS noticed the input file was a symbolic link, read its contents and interpreted those contents as a POSIX pathname $ TYPE LINK_TO_B.TXT This text is in file [.a]b.txt $ DIR [.a] Directory SYS$SYSDEVICE:[MCCUSKER.WORK.A] b.txt;1 Total of 1 file. $ type [.a]b.txt This text is in file [.a]b.txt $

20. RMS support for symbolic links • sys$open() operates on the target file pointed to by the symbolic link • sys$create() creates the file pointed to by the symbolic link • sys$search() returns the DVI and FID of the target file; DID is zero; resultant name is that of the symbolic link and not the target file • Setting flag NAML$V_OPEN_SPECIAL cause sys$open() and sys$search() to not follow the symbolic link

21. C RTL support for symbolic links • Six newly documented APIs: • symlink() -- create a symbolic link • readlink() -- read the contents of a symbolic link • unlink() -- delete a symbolic link • realpath() -- return a direct pathname from the root • lchown() -- change the owner of a symbolic link • lstat() – return attributes of a symbolic link • Other APIs that accept pathnames recognize symbolic links

22. POSIX pathname processing • To provide a consistent programming environment, developers must be able to use POSIX pathnames through OpenVMS interfaces such as the C RTL and system services • Other standard POSIX features (system-wide root, mount points, current working directory, version limits) must also be provided • Rules must be devised to deal with the differences between POSIX pathnames and OpenVMS file names

23. POSIX pathnames for RMS and DCL • Issue: The POSIX name-separator ‘/’ character has a different meaning to DCL (as a qualifier indicator) • Quoting a pathname allows us to pass the pathname through DCL (since quoted strings are already allowed in DCL for DECnet) • Adding a prefix to the pathname allows RMS to recognize the string as a POSIX pathname • Format: “^UP^pathname” • Example: a/b.txt becomes “^UP^a/b.txt”

24. DCL POSIX pathname example $ dir [.a] Directory SYS$SYSDEVICE:[MCCUSKER.WORK.A] B.C;2 Total of 1 file. $ cc "^UP^a/b.c"/obj="^UP^a/b.obj“ $ $ link/exe="^UP^a/b.exe" "^UP^a/b.obj“ $ $ dir [.a] Directory SYS$SYSDEVICE:[MCCUSKER.WORK.A] B.C;2 b.exe;1 b.obj;1 Total of 3 files. $

25. System-wide root • Available for use with POSIX pathnames • New ROOT keyword for SET • Example: $ set root sys$sysdevice:[000000] %SET-I-PSXROOSET, system POSIX root set to SYS$SYSDEVICE:[000000] $ sho root DISK$ALPHASYS:[000000] $ $ ty "^UP^/mccusker/work/link_to_b.txt" This text is in file [.a]b.txt $ $ ty sys$sysdevice:[mccusker.work]link_to_b.txt This text is in file [.a]b.txt $

26. Mount points • Allows the crossing of volumes from the root • New mnt and umnt utilities • Example: • $ dir dkb100:[newtest] NEWDIR.DIR;1 Total of 1 file. $ mnt dkb100:[newtest] /a/mnt $ dir DKB0:[TEST.A.MNT] NEWDIR.DIR;1 Total of 1 file $

27. Current working directory • Similar to OpenVMS default directory • May not be a search list • Must exist • Example: • $ SET DEFAULT “^UP^/a/mnt” $ SHOW DEFAULT DKB100:[NEWDIR] $

28. C RTL and GNV • The C RTL and GNV will accept pure POSIX pathnames (no need for the ^UP^ quoted format) • This is an alternative to the standard C RTL UNIX pathname features • Post V8.3, file versions will be treated as on UNIX • only one version of a file will be deleted on a delete operation • no new version of a file will be created on a create operation

29. C RTL and GNV (continued) • DECC$POSIX_COMPLIANT_PATHNAMES controls how input pathnames are interpreted • 1 = UNIX-only • 2 = leans UNIX (unless pathname contains brackets or ends with a colon and passes sys$filescan()) • 3 = leans OpenVMS (unless pathname contains a slash) • 4 = OpenVMS only • Modes 1 and 4 are not recommended due to interactions with other shareable libraries and utilities

30. File naming • POSIX allows filenames “a” and “a.” in the same directory • A file created through GNV and the C RTL that does not have a “.” or that ends in a “.” will have an additional “.” appended to its name to ensure uniqueness • To allow POSIX filename “a.DIR” and directory “a” to co-exist in the same directory, GNV and the C RTL will append a “.” to a filename ending in “.DIR”

31. Byte-range locking • Implemented in the C RTL • fcntl() API • F_GETLK, F_SETLK, F_SETLKW options • Can be used against any file type • Accessed via a file descriptor • Functional across processes and across a cluster • Locks are advisory; to be effective, processes must agree to cooperate • 4 GB limit • New privileged image: DECC$SHRP

32. Encryption routines • crypt() • Encrypts an input string • Note: algorithm has nothing to do with OpenVMS password encryption • setkey() • Sets an encoding key to be used with encrypt() • encrypt() • Encrypts an array in-place using key generated by setkey() • Note: algorithm has nothing to do with OpenVMS password encryption

33. Other C RTL changes • fchmod() • New function that changes the mode of a file that is specified with a file descriptor • confstr() • Existing function that returns system configuration information • New symbols supported include: • _CS_MACHINE_IDENT • _CS_PARTITION_IDENT • _CS_MACHINE_SERIAL

34. GNV utilities • Better handling of pipes and subprocesses in bash • Behavior of cc is now more UNIX-like, particularly for certain config scripts • Improved script processing • make files run much better

35. Other features • NFS improvements • Perl has been updated and made POSIX-aware (Note: this kit is not in general release yet) • terminate() can now be called from any thread in a C++ program (previously, terminate() was only callable from the default thread)

36. Proposed Future UP Features

37. Shared-stream I/O • Shared-write access to stream files • Standard UNIX I/O default • Applicable across the nodes of a cluster • Can be used on any stream file type • Accessible via the C RTL • Set per file by an option on file open APIs • Set per process by C RTL feature switch

38. Shared memory • Implemented via the C RTL • POSIX shared memory • shm_open(), shm_unlink() • System V shared memory • shmat(), shmctl(), shmdt(), shmget() • Possibly ftok() • Not applicable across nodes of a cluster

39. ioctl() enhancements • Support more than socket operations • For terminal I/O, allow read termination when a given number of characters is entered (TCGETA, TCSETA) • For UDP sockets, I_SETSIG (signal on data in) • For ethernet, SIOCGIFMTU (check MTU size)

40. UNIX Domain Sockets • Sockets within a node • Uses socket APIs • Within a node • Doesn’t use the IP stack • Alternative to pipes • Bi-directional

41. Non-socket poll()/select() • poll/select currently only work for TCP/IP sockets • Extend to work with pipes, devices, etc

42. Child process survival • Ability of a child process to survive termination of its parent • Console shareable between a parent and its detached child • Job logical name table and mounted volume list shareable and would persist after parent process termination • Used by nohup in bash, fork(), cron

43. Semaphores • Implemented via the C RTL • POSIX semaphores • sem_open(), sem_post(), sem_wait(), etc. • System V semaphores • semop(), semctl(), semget() • Thread-aware

44. Upgrade BASH • Current BASH in GNV is based on v1.14.8 • Upgrade to current version of BASH • V3.* is current • Gets latest functionality • Easier to keep up with bug fixes

45. UNIX Portability Roadmap 2010 2009 2008 2007 2006 • OpenVMS V8.2Symbolic Links SDK • CRTL APIs • DCL support • NFS support (in TCP/IP ECO) • OpenVMS V8.x • Ongoing enhancements • OpenVMS V8.3 • Symbolic Links • Byte range locking • POSIX Pathname SDK • Support for OpenVMS 8.2-1 and 8.3 • GNV update • OpenVMS V8.4 • POSIX style pathname support • Semaphores • Shared write for stream files • ioctl() • GNV update • Shared memory APIs

46. References • OpenVMS web site: http://www.hp.com/go/openvms • UNIX Portability http://h71000.ww7.hp.com/portability/index.html • CRTL Reference Manual http://h71000.ww7.hp.com/doc/83final/5763/5763pro.html • GNV Page – OpenVMS http://h71000.ww7.hp.com/portability/GNV.html • GNV Sourceforge http://gnv.sourceforge.net/

47. Contact info • Paul Cerqua (UNIX Portability Leader) • Paul.Cerqua@hp.com • Gaitan D’Antoni (OpenVMS Technical Director) • Gaitan.Dantoni@hp.com • Leo Demers (UP Program Manager) • Leo.Demers@hp.com