User Group Conference 2012 Advanced Tech Track

1. User Group Conference 2012 Advanced Tech Track 1

2. Advanced Tech Track (2012) • Introductions • Course Materials • Discuss Agenda • Advanced Tech • Print Automation & Document Management • System Administration Fundamentals (Part 1) • System Administration Fundamentals (Part 2)

3. Introductions • Ron Motley (Host) • CTO/CIO • Director ASP Operations • Chief Programmer/Analyst • Older than I sound, Younger than I look…  • And you are … • Managers • Administrators • Engineers

4. Course Materials • Handouts • Have placed them on the Flash-Drives. Providing notepads instead. • Flash-Drives • Presentations • Supporting Documents • Tools & Utilities

5. Agenda • Print Automation & Document Management • Mon (3pm – 4:15pm) • System Administration Fundamentals (Part 1) • Tue (1:15pm – 2:15pm) • System Administration Fundamentals (Part 2) • Tue (2:45pm – 4:00pm)

6. Print Automation & Document Management • Reform Forms Management Solution • What is “Reform” • Fabsoft Partnership • What does Reform Cost • Hosted Solution • Forms Modification • More Q & A

7. What is “Reform” • Reform is an electronic Forms application intended to replace pre-printed Forms • Reform is a third party software product from Fabsoft Inc. that interfaces with the Élan Software • When generating a form from Élan (for example, an Invoice or Statement) you can choose to use an electronic form. If you do so, then instead of loading a pre-printed form in to your printer, the electronic form is overlaid on to the data from Élan E-mail Fax Print Archive

8. Fabsoft Partnership • Variable Data Printing (VDP) • Spooler Separated From Designer • Plug-In Architecture for all Options • PDF Capabilities Embedded • Domain Support • Reform Dashboard • End User - Will see on a webpage their documents and status of their documents • Help Desk View - Will Allow the help desk personnel to filter the activities of the end user such as a failing device causing the end user's documents to not process correctly • Health Monitor • Will monitor all aspects of the solution and will alert anyone if something goes wrong with the workflow

9. What does Reform Cost • There are three versions of Reform • With the Basic version of Reform you are limited to changing the logos on the forms. • For Developer+ you will need to run Reform on your own dedicated server. • The Pricing of the Developer and Developer+ versions include 4 hours of Web training. • Standard Hourly Rate for Customization ($165.00).

10. Hosted Solution Print-Job (PDF) • Secure Access • 128bit Encryption • Any Postscript Device • Automation is Simple Forms Printer Fabsoft Virtual Machines Client Dedicated! PDF Generated by Reform Discuss Linked Files! Forms Data Élan Login Élan Gateway (RDP) Server Hosted Élan Database Server • Where there are remote client sites, all printers should be networked to the main print server with IP addresses and an Internet connection. If printers are not networked to the main print server each office must have a print server.

11. Forms Modification • With Developer or Developer+ versions you can make changes – with some limitations: • The data on the form (generated by Élan) can be repositioned. • You can also create multiple versions of a form. For example, you may have invoices with different logos for different companies or products. • You can change fonts and font sizes on forms. • You can add graphics and other formatting to the design layer of a form • You can add your own static text, such as Terms and Conditions

12. Forms Modification • Some Standard Forms:

13. Questions & Answers • Features • Functionality • Migration to Hosted Model http://fabsoft.com/Products/Reform/Reform-PDC14.html $1,500.00 $2,500.00

14. System Administration FundamentalsPart 1 • Concepts & Tools • Service Management • Memory Management • Process Management • Lock Management • Backups & Recovery

15. Concepts & Tools • Concepts • Client Server Architecture • Unidata DBMS (Overview) • Unidata Architecture • Unidata Shell (ECL) • Tools • UniAdmin • Extensible Admin Tool • Command Line(Introduced as we go!)

16. Client Server Architecture • Unidata • Windows TCP/IP (WinSock) • Ports • Telnet (TCP 23) • UniRPC (TCP 31438) • SSL (992) Lets take a look at RPC Conduit! Even with Web front-end (Elan.Web) – Backend will be Unidata (Maintenance will not change)

17. Unidata DBMS Overview • UniDatais a relational database management system (RDBMS) designed specifically for business use. • A database management system is software that enables users to control the organization, storage, retrieval, security, and integrity of data in a database. • UniData’s technology is based on the nested relational database model. This nested design extends the standard relational model from a redundant, flat table structure to a three-dimensional database. • In the nested relational model that UniData employs, data is stored in files, and the records in a file can have attributes that contain values that can be either singlevalued, multivalued, or multi-sub-valued. This allows UniData to store relations within relations. • UniData’s method of storing data accomplishes the following: • Simplifies the data modeling process • Reduces the number of files and indexes in a database • Eliminates redundant data storage • Maintains relational access to all data in the database

18. Unidata DBMS Overview • Natural Database! • Has variable length fields and Files • Allows for Tables within Tables • Resulting in extremely condensed, efficient data • Comparison to “first normal form” database • 40% to 60% of SQL data is spaces and zeros • (PAD filling for fixed length fields)

19. Unidata DBMS Overview • U2’s variable length fields, files & data provide for a very fluid environment • If you don’t do anything on the system, it will just run slower! • 40 to 60% the size of a comparable first normal form data server • It’s compressed data • Random –no sequential IO • Disk subsystems must be tuned differently! • Places the burden on the CPU at the right time • After the data has been transferred from disk to memory • U2 is a relational database • Like all relational DBs, 85% of our Disk I/O is READS (Triggers!) • Anything we can do to reduce READS is a good thing • Smokinfast!

20. Unidata DBMS Overview • The nature of U2 Database (More when we discuss Monitoring/Tuning) • Unidata and Universe (U2) are both written in the “C” language • U2 uses standard “C” calls to the operating system (MALOC) • Whatever is presented via the O/S, Unidata can utilize • Disk subsystem architecture (Shadow-Copy API) • From it’s variable length records U2 generates millions of 2k – 8k Read/Write (IO) requests (we actually like to align with O/S read/write buffers) • U2 will overwhelm a system that was configured for “First Normal Form” Data (Default) • Default stripe sizes are too large! – 16k to 64k is best • Stripe sizes must be reduced, otherwise entire files will be placed on single disk • FNF servers perform “Prefetch”, U2 needs this disabled • Selects will use, but not worth having on for just this use • Raid5 can compound IO issues in U2 (We will discuss extensively)

21. Unidata Architecture • Elements of the Unidata Database • Database (Account) – SBLIVE is software Account • A UniData account is a directory that contains a default set of UniData files, including a VOC file and its dictionary. • The VOC file identifies commands, paragraphs, and all data files that are used in the UniData account. • The data files may be in the same directory as the VOC file, or the VOC file may contain pointers to data files in other directories. • File (CLIENTS, PBOOKINGS, SU.SUB) • UniData stores your data in hashed files of several different types. • UniData also supplies other types of files to support your database, including index files, program files, and directory files. • While files contain records, records contain one or more attributes (Fields) that contain specific information about one element of the record. • For example: • The CLIENTS file contains records about your customers. • Each record contains elements of information including; name, address, and phone number, each of which are stored in separate attributes .

22. Unidata Architecture • Elements of the Unidata Database • Dictionaries • Every UniData data file has a corresponding dictionary file. • A dictionary contains a set of records that define the structure of the records in the data file, called D-type records. • A dictionary may also contain phrases, called PH-type records, and items that calculate or manipulate data, called virtual fields, or V-type records. • Dictionary File (D_CLIENTS) • A dictionary file, just like a data file, is a collection of records containing attributes. • The purpose of a D-type dictionary record is to define the location of attributes in the data file. Other information, including the conversion code, column display heading, display format. • Primary Key • A primary key is a unique identifier for a record; it is also called a record ID, or @ID. • Each record must have a unique identifier that differentiates it from other records in the database. • UniData uses the primary key to store and locate records in files

23. Unidata Architecture • Elements of the Unidata Database • Variable-Length Attributes and Records • Unlike other databases that require you to specify an attribute length, UniData does not limit the length of your data within an attribute. • Nor does it pad the attribute or record to achieve a fixed length. UniData does not have to rely on fixed lengths to recognize where an attribute begins and ends. • Instead, it inserts delimiters between attributes and between records. This is called a dynamic array structure. • (nonprinting) 255 Record mark (@RM) * Marks the end of a record. • ~ 254 Attribute mark (@AM) * Marks the end of an attribute. • } 253 Value mark (@VM) * Marks the end of multivalue. • | 252 Subvalue mark (@SM) * Marks the end of multi-sub-value. • { 251 Text mark (@TM) * Marks the point where values wrap to a new line

24. Unidata Architecture • UniDatauses several access methods to store, manipulate, and report data. Briefly, these are: • UniBasic: • UniBasic is a powerful programming language that is used to write sophisticated programs. • From within UniBasic, we can run system-level processes or call C programs. • SB+ (System Builder 4GL) is written in UniBasic • UniQuery: • UniQuery is UniData’s nonprocedural query language for creating reports and viewing your database (Abstracted by SB). • UniData SQL: • UniData’s implementation of the SQL (Structured Query Language) data manipulation language. • AE: • AE is UniData’s “Alternate Editor”. • AE is used to edit records in the database and create UniBasic programs. (Let’s Discuss).

25. Unidata Shell (ECL) • UniDataprovides the Environment Control Language (ECL), a proprietary command language to handle database management functions. • ECL commands execute from the UniData colon prompt (:). • SB+ Abstracts this functionality (Discuss LOGIN/LOGOUT Paragraphs). • ECL commands and keywords install when you install UniData. They are stored in the UniData Vocabulary (VOC) file. • UniData also provides system-level commands which you execute from the shell prompt (stored in the udtbin directory). • You can execute system-level commands from the UniData colon prompt by entering the ! (bang) command first (for example, :!listuser). • Let’s discuss some key ECL commands. • LIST • SORT/SELECT/SSELECT • COUNT

26. Tools Will be your primary tools! D:\Unidata\Bin\*

27. Service Management • Unidata Database Service 7.2 • Shared Basic Code Server (sbcs) • Shared Memory Manager (smm) • Clean Up (cleanupd) • Unidata Telnet Service 7.2 • Attaches udt.exe to Database Service • Unidata Terminal Service 7.2 (Not needed – Disable) • Unidata ObjectCall Service 7.2 (No longer used) • UniRPC Service (UniObjects 3.0) • Unidata Mail Service (SMTP Bridge) Task Manager

28. Shared Basic Code Server (sbcs) • The shared basic code server (sbcs) manages shared memory used by globally cataloged UniBasic programs. UniData starts sbcs when you start the UniData Database Service (startud), and stops it when you stop the UniData Database Service (stopud). • The functions of sbcs include: • Loading and tracking globally cataloged programs—sbcs loads globally cataloged programs into shared memory as needed, and keeps track of the programs loaded and the number of processes executing each one. • Periodically, sbcs checks shared memory and removes loaded programs that are no longer in use. • Controlling shared memory—The sbcs daemon can attach up to 20 shared memory segments. • Process cleanup—At periodic intervals, the sbcs process checks the cleanupd service to see if terminated process flags have been set. If sbcs detects a terminated process flag, it removes all messages sent for the process. If the terminated process is the only process using a program in shared memory, it removes the program from shared memory. Sbcsprogs

29. Shared Memory Manager (smm) • The shared memory manager (smm) builds and manages structures and tables within shared memory. UniData starts smm when you start the UniData Database Service (startud), and stops it when you stop the UniData Database Service (stopud). • UniData processes (udt processes) communicate with smm to request and return shared memory. The UniData processes request shared memory from smm for the following tasks: • License control—The smm process tracks the number of users for which a site is licensed, and prevents more than that number of users from logging on to UniData. smm also displays warning messages when a license is about to expire. • User process tracking—When a user logs on to UniData, smm assigns an internal tracking number to the user’s process and records information about the process in tables within UniData. • Buffering program variables. • Storing query records and intermediate results. • Storing select lists. • Storing expression buffers. • Managing a current modulo table for dynamic files.

30. Clean Up (cleanupd) The clean up process (cleanupd) detects terminated user processes at check time intervals. If cleanupd detects a terminated process, internal flags are set. The smm and sbcs services periodically check to see if cleanupd has set internal flags. If these services detect flags, each service performs the necessary clean up and resets its own flag to zero. The cleanupd service performs clean up that is not handled by smm or sbcs. When the smm and sbcs services have reset their flags to zero, the cleanupd service resets its flag to zero, makes the user process ID available, and frees the local control table. UniData starts smm when you start the UniData Database Service (startud), and stops it when you stop the UniData Database Service (stopud).

31. UniData Telnet Service (udinetd.exe) The UniData Telnet Service (udinetd.exe) enables multiple users to log on to a single Windows platform to run UniData. With the UniData Telnet Service installed and started, your Windows platform exports a logon prompt to its network so that network users can log on and run UniData. When a user opens a UniData session through udinetd.exe (23 or 992), UniData writes screen output directly to a socket.

32. UniRPC Service The UniRPC service allows the local UniData systems to communicate with remote systems/software. The communicating systems must use TCP/IP networking software to make connections. The UniRPC service waits for a request from a client system to connect to a server process. When it receives a connection request, it checks the unirpcservices files to verify that the client system is allowed to request a particular service. If it can, the UniRPC starts the requested service, then returns to the listening state. Each client process connects to its own server process. Each server process uses the same amount of system resources as a local database user. Note: The terms local and remote refer to client and server programs or systems. However, because client programs can connect to server programs running on the same computer, remote does not necessarily imply that the server is on another physical computer system. Uses Port 31438

33. Log Files (D:\Unidata\Bin) The sbcs, cleanupd, and smm services each record messages in a pair of logs in the udtbin directory. In addition, the udt process writes messages to a log file called udt.errlog if a UniData process encounters file corruption in a data file. More on logs later! UniAdmin is best way to review.

34. Unidata Mail Service • Mail Service provides bridge to SMTP service by picking up records written to D:\Unidata\Mail and then formatting for delivery using standard SMTP compliant calls. Much more efficient than MAPI. • C:\Program Files (x86)\MSGL\Unidata Mail Service\ EzMailServ.exe • D:\UniData\ • Mail • Mail_History • Mail_Attachments • Mail_Bad

35. Memory Management • Shared Memory Overview • How UniData Uses Shared Memory • UniData Shared Memory Errors

36. Shared Memory Overview • What is shared memory? • Shared Memory is part of Windows Interprocess Communication facilities (IPC) that allows memory outside of process space to be configured and accessed by multiple processes • Why does Unidata use it? • Allows udt processes to share specific resources (eg - globally cataloged programs) • Allows udt processes to expand and contract memory use without having to grow and shrink the actual OS process size • Allows all udt processes to quickly access system-wide tables (eg - record locks, dynamic files current modulo) UniData shared memory implementation in Windows uses memory mapped files

37. How Unidata Uses Shared Memory • UniData interacts with shared memory by using system-level calls, UniData services, and UniData configuration parameters (udtconfig) to build its own structures in shared memory. • UniData defines shared memory segments that can be attached by UniData processes. The sbcs (shared basic code server) service creates shared memory structures for storing active globally cataloged UniBasic programs. • The smm (shared memory manager) service creates shared memory structures for internal tables required by UniData processes. UniData processes request memory for: • Buffering UniBasic variables • Storing intermediate results • Storing a current modulo table for dynamic files

38. How Unidata Uses Shared Memory • The shared memory manager (smm) creates shared memory segments as needed. • The size and characteristics of segments smm or the UniData Database Service create are determined by UniData configuration parameters. • Whenever UniData starts, it reads the udtconfig file located in D:\Unidata\Include and stores these values in shared memory. • smm subdivides each of its segments into global pages, and subdivides each global page into local pages. . • smm also creates and maintains internal tables that track the use of the structures it creates. These internal tables are stored in a shared memory structure called CTL.

39. How Unidata Uses Shared Memory • Segments a udt process attaches • Sbcs – if running globally cataloged UniBasic programs • Smm (ctl) – global control table • Smm (shmbuf) – for storing UniBasic variables and ECL intermediate results. • Smm (glm) – for storing group (physical) and record locks. • Unidata Shared Memory • DOC= ADMINNT.PDF (Unidata and Shared Memory) • gstt • lstt

40. UniData Shared Memory Errors • Run-Time Shared Memory Errors • Error when attaching shm (nnn,nnn,nnn), errno=22 • ^ SHM_GPAGESZ • Error when creating a shared memory segment (size=nnnn), errno=22 • ^ SHM_MAX_SIZE • No more entries in MI table in LCT – n • ^ SHM_LMINENTS • Monitor UDT.ERRLOG (Handled by WinVadm) • ipclog – created in the UniData account directory where the application was launched • Example name: UDT197860c0.ipclog

41. Process Management • List/Remove Users • Stop/Start Unidata • Pause/Resume Unidata (7.2.x) • Troubleshooting SBClient SBClient Users (udt.exe)

42. List & Remove Users • UniAdmin (Preferred Method) • Listing User Sessions (udt.exe) • listuser.exe • ECL = !listuser • Remove User Sessions • Ask Them to Logout (why?) • deleteuser.exe • Forces a user out of UniData and removes the user’s entry from the internal tables. • stopudt.exe • Logs a user out of UniData; a current write completes, but subsequent operations for that udt do not take place D:\Unidata\Bin

43. Stop & Start Unidata Correct Procedures • Stop Unidata • Stop Ancillary Systems (Discuss) • Stop MSG Schema/SQL Services • Stop Unidata Mail Service • Stop Phantom Scheduler (pstop.bat) • Pause Unidata (dbpause) – prevents new sessions • Have Users Logout (check with listuser or UniAdm) • Remove User Sessions (UniAdmin or deleteuser) • Resume Unidata (dbresume) • Stop Unidata (stopud –f) • Start Unidata • Consider Reboot (shutdown /f /r) • startud • Start Phantom Scheduler (pstart.bat) • Restart all Services (Mail, Schema, SQL) Must have Admin Privileges!

44. Pause & Resume Unidata • dbpause - system-level command that enables you to temporarily block updates to the database. • You can use this feature to perform maintenance tasks that require UniData to be stopped, such as backing up your data. • Now used by WinVadm scripts to provide non-stop (CONOPS) environment. • dbpause blocks most updates to the database that are made within UniData. Writes or transactions in process when you issue the dbpause command are completed before dbpause takes effect. UniData blocks updates until you execute the dbresume command. • UniData does not block system-level commands, such as COPY or MOVE. In addition, UniData does not block updates to the _HOLD_ file and the _PH_ file, and does not interrupt report printing.

45. Pause & Resume Unidata • Pause Status • The UniData system-level dbpause_status command returns information about the status of dbpause. • Resuming Processing • To resume processing after issuing the dbpause command, issue the dbresume command. • User processes resume, and writes that were blocked when the dbpause command was issued complete.

46. Troubleshooting • Stopping Unidata (stopud.exe) • Makes Windows calls to stop the Unidata Database service after: • Checking for Administrative Privileges • Checking to see if any Unidata processes are still active • NOTE: Stopud –f (force) completes current write and terminates (uses deleteuser on all active sessions) • Run guide if you have to force stop • Stopud makes calls to the following programs in D:\Unidata\bin: • Stopsmm (stops shared memory manager) • Stopsbcs (stops shared basic code service • Stopcleanupd (stops cleanup service)

47. Troubleshooting • Terminating a Unidata Process • As discussed, UniAdmin is best (uses Deleteuser) • deleteuser <PID> • Sends TERM signal; sleeps; checks if PID still exists • If TERM signal is not sufficient to remove PID • Create entry in D:\Unidata\Bin\deleteuser.log • SIGKILL Signal is sent to process (TASKKILL) • Stopudt <PID> • SIGHUP is caught by udt.exe • Udt process send message (IPC) to SMM to release locks and clear license; aborts • Windows hard kill (Task Manager) • Like stopudt (timing issues with messages). If a process dies or is terminated from hard kill cleanupd and smm work together to remove all shared memory traces! (cleanupd logs)

48. Troubleshooting • Starting Unidata (startud) • Startud calls StartUdSrv.exe to start the Unidata Database service after: • Checking for Administrative Privileges • Copies all *.log & *.errlog files to D:\Unidata\Bin\Saved_Logs • Preserves the last 20 iterations • Unidata Dabatase Service Then: • smm (starts shared memory manager) • sbcs (starts shared basic code service • cleanupd (starts cleanup service) • Each program makes entries in the log files (smm.log, sbcs.log, and cleanupd.log) that identify the system resources used by the services

49. Lock Management • Global Lock Manager (GLM) • The GLM is an internal software module that is linked into each udt (UDT.EXE) process to manage logical record locks. • UniBasic Locks (Records Locks) • A series of UniBasic commands enable programmers to set read-only locks and exclusive locks on UniData files and their contents. • Group Locks (Managed by Unidata Processes/Services) • Protects the integrity of hashed files groups by controlling access.

50. Global Lock Manager (GLM) • The GLM manages local lock tables for each udt process and a shared global lock table in shared memory, which can be accessed by multiple udt processes. • The lock tables are hashed tables containing linked lists, which contain lock nodes. • When a udt process locks a record, UniData writes the file name, record ID, and lock mode to both the local lock table and the global lock table. • When a udt process requests a lock, UniData first searches that local lock table for the udt to see if that process is holding the lock, then the global lock table to see if another udt process is holding the lock.