Teradyne License Service and Monitoring

1. Teradyne License Service and Monitoring Bob Van der Kloot Teradyne, Inc 978-370-2014 bob.vanderkloot@teradyne.com

2. License Service Goals • High Availability • Minimize effort for maintenance/support • Standardization of operation • Common code across platforms • Compliance with license agreements • Provide information necessary to manage license portfolio • Easy access to metrics • Standard access

3. License Server Technologies • Acresso (Macrovision) FlexLM • The vast majority of our products are licensed with FlexLM. • We support on Solaris and Windows servers • We support complete logging (1 sec resolution) • No LMAdmin!!! • Reprise License Manager (RLM) • A new product by the creators of FlexLM, and very similar to FlexLM • We support on Solaris servers • Currently only used by RTDA • We support only daily peak logging, though full logging should be trivial • LUM • Now owned by IBM • We support on Solaris servers • Currently only used by SmarTeam • We support only daily peak logging, more may be available via FNM • Vendor Proprietary Licenses • MicroFocus/Borland SilkTest License • Rational Atria License (deprecated, but still in use) • Compuware DLM license (not currently in production) • We support only daily peak logging

4. Support Technologies • Veritas Cluster Server • Windows and Solaris • Veritas Volume Replicator • Windows and Solaris • Acresso Flex Net Manager • Linux Master • Windows and Solaris Clients • Oracle Database (Linux Server)

5. Agoura Hills, California “means” Solaris 8 cluster Global license service “cadet” Solaris 8 cluster North America license service “devways” Solaris 8 cluster Disaster recovery / development Clone of “ways” North Reading, MA “ways” Solaris 8 cluster Global license service “adriatic” Solaris 8 cluster North America license service “wpcluster” Windows Server 2009 cluster Global license service Types of License Servers • Solaris vs Windows Architecture • Driven by availability of license daemons • Global vs North America Service • Driven by Contractual agreements • All are Veritas VCS Clusters • Achieve High Availability by: • High availability cluster • Distribute pool between locations when possible • 50/50 distribution for vendors that support multiple lic servers

6. License Server Principles • Management of licenses is as regular as possible across: • License Server Platform • Solaris • Windows • Linux (when we choose to implement it) • License Technology • FlexLM • LUM • etc… • Cluster or non-cluster • All production licenses are on clusters • Consistent Abstraction layer provided for automation… same for all licenses • Cluster control • Start • Stop • Health Check • Clean • Monitoring • List of licenses in consistent format • Quantity • Expiration • Characteristics (UHD, etc) • Current Status • Current checkouts • Duration of each checkout

7. License Cluster Basics – Veritas Cluster Server • Keep it Simple • Two cluster members • Active member: • Hostid is faked • Licenses are served • Passive member: • Native hostid of the server • Licenses are disabled • Veritas Volume Replicator for volume containing licenses • /data/vol/licenses/<vendor> /keys /bin /logs • Exact same path on windows server (via links in cygwin) • Standardized control of licenses: • Cluster needs to do: • Start • Stop • Health Check • User needs to do: • Request cluster to start license • Request cluster to stop license

8. License Daemon Control lmgrdFOO lmgrd_generic cluster_control Administrator (human) Customization ofcontrol of FlexLMdaemons Custom each Vendor Standardizedcontrol of FlexLMdaemons Encapsulate Control And Status Check Of Cluster Environment lmutil hares lmgrdFOO lmgrd_generic AcressoFlexLM daemoncontrol HA softwareevent detectionorAdministrator VCS cluster software controlinterface Customization ofcontrol of FlexLMdaemons Custom each Vendor Standardizedcontrol of FlexLMdaemons lmgrd AcressoFlexLM daemoncontrol The VCS cluster software uses the same control programs thatthe human administrator uses(but with different arguments) Vendor Daemon SuppliesLicenses Custom each Vendor

9. Typical customized calling module #!/bin/sh # This is lmgrdCADENCE # control script for Cadence daemons # this script must be run by "cds“ VENDOR='cadence' /etc/init.d/lmgrd_generic -o $1 -v $VENDOR

10. Command interface of lmgrd_generic -------------------------------- lmgrd_generic version: 1.60 -------------------------------- Usage: lmgrd_generic -o operation -v vendor # vendor name (directory in /data_vol/licenses ) [-k key_spec ] # license file name (.lic implies all .lic files in directory) # (default to license.dat) [-l lmgrd_name ] # lmgrd name override (default to lmgrd ) [-d daemon] # vendor daemon executable name (can replace one only) [-e effective ] # effective daemon name (when daemon runs) [-h hostid] # hostid override [-s sequence] # daemon start sequence ( 0 to 99 allowed ) [-p prefix] # prefix of vendor for control script name [-n] # dongle ( license file will be license_<host>.dat ) [-g] # enable logging [-rev] # print revision and exit [-rlm] # reprise license server (not macrovision) [-w] # windows operation is one of: check - examine files for correct settings start - start license daemon stop - stop license daemon restart - restart daemons (use when license file is changed) clean - clean up from crashed daemon info - report PID numbers of running processes for use only by VCS cluster control: vcs_start - USED BY CLUSTER CONTROL vcs_stop - USED BY CLUSTER CONTROL vcs_clean - USED BY CLUSTER CONTROL vcs_monitor - USED BY CLUSTER CONTROL Written in perl, same executable runs on solaris, windows (cygwin) and linux

11. License Monitoring Goals • Standard Interface across all products • Maintainable • Standard WEB interface useful for beginning and advanced users.

12. Two license monitoring methods used • Sample use every 15 minutes • Program “license_use” run remotely on each license server • Returns record for each license in use, across all used license technologies • vendor|product|version|qty|user|host|display|duration • Data stored in Oracle, plots produced on demand • FlexNetManager • Records to-the-second license use, but only for FlexLM • Possibly also for LUM, have not tried • Output in “raw” format done once per day, read into Oracle • Plots produced on demand.

13. Demo of license monitor • I can’t include slides without violation of vendor license agreement(because it would have real usage data) • I’ll show a live demo

14. LSF Batch Environment • 6 sites • 200+ machines • Monitoring by homemade programs • Read LSF log into Oracle

15. What’s Next? • Major effort to continue automation and standardization • Reduce Tool and License Administration from 2.5 FTE to ~.6 FTE • Significant improvement in prediction tools • Give better access to historic data, so project teams can forecast their future demand.