WW TSS-13 Wonderware MES Support and Troubleshooting Technique

2. WW TSS-13Wonderware MESSupport and Troubleshooting Technique Presenters: Dan Scott Jim Stella

3. Session Agenda • MES Architecture Overview • Middleware Technology Changes 3.5 to 4.x • Database Tables • Tips and Common Mistakes • Troubleshooting

4. MES Architecture • Overview

5. Wonderware MES Architecture MES Client Operator MES Client Supervisor InTouch w/.NET Controls Supervisor Galaxy Repository Reports Configurator Field I/O InTouch w/MES API ArchestrA IDE Supervisory (P) Shop Floor (P) MES Admin (P) AOS Platform (M) FactoryConnector Server (M) OperationsCapability Obj UtilizationCapability Obj Middleware Server (M,S) Sample Recording Obj Reports Server (M or None) Database Server App Objects w/ MES API Key: M = Full Middleware P = Middleware Proxy S = MES Service

6. Technology change in • MES Middleware • from 3.5 to 4.x

7. Wonderware MES Middleware 3.5

8. Wonderware MES Middleware 4.x WCF (Windows Communication Foundation)

9. MES Database Tables

10. Common MES Database Tables Used in Troubleshooting Sessn – User sessions • Who is logged into which session ID, or not in some cases • Records with heartbeat older than 10 minutes can be purged • These should be auto-purged by MES Service if it is running • Only ever one session type 53 – Background User for MES Service • Orphaned type 53 record can stop MES Service from starting • Tech Note 699 covers this table and the client type in more detail Ent_Logon – User logged on entities • Which user is logged into which entities • User must have privilege for those entities

11. Common MES Database Tables Used in Troubleshooting Item_Prod – Production records • Distinct: Default selection • Can report hourly or shift OEE • Records stored for every production or consumption transaction • Can result in large record count • Shift: • Can report shift OEE only • Most compact record count • Hourly: New for 3.5 p02, MES 4.0 SP2+ (replaces shift) • Can report hourly or shift OEE • Compromise between Distinct and Shift option • Should be purged periodically to maintain high performance

12. Common MES Database Tables Used in Troubleshooting Util_Log – Utilization history • Should be purged periodically to maintain high performance TPM_Stat – OEE raw data • Data only stored when a job is running • Should be purged periodically to maintain high performance Job – Jobs and related status • Contains total production count, allowing item_prod details to be purged Job_Exec – Active job information • One placeholder record per allow Job Exec per entity • Never delete records

13. Common MES Database Tables Used in Troubleshooting Util_Exec– Current utilization information • Lookup current raw reason OEE_Exec – Current OEE information • See current Quality and Performance OEE subcomponents • Quality % x Performance % x Availability % = OEE % Ent – Entities • View current shift ID and times • View capabilities

14. Common MES Database Tables Used in Troubleshooting Shift_to_go – Upcoming shifts • Can be purged, then restart MES Service to rebuild list Tz_offset – Pre-calculated time zone offsets from UTC • Only time zones of interest are calculated • Can be purged, then restart MES Service to rebuild list • Purge is required if the database is moved to another time zone WO – Work Orders • Should be purged at some point. (Archive/Purge has System parameters that can be defined to automatically remove the data) Item_inv – Inventory

15. MES Tips and • Common Mistakes

16. MES Tips and Common Mistakes:Middleware • Middleware proxy must be installed on IDE node if using Entity Model Builder (EMB) • You must open/close a capability object editor window before running EMB • Full middleware should be installed App Object and Manager nodes • License purchase required per full middleware server (multiple inc.) • Middleware is not deployed, must be installed manually • Use Proxy Configuration Editor to test the connection to MW server • Use MES Client or Configurator to test client connectivity since you will get better errors • If middleware is remote from database, MES Client must be installed

17. MES Tips and Common Mistakes:Load Management • Server / Platform Loading • What else runs on this node? • Available system resources? • Engine Loading • Engine Scan Rate? • Many capability object functions result in MES database transactions. • Consider separate engines for MES objects versus regular, higher speed, App Objects • Middleware Loading • Capability Object count? • No more than 200-300 per platform* • No more than 50 SROs per platform* • No more than 30-50 per engine* • Full middleware on the platform hosting MES App Objects • Reporting? • Database Loading • Dedicated MES Database server? • MES has high transaction count *Suggestions dependent on transaction volume and scan rate

18. MES Tips and Common Mistakes: Maintain Distinct Production Records • Enabled to report Hourly OEE • Daily/Shift OEE can still be reported with this disabled • This setting can have a significant impact on the MES database size • If enabled (default), a new item_prod record is created every time production is recorded • If disabled, the system will update existing item_prod records • New feature in 3.5 p02, 4.0 SP2 aggregated hourly prod records

19. MES Tips and Common Mistakes: Multiple Job Execution • How many Jobs can be run concurrently on each entity • More than 10 can cause problems • OCO has a hard cap of 50, but not recommended for entities with large transaction numbers

20. MES Tips and Common Mistakes: OCO/UCO Production Counters • Misconfiguration, especially with a rollover counter, can be disastrous • NEVER set Deadband, Max Value, and Update Interval to 0 • Production may be recorded every engine scan • Every time production is recorded, you are transacting with the MES Database • If Maintain Distinct Production Records is enabled (default), you are creating a new item_prod records every time production is recorded • UCO rollover counter Max Value MUST be synced with PLC rollover • OCO rollover counter can be reset on demand, or PLC sync (TN 710)

21. MES Tips and Common Mistakes: Raw Reason Code Management (UCO) • Every Raw Reason Code change results in multiple MES Database transactions • Changing reasons too fast can cause reason codes to be missed (TN657) • UCO can only handle current change plus queue the next change • Micro stops should be recorded to Historian (events < 30-60s)* • Future product plans in the works to allow for faster changes *Suggestion dependent on transaction volume and scan rate

22. MES Tips and Common Mistakes: Continuously Running Work Order • “But we’re only using Performance, we don’t care about Work Orders” • The underlying MES database structure and table relationships makes Work Order management important • Problems with continuously running work orders: • Database performance degrades over time • Updating database records require the system to query large datasets • Archiving capabilities are limited (ex: Archive WO where closed > 30 days) • General recommendation is a new Work Order daily or at shift change

23. MES Tips and Common Mistakes: Work Queue Grid Filtering • The more that you try to display on the Work Queue grid, the slower it will load and refresh • How many Jobs does an operator really need to see? • Use New/Ready status to limit records • Create jobs as New, have a supervisor set them to Ready when they assign the work. Hide New jobs from the operator’s work queue. • Eliminates operator confusion

24. MES Tips and Common Mistakes: Utilization Grid Filtering • The more that you try to display on the Utilization grid, the slower it will load and refresh • No filter will query the database for ALL utilization records for this entity • What does the operator really need to see vs. what can just show up on a report? (right-click)

25. MES Tips and Common Mistakes:Security • If utilizing OS-based security, the Wonderware Change Network User account must be a domain user • Special steps required to enable OS Group Security - Tech Note 708 • Lost Database Connection Editor password • Delete C:\Program Files\Wonderware\ MES\Middleware\Middleware.cfg

26. MES Tips and Common Mistakes: OEE Calculation

27. MES Tips and Common Mistakes:Data Inconsistencies • Moving database to a new timezone? • Purge shift_to_go and tz_offset tables • Restart MES Service • Errors that Job exec limit exceeded, but Operator work queue does not show that many jobs running • Job state and Job_Exec current job are out of sync

28. MES Tips and Common Mistakes:What does MES Service do? • Only one MES Service can run per database • Executes minutely, hourly, and daily housekeeping tasks • Shift changes • Session clean-up • Auto-ready jobs • Run scheduled Supply Chain Connector tasks • Run scheduled archive and purge tasks • Generate time-based Quality Samples

29. MES Tips and Common Mistakes: System Maintenance • Database maintenance • Truncate transaction logs and shrink files and database (TN 664) • Archive and/or Purge • Large tables: • item_prod (Production details) • util_log (Utilization states) • tpm_stat (OEE stats) • Custom reports can union data from Prod and Archive databases • Available from MES Services, not part of the standard product • Tech Note 746 - Using the New MES 4.0 Archive, Purge and Restore Functions

30. MES Tips and Common Mistakes:Supply Chain Connector MES Database • Supply Chain Connector can have data brought in by either data file or database Table. File Xml or delimited Supply Chain Connector (MES Service) Middleware Table 3rd party or custom

31. MES Tips and Common Mistakes:Supply Chain Connector • Questions to consider when deciding whether to use files or tables: • How much data will I be importing? • How often will the data be imported? • How much of the importing data will be new and how much will be updating existing records? • What will be creating the new data that will be imported?

32. MES Tips and Common Mistakes:Supply Chain Connector • File Base imports: • Pros • Easy to implement • Good for smaller amounts of data • Easy one time configuration scenarios • Cons • Cannot track the processing of individual records • Higher chances of data integrity issues • Slower performance with larger amounts of data

33. MES Tips and Common Mistakes:Supply Chain Connector • Table Base imports: • Pros • Able to track and keep information on individual processed records • Reprocessing of individual records is easier • Troubleshooting bad records is easier to find. • Cons • More difficult to implement • Better for larger amounts of data. • Needs more maintenance (cleaning out old records, backing up of tables….)

34. MES Troubleshooting • Tools and Techniques

35. MES Troubleshooting:Middleware • Licensing • MES Licenses must be installed on Middleware servers (restart MW) • Performance-only or Quality-only license can use OCO for production counts and basic job execution • MES 4.0 license is valid for MES 4.5 • Database Connection Editor (restart Middleware if changed) • Verify SQL connectivity • Middleware Configuration Editor • Enable XML Trace of all middleware/SP calls with parameters (restart MW) • Middleware Proxy Configuration Editor • Test connection to Middleware server

36. MES Troubleshooting:Error Messages • MES SMC Log Flags • UCO – EquipmentStatusRuntime1 • OCO – JobExecutionRuntime1 • SRO – SampleCharacteristicRuntime1 • Entity Model Builder – EntityModelBuilder • MES Service – FactMES.Server.Services • MES ArchestrA Objects • .ErrorMessage attribute

37. MES Troubleshooting:MES Client Debug • Modify C:\Program Files\Wonderware\MES\Client Framework Modules\aaClient.exe.config Change this section: <!-- Uncomment to enable the file based trace listener.<add name="LogTraceListener" type="System.Diagnostics.TextWriterTraceListener“initializeData="Trace.log" /> --> <!-- Uncomment to enable the file based trace listener. --><add name="LogTraceListener“type="System.Diagnostics.TextWriterTraceListener“initializeData="Trace.log" />

38. Useful Tech Notes • Tech Note 753 - Resolving MES Entity Model Builder Errors on a Remote IDE Development Node • Tech Note 746 - Using the New MES 4.0 Archive, Purge and Restore Functions • Tech Note 744 - Resolving Incorrect Shift Times When MESDB is Moved to Another Time Zone • Tech Note 723 - Solving Wonderware MES 4.0 Known Installation Issues • Tech Note 711 – Wonderware MES Operation Capability Object: Configuring Multiple Job Counters

39. Useful Tech Notes • Tech Note 710 - Wonderware MES Operations Capability Object: Configuring Rolling Counter Reset • Tech Note 708 - Enabling MES 4.0 Operating System (OS) Group Security • Tech Note 699 - Wonderware MES Sessn Table and Client Types • Tech Note 664 - Truncating Operations & Performance Transaction Logs • Tech Note 657 - Wonderware Performance 3.5 Software: Performance Tips

40. Other Useful Documents • MES 4.0 Deployment Guide • MES Database Diagrams • MES 4.0 Development Guide

41. Useful Sessions WW TSS-09: MS SQL Server Performance & Tuning for Wonderware Database

42. Questions? THANK YOU