40 likes | 48 Views
This report highlights the service availability performance of ERCOT's IT systems for Market Operations, Grid Operations, and Retail Market in October. It includes details about the performance degradation that affected the processing of 867 transactions and the root cause of the issue.
E N D
Information Technology Report Trey Felton Manager, IT Service Delivery ERCOT Public
Highlights Service Availability – October • Market Operations IT systems met all SLA targets • Market Data Transparency IT systems met all SLA targets • Grid Operations IT systems met all SLA targets • Retail Market IT Systems met all SLA targets October • ERCOT experienced Retail system performance degradations primarily impacting processing of 867 (meter read) transactions, beginning Monday afternoon (10/28) and continuing through late Wednesday (10/30). • Details in following slides 3-4 • R6 – Dec 7/8 • 2014 SLA Review • Draft ERCOT Public
Retail Transaction Processing Delay Update – Summary • Issue: Retail system performance degradations primarily impacting processing of 867 (meter read) transactions, beginning Monday afternoon (10/28) and continuing through late Wednesday (10/30). • Root Cause: The issue has been isolated to a security application’s interaction with ERCOT’s EDI (Electronic Data Interchange) processing. 867 transactions were noticeably impacted due to the size and increased processing complexity compared to other transactions. Root cause analysis is still ongoing with ERCOT and our vendor. • Impact: Preliminary impacts are as follows: • Approximately 250,000 867_03 Monthly Meter Reads processed from TDSP’s and forwarded to LSE’s were processed outside of protocol timelines. • Approximately 525,000 867_03 Monthly Meter Reads processed from TDSP’s and forwarded to LSE’s were processed outside of Commission required timelines. • Approximately 15,000 non-monthly read 867s processed from TDSP’s and forwarded to LSE’s were processed outside of protocol timelines (Historical Usage Reads, Initial/Final Meter Reads used to complete customer enrollment related activity).
Retail Transaction Processing Delay Update – Timeline • Monday afternoon, 10/28/13: performance degradations in Retail processing began to result in a backlog • Large 867 transaction files (mostly monthly meter reads) received in batches from TDSPs began processing slowly • These processes began to ‘hold up’ other files not necessarily impacted by the root cause. • Many files received Monday evening were backlogged through Wednesday • Tuesday morning, 10/29/13: ERCOT Operations began making changes to improve throughput, while investigating the root cause • Files which were observed to be processing more quickly were routed through alternate paths to mitigate backlogs. • 814 transactions (enrollment requests and responses) were processed largely as received and within protocol guidelines. • Market Notices sent at ~ 12:08 PM and ~11:00 PM • Wednesday evening, 10/30/13: ERCOT identified the root cause of the issue, mitigated the issue, and began clearing the backlog of the 867 transactions • Market Notices sent at ~8:30 AM, ~12:50 PM, ~5:22 PM, and ~11:13 PM (resolution notice) • Market calls at 9:30 AM and 2:30 PM • ERCOT began providing lists of impacted 867s to the relevant impacted parties, if requested, for synchronization/comparison purposes (this is an ongoing effort, on request) • Thursday, 10/31/13: Issue resolution follow-up communications • Market call at 9:30 AM • Market Notice sent at ~2:06 PM