1 / 28

Machine-to-Machine Interfaces June 28 2007 Stephen Kerr

Machine-to-Machine Interfaces June 28 2007 Stephen Kerr. Agenda. Updates to External Interface Specification Issues List Web Services Update Release Schedule Early EDS3 EDS3 & API Subgroup Face to Face State & Sequence for submission Notifications & Listeners

jolie-williams
Download Presentation

Machine-to-Machine Interfaces June 28 2007 Stephen Kerr

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Machine-to-Machine InterfacesJune 28 2007Stephen Kerr 1

  2. Agenda • Updates to External Interface Specification • Issues List • Web Services Update • Release Schedule • Early EDS3 • EDS3 & API Subgroup Face to Face • State & Sequence for submission • Notifications & Listeners • Sandbox vs EDS3 Implementation • Upgrading from demo services (WhoAmI, Boomerang, 3PO demo) • Potential Additional Services • Potential private network implementation • Q&A 2

  3. Updates to External Interface Specification • Certain elements of the External Interface Specification were approved on May 21st on condition that regular updates were provided to the specification • Revision 1.01 was published to TPTF on June 15th for comment • The major changes are the addition of Pending state and Notifications • Received comments related to corrections to message content e.g. expiration Time and curve resolution and Submission Validation and language corrections. Message content and language will be addressed in the next minor revision planned for mid July, submission validation in the next major revision • The next major revision will be published after the MMS design review and walkthrough on July 9 - 13 • Next major revision to the Interface specification will occur by early August • We will publish as soon as we have concrete information 3

  4. External Interface Specification – to do’s • Revise ASOffer to be consistent with final MMS design • Finalize AS type codes (e.g. Reg-Up or REGUP) • Multi-hour block indicator for ASTrade • Add notification for non-spin and RRS deployment, dependent upon MMS design • There is a UI requirement but this is not on the current plan for web services • Originally logged on 3/30 but raised recently by GPL • Finalize transactions states and associated handling. Document has been revised to reflect PENDING state. This will currently not be handled by MMS and will need to be handled by the integration layer • Finalize usage of cancel energy flag. This is dependent upon MMS design. • Define SLAs • Identify the means to notify market participants when documents in the repository have been created or updated 4

  5. Jun Jul Aug Sept Oct Nov Dec Objectives • The following Web Services should be available by June 30th, 2007 Objectives • The following Web Services should be available by July 30th, 2007 Objectives • The following Web Services should be available by August 30th, 2007 Objectives • The following Web Services should be available by September 30th, 2007 Objectives • The following Web Services should be available by October 30th, 2007 Objectives • The following Web Services should be available by November 30th, 2007 • Web Services – EIP • Three Part Offer • Incremental/Decremental Offers • Current Operating Plan • Output Schedule • Bid Set Acceptance • Bid Set Errors • Get mRID • System Status • Web Services – EIP • Market Totals • Binding Constraints • Active Contingencies in SCED • Voltage Profiles • Competitive Constraints • Shift Factors • Ancillary Service System Plan • Pending Trade • Energy Only Offer Awards • Energy Bid Award • Ancillary Service Awards • Web Services – EIP • Self Arranged Ancillary Services • Ancillary Services Offer • Ancillary Services Trade • Energy Bid • Energy Only Offer • Forecasted Load • Real-Time System Load • Market LMPs and SPPs • Mitigated Curves • Outage Creation • Web Services – EIP • Congestion Revenue Rights • PTP Obligation • Self Schedule • Total Regulation • Load Ratio Share • Unit Availability • Startup and Shutdown Instructions • Proxy Curves • Notices and Alerts • CRR Awards • Outage Notifications • Web Services – EIP • Dynamic Ratings • Aggregated Ancillary Service Offer Curves • Total Ancillary Service Offers • Total DAM Energy • Derated CRRs • Alert Acknowledgement • Outage Query • Outage Cancellation • Web Services – EIP • Capacity Trade • DC Tie Schedule • Energy Trade • Get AwardSet • Market MCPCs • Ancillary Service Schedule Obligations • Load Distribution Factors • Customer Load Profile • Energy Offer Awards • PTP Obligation Awards Web Service Sandbox Implementation Timeline - original • Testing cycle time is 6 weeks, not the original 4 week estimate, so the schedule must change 5

  6. Jun Jul Aug Sept Oct Nov Dec Objectives • The following Web Services should be available by June 30th, 2007 Objectives • The following Web Services should be available by July 30th, 2007 Objectives • The following Web Services should be available by August 30th, 2007 Objectives • The following Web Services should be available by September 30th, 2007 Objectives • The following Web Services should be available by October 30th, 2007 Objectives • The following Web Services should be available by November 30th, 2007 • Web Services – EIP • Three Part Offer • Incremental/Decremental Offers • Current Operating Plan • Output Schedule • Bid Set Acceptance • Bid Set Errors • Get mRID • System Status • Web Services – EIP • Market Totals • Binding Constraints • Active Contingencies in SCED • Voltage Profiles • Competitive Constraints • Shift Factors • Ancillary Service System Plan • Pending Trade • Energy Only Offer Awards • Energy Bid Award • Ancillary Service Awards • Web Services – EIP • Self Arranged Ancillary Services • Ancillary Services Offer • Ancillary Services Trade • Energy Bid • Energy Only Offer • Forecasted Load • Real-Time System Load • Market LMPs and SPPs • Mitigated Curves • Outage Creation • Web Services – EIP • Congestion Revenue Rights • PTP Obligation • Self Schedule • Total Regulation • Load Ratio Share • Unit Availability • Startup and Shutdown Instructions • Proxy Curves • Notices and Alerts • CRR Awards • Outage Notifications • Web Services – EIP • Dynamic Ratings • Aggregated Ancillary Service Offer Curves • Total Ancillary Service Offers • Total DAM Energy • Derated CRRs • Alert Acknowledgement • Outage Query • Outage Cancellation • Additional Services • Web Services – EIP • Capacity Trade • DC Tie Schedule • Energy Trade • Get AwardSet • Market MCPCs • Ancillary Service Schedule Obligations • Load Distribution Factors • Customer Load Profile • Energy Offer Awards • PTP Obligation Awards Web Service Sandbox Implementation Timeline - revised • Completed smoke testing • Plan to complete 1st pass functional testing on June 29th • 30% complete & pass rate of 75% • Risk of slippage to July 10th 6

  7. Early EDS3 Web Services • Web Services will be available on the Sandbox July 10th in worst case • Three Part Offer • Incremental/Decremental Offers • Current Operating Plan • Output Schedule • Bid Set Acceptance • Bid Set Errors • Get mRID • System Status • The services will initially be “loop-back” which means there is no connectivity to MMS validation logic • Integrated web services with MMS validation is forecast to complete in late August 7

  8. Pending State • Introduced Pending to indicate the initial validation of a submission prior to the Day Ahead Validation • There will be internal interaction with MMS to determine when the DAM validation occurs • The rules around DAM validation will be published by MMS team 8

  9. Notifications & Listeners • Market Participants will subscribe to Notifications of content changes through the MIS system • The details of which content MP’s can be notified on is not yet defined • MP systems must register listener URLs and email addresses • The Integration Layer monitors for changes to subscribed content and issues a notification to the registered URL’s and email addresses • The notification structure is defined but content is not yet designed • MP systems must be capable of listening for and receiving these asynchronous messages and responding with an Acknowledge • ERCOT also transmit a heartbeat during quiescent periods to monitor the listener network • Emails are transmitted on error conditions • ERCOT do not plan to provide working samples for listeners but a useful guides can be found on the Apache AXIS project 9

  10. Notifications & Listeners 10

  11. Outline Variations to WS-Notifications Acknowledge messages Heartbeats Notification test interface Notification media 11

  12. Changes to Notification Interface WS-Notifications: Requires many XSD files Defines many interfaces, of which only one is needed by MPs Specification has an implementation issue/gap related to confirmation of receipt Can be simplified and improved for use by ERCOT Goal is to simplify while improving reliability 12 12

  13. ERCOT Variations from WS-Notifications Using WS-Notifications base XSD and WSDL files as a starting point: Eliminated unnecessary interfaces and data structures Now defined using two files: Notification.xsd and Notification.wsdl No need to import external files Added ‘Acknowledge’ reply message, which MUST be sent after the receipt of each notification Added SOAP fault 13 13

  14. Acknowledge Messages 14 Fills a ‘gap’ in the WS-Notifications specification Provides a confirmation to ERCOT that the Notification message was in fact received by the MP Provides a confirmation that the MP listener is functioning and that the URL was correctly specified by the MP Recommended that the MP listener put the Notification on a queue and then reply immediately, where the queue would insure for the MP that the Notification is not lost If there is too long a delay (e.g. as a consequence of MP application processing) before the Acknowledge is sent back, the rate at which ERCOT can send Notifications to the MP will be degraded Structure of Acknowledge message is very simple 14

  15. Notification.xsd <xsd:schema xmlns:wsnt="http://www.ercot.com/schema/2007-06/nodal/notification" xmlns:xsd="http://www.w3.org/2001/XMLSchema" targetNamespace="http://www.ercot.com/schema/2007-06/nodal/notification" elementFormDefault="qualified" attributeFormDefault="unqualified"> <!-- ================== Message Helper Types ===================== --> <xsd:complexType name="NotificationMessageHolderType"> <xsd:sequence> <xsd:element name="Message"> <xsd:complexType> <xsd:sequence> <xsd:any namespace="##any" processContents="lax"/> </xsd:sequence> </xsd:complexType> </xsd:element> </xsd:sequence> </xsd:complexType> <xsd:element name="NotificationMessage" type="wsnt:NotificationMessageHolderType"/> <!-- ========== Message Types for NotificationConsumer =========== --> <xsd:element name="Notify"> <xsd:complexType> <xsd:sequence> <xsd:element ref="wsnt:NotificationMessage" maxOccurs="unbounded"/> <xsd:any namespace="##other" processContents="lax" minOccurs="0" maxOccurs="unbounded"/> </xsd:sequence> </xsd:complexType> </xsd:element> <xsd:element name="Acknowledge"> <xsd:complexType> <xsd:sequence> <xsd:element name="ReplyCode" type="xsd:string" minOccurs="0"/> <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/> <xsd:any namespace="##other" processContents="lax" minOccurs="0" maxOccurs="unbounded"/> </xsd:sequence> </xsd:complexType> </xsd:element> <xsd:element name="Fault"> <xsd:complexType> <xsd:sequence> <xsd:element name="FaultCode" type="xsd:string" minOccurs="0"/> <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/> <xsd:any namespace="##other" processContents="lax" minOccurs="0" maxOccurs="unbounded"/> </xsd:sequence> </xsd:complexType> </xsd:element> </xsd:schema> 15

  16. Notification.wsdl <wsdl:definitions name="WS-BaseNotification" targetNamespace="http://www.ercot.com/wsdl/2007-06/nodal/notification" xmlns:wsntw="http://www.ercot.com/wsdl/2007-06/nodal/notification" xmlns:wsnt="http://www.ercot.com/schema/2007-06/nodal/notification" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"> <!-- ===================== Types Definitions ====================== --> <wsdl:types> <xsd:schema> <xsd:import namespace="http://www.ercot.com/schema/2007-06/nodal/notification" schemaLocation="Notification.xsd"/> </xsd:schema> </wsdl:types> <wsdl:message name="Notify"> <wsdl:part name="Notify" element="wsnt:Notify"/> </wsdl:message> <wsdl:message name="Acknowledge"> <wsdl:part name="Acknowledge" element="wsnt:Acknowledge"/> </wsdl:message> <wsdl:message name="Fault"> <wsdl:part name="Fault" element="wsnt:Fault"/> </wsdl:message> <!-- ========= NotificationConsumer PortType Definition =========== --> <wsdl:portType name="NotificationConsumer"> <wsdl:operation name="Notify"> <wsdl:input message="wsntw:Notify" /> <wsdl:output message="wsntw:Acknowledge" /> <wsdl:fault message="wsntw:Fault" name="fault1"/> </wsdl:operation> </wsdl:portType> </wsdl:definitions> 16

  17. Notification Handling: Normal Case 17 • ERCOT Nodal application triggers notification processing • ERCOT NotificationBroker logs the notification and determines destination(s) • ERCOT sends Notification message to the MP at configured URL • MP receives Notification message (at which point they should queue it for use by their applications) • MP sends Acknowledge message as a response • ERCOT marks Notification as ‘sent’ 17

  18. Notification Handling: Error Handling 18 When ERCOT NotificationBroker tries to send a message to MP, if can not connect, timeout encountered or Acknowledge message indicating ‘OK’ is not received, an error condition exists and it is logged ERCOT NotificationBroker will wait a short period of time before retrying ERCOT NotificationBroker will try up to N (e.g. 3) times before flagging this as a link failure After N failed attempts, an e-mail is issued to designated MP administrator Depending upon configuration, an attempt may be made to fail over to a backup URL 18

  19. NotificationBroker: Configuration and Status 19 For each MP: • URL identifying the primary address to which notifications are to be sent • Backup URL can be specified (optional) • URLs are obtained from MP through Registration, stored in a directory server • Flag to indicate whether ERCOT should automatically fail over from the primary URL to the backup URL, or backup URL to the primary URL • Administrator e-mail alias to which errors will be reported For each MP-specified URL the NotificationBroker maintains a link status: • ACTIVE • STANDBY • FAILED • UNKNOWN 19

  20. Link Failover 20 • There can be at most one ‘active’ link (i.e. as identified by primary or backup URL) at a time • Web Service provided (change ActiveNotificationURL) to allow an MP to designate whether primary or backup URL should be used for notifications (see section 8.2.4) • URL definitions can only be changed through Registration process, there is purposely NOT a web service to change this • Configuration flag is used to decide whether ERCOT should automatically fail over from the primary URL to the backup URL, or backup URL to the primary URL in the event of a failure (MP will define this depending upon their implementation) • All switchovers are reported to designated administrative e-mail address 20

  21. Status Check ‘Heartbeat’ 21 • There may be long periods of time where there may be no notifications sent, but it is still important to verify that the notification ‘link’ is functioning • Need to be able to check status notification links to MPs • Added ‘StatusCheck’ notification type, which is effectively used as a ‘heartbeat’ message (see section 5.3.1) • MPs should simply reply to StatusCheck notification message with Acknowledge, but no other action is required 21

  22. Test Notification Interface 22 Web Service is provided (create TestNotification) to allow an MP to request that a test notification be sent (see section 8.2.3) Notification payload desired is supplied as the payload of the request 22

  23. Notification Media 23 Internet: • May be sent over the Internet using security that is ‘symmetric’ with the security scheme described by the External Interface Specification • MP must allow ERCOT to connect to their notification server • All connections use SSL ERCOT WAN: • Used for ICCP • Decision by ERCOT NOT the use WAN for notifications in Nodal, notifications should use Internet 23

  24. WAN availability • Access on public internet is a given • The bandwidth usage of the web services ERCOT exposes is an unknown and ERCOT plans to host these on the public internet.  The private network will be used exclusively for ICCP traffic.  Once we have measured the bandwidth usage of web services and eliminated the possibility of impact on ICCP we will then consider hosting web services on the WAN also. • However, while the WAN upgrades may mean that there is very little impact, we still have to wait and see what the bandwidth actually is • The notification traffic on the Private Network is probably less than 0.5% of the total ICCP traffic • The security on the private network is much stronger, and easier to • implement • The bandwith of the public internet is subject to large fluctuations, and wide scale hacks into DNS, SMTP, and www sites could disable large sections of the web • eployments of Non-Spin, and LAAR responsive  services are done through xml notifications, and these services usually precede or are preceded  by emergency notifications. 24

  25. Potential Settlements Interfaces • Implementing the get report into MID-MIR, which allows access to preconstructed reports • Will not offer a service to generate ad hoc reports • Open MID picture below 25

  26. Q&A Received so far 1/3 • Q. Will the compress/encode be implemented for EDS? • A. Yes. It is in the sand box now. • Q. Does EDS run on the sandbox?  If so, is structured testing on an EDS server and unstructured in the sandbox, or can we do both in the sandbox. • A. The sandbox provides a loop back. EDS will have actual back end integration. These will exist in parallel for some period of time. • Q. What version of Header/Request/Payload/etc objects will we use for testing?  I see several versions with _YYYY_MM in the package name.  As new services are released in the sandbox, if it necessitates changes to the Header/Request/Payload objects will these remain backward compatible.  Meaning everything in _2007_01 will work in _2007_02.  Will the use of Header->revision allow more than one set of classes to be used? • A. We will need to clearly identify which package/version will be used for specific tests. Right now, changes are being made largely on a reactive basis to changes in MMS. In the future it is hoped that change could be better managed. It would be the intent to support more than one version at a time after go live as needed to support necessary evolution in a more strategic manner. For EDS releases, we only want to support one version at a time, although we recognize that cases may arise to force us to support multiple versions at some point in the project. • Q. If an MP has the Three Part Offer working in an in-house production quality setup, should adding additional services (inc/dec, cop, output schedule) be as easy as pointing to an additional url and feeding it the correct xml? • A. Yes. The TPO is the most complex XML. The same URL is used for all. 26

  27. Q&A Received so far 2/3 • Q. Can ERCOT provide wsdl, xsd, xml, etc. files in addition to the interface or other documents?  I currently have to cut/paste them. • A. Yes. The WSDL and XSDs will be posted. We should do it in the same ZIP file with the document. It was not the intent that you would need to cut them out of the specification. We will be sure to post when version 1.02 comes out. • Q. On p15 of the EDS3 handbook there is a script where we submit, it’s rejected, and I guess we fix it and try again.  Will we get all of the XML we need for phase I of EDS3, including the XML that’s rejected and the subsequent XML that’s accepted (each iteration). • A. The actual XML examples will need to come from the IRT group. The examples provided were for the purposes of the API document, not EDS testing. For example, the XML examples would need to be enclosed in a BidSet as a part of a message, use correct dates, use appropriate QSE IDs, use appropriate resources, etc. • Q. Reading the API doc, I see where there are several ways to receive the status of a BidSet.  When we submit, we can get a CANCEL, • ERROR, SUBMITTED, REJECTED(?).  After we submit, we get a pending(?) notification and then an accepted/errors notification.  Is it possible to get a sample of one of everything so I can figure out how to keep whoever submits aware of the status of all mRIDs. • A. When you submit (see sections 3.5.1, 3.5.3), you can get ERROR or SUBMITTED. Cancels are described in section 3.5.4, noting that there may be details related to ABB that need to be finalized. After that you can bet a notification with ERROR, PENDING, CANCELED, ACCEPTED or REJECTED. Unfortunately we don’t have samples for everything. • Q. Is the hour a part of the MRID?  I see hours on them in the API doc (in the section that describes the repeated hour 2, 2.3.4) but not in the BidSet status XML I copied from the API doc. • A. The hour can be appended onto an mRID the reference a specific hour. For example, the hour could be appended to the mRID on a ‘get’ request for a specific hour. 27

  28. Q&A Received so far 3/3 • Q. If I use the external id when I submit, will I get it back in the <BidSet> status xml?  Submit response and subsequent notifications. • A. Yes, that is the intent. • Q. When I got boomerang and tpo working, ERCOT provided additional certificates via email.  Could ERCOT provide documentation that describes what the truststore and keystore are for, and describe what each cert in these stores does and how each should be procured. • A. There is a good explanation of these in the following Java Security book: • http://books.google.com/books?id=eqFZUksRDcMC&printsec=frontcover#PPP1,M1 • But a prefect place for a quick explanation is at IBM Web Sphere web site: • http://www.ibm.com/developerworks/websphere/techjournal/0502_benantar/0502_benantar.html#sec2 • Q. If we install a web server and axis, and I modify the client supplied in the DAM transaction processing power point presentation to point to it instead of http://ws.apache.org:5049/axis/services/echo and it worked, would that mean I have the foundation for a web services server?  I also see that exact client on the axis user guide web site. • A. Yes. You have identified the source of the code. • Q. What is the purpose of the Notification wsdl and xsd you provided in the External Interfaces Update on June 7th? • I fixed a typo and made some java, not sure what to do with it. • A. The WSDL and XSD should be used when implementing your notification listener. These files define a simplification over WS-Notifications, as well as adding the Acknowledge message. As noted on past calls, ERCOT has decided not to provide any example code as a matter of policy. 28

More Related