TTCN-3 based Implementation of ETSI TISPAN IMS Benchmark

1. TTCN-3 based Implementation of ETSI TISPAN IMS Benchmark George Din, Diana Vega, Razvan Petre, Andreas Hoffman Fraunhofer Institute for Open Communication Systems

2. Contents • Motivation • What do we test? • How do we test? • Implementation details • Benchmark Results • Conclusions TTCN-3 User Conference, 30.05.2007

3. Motivation • Goal - performance benchmark for IMS components • Performance and scalability testing of all IMS and related components with simulated real-world traffic • Measurement and analysis of important QoS parameters • Why • Creation of objective means to compare overall IMS of different systems by performance (and price) • Check ability of hardware/software to run the IMS • How • Define standard scenarios and traffic models for the work load • Define the metrics to be measured • Standardize the test procedure, the test parameters and the Benchmark test report • Where • Standardization of IMS benchmarking at ETSI TISPAN WG6 • Version 1.0 of IMS benchmarking standard has been released. TTCN-3 User Conference, 30.05.2007

4. SGF IWF SLF IBCF UPSF Charging Functions AS BGCF Network Attachement Subsystem P-CSCF MRFC I/S-CSCF Other IP Networks PSTN / ISDN MGCF MRFP UE UE IP Transfer (Access and Core) T-MGF I-BGF System under Test Fokus Open IMS Rf / Ro IMS Core Ut Sh Dh Rf/Ro ISC/Ma Iw Dx Cx Ib Mx Mw Ic e2 Ml Mx Mk Mr Mw Mj Mx Mg Gq’ Ie Gq’ Mp Mn Gm Resource and Admission Control Subsystem TTCN-3 User Conference, 30.05.2007

5. Use case 2 (e.g.: session setup) Use case 1 (e.g.: registration) Benchmark Test Test Report Scenario 1.1 (e.g. user not registered) Scenario 2.1 (e.g. successful call) Disclaimer Benchmark Test Run X% of scenario 1.1 y% of scenario 2.1 Scenario 1.2 (e.g. already registered) z% of scenario 2.2 Traffic Profile Message flow Traffic Set Design objectives Metrics & graphs to collect SuT parameters Scenario 2.2 (e.g. rejected call) Preamble Registration Phase Background Load SuT Characteristics Benchmark Information Model Message flow Design objectives Metrics & graphs to collect SuT configuration and parameters SuT parameters Test System configuration and parameters Metrics and graphics with Design Objective Capacity (overall and for each scenario) Initial load Metrics and graphics with SuT resource consumption Step duration Step Increase Amount Observations and interpretations Number Of Steps Exceptions Other parameters TTCN-3 User Conference, 30.05.2007

6. Client 1 IMS CN Client1 CN Client2 REGISTER REGISTER REGISTER REGISTER ACK ACK BYE BYE INVITE INVITE 401 200 OK 401 200 OK 200 OK 200 OK 100 Trying 180 Ringing 200 OK 200 OK 180 Ringing Client 2 IMS CN Scenario: Successful Call Registration Section Tear Up Tear Down RingingTime TalkingTime TTCN-3 User Conference, 30.05.2007

7. BYE BYE ACK INVITE ACK INVITE TRT-SES2 100 Trying 200 OK 200 OK 200 OK 200 OK 180 Ringing 180 Ringing Scenario: Successful Call Section Tear Up Tear Down Client2 Client1 CN RingingTime TRT-SES1 TalkingTime TRT-REL1 TTCN-3 User Conference, 30.05.2007

8. Benchmark Procedure Stop Condition Met Estimated Maximum Load Load Steady Time To Stop Condition Fluctuations, Steady state? Load Ramp Stability Or Settling Time Linear, Immediate? Initial Call Load Rate Increase Stability? No Load Initial Load Plus Load Plus Load Plus Load Plus Load TTCN-3 User Conference, 30.05.2007

9. Module for Instant Messaging Scenarios Module for Registration Scenarios Module for Calling Scenarios Data Templates Data Templates Data Templates Scenario-dependant Scenario-Independent Imports from Common Types Benchmark Constants Common Templates Common Behavior Benchmark Parameters TTCN-3 ATS Architecture User State Machine Benchmark Testcase TTCN-3 User Conference, 30.05.2007

10. Component Sender Component Handler 1 Component Handler 2 Users Database Users Database usrN+2 usr2N usr1 usr2 usrN usrN+1 User 1 stack User 2 stack User N stack Test Adapter Load Generator Test Configuration SuT TTCN-3 User Conference, 30.05.2007

11. User Concept • A user is a state machine running a scenario • A user may be „callee“ or/and „caller“ • A user may create > 1 calls • A user may be reused to create other calls • A user may call randomly any other user TTCN-3 User Conference, 30.05.2007

12. Simultaneous Scenarios Same user involved in more than another instance of same scenario Same user involved in different scenario messaging calling calling calling TTCN-3 User Conference, 30.05.2007

13. Population Clusters all other scenarios like messaging, calling, reregistration deregistration (registerred and not involved in any other scenario) registration (not registerred yet) b a c TTCN-3 User Conference, 30.05.2007

14. TTCN-3 Message Types // general type for a SIP Request // can be extended by any // other request type (i.e. INVITE_Request) type record Request { integertransactionId, RequestLine requestLine, MessageHeader msgHeader, charstringmessageBodyoptional } type set MessageHeader { Authorization authorization optional, CallId callId optional, Contact contact optional, CSeq cSeq optional, Expires expires optional, From fromField optional, RecordRoute recordRoute optional, Route route optional, ServiceRoute serviceroute optional, To toField optional, Via via optional, MaxForwards maxForwards optional, ContentLength contentLength optional, WwwAuthenticate wwwAuthenticate optional, Event event optional } // [20.7 RFC2617 3.2.2] type record Authorization { FieldName fieldName(AUTHORIZATION_E), //Credentials body charstring body optional } TTCN-3 User Conference, 30.05.2007

15. TTCN-3 State Handling alt { // P-CSCF --------REQUEST--------> User [] p2SUT.receive (TRequest) -> value request { // P-CSCF --------INVITE--------> User if (match(request, Request_INVITE_r)) { } // P-CSCF --------MESSAGE-------> User else if (match(request, Request_MESSAGE_r)) { } // P-CSCF --------ACK--------> User else if (match(request, Request_ACK_r)) { } ………………. } // P-CSCF --------RESPONSE--------> User [] p2SUT.receive (TResponse) -> value response { // P-CSCF -------- 200 OK --------> User if (match(response, Response_200_r)) { if (response.msgHeader.cSeq.seqMethod == "INVITE") {} else if (response.msgHeader.cSeq.seqMethod == "BYE") {} ………………. else if (response.msgHeader.cSeq.seqMethod == "REGISTER") {} } // P-CSCF -------- 401 OK --------> User else if (match(response, Response_401_r)) { } ………………. } } TTCN-3 User Conference, 30.05.2007

16. Messages Creation • // P-CSCF --------RESPONSE--------> User • [] p2SUT.receive (TResponse) -> value response { • // P-CSCF -------- 200 OK --------> User • if (match(response, Response_200_r)) { • if (response.msgHeader.cSeq.seqMethod == "INVITE") { • ackReq := ACK_Request_s; • ackReq.transactionId := getNewTransactionId(); • ackReq.requestLine.method := ACK_E; • . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • ackReq.msgHeader.callId := response.msgHeader.callId; • ackReq.msgHeader.cSeq := response.msgHeader.cSeq; • ackReq.msgHeader.cSeq.seqMethod := "ACK"; • . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • ackReq.msgHeader.fromField := response.msgHeader.fromField; • ackReq.msgHeader.toField := response.msgHeader.toField; • ackReq.msgHeader.via := response.msgHeader.via; • ackReq.msgHeader.route := getServiceRoute(response); • userChannel := getChannel(response); • p2SUT.send (ackReq) to userChannel; • delTransactionId(response.transactionId); TTCN-3 User Conference, 30.05.2007

17. Processing of Messages test component Platform adaptor identify transaction state RESP message processor SUT validate update state REQ process response TTCN-3 User Conference, 30.05.2007

18. Test System Test Management (TM) Component Handling (CH) TTCN-3 Executable (TE) Coding Decoding (CD) TTCN-3 Executable (TE) System Adapter (SA) System Adapter (SA) Platform Adapter (PA) System under Test Adaptation Layer Aim:Maximize TTCN-3 system performance ! • Handles multiple parallel connections with SUT • Uses Pools of Threads and Listeners • Smaller number of • threads and context switches • mutex-es and semaphores • contended message queues TTCN-3 User Conference, 30.05.2007

19. Transactions PTC PTC Adapter SipListener SipListener SIP Messages SIP Events SIP Messages SIP Events SipProvider SipProvider ListeningPoint ListeningPoint Stack Stack SUT TTCN-3 User Conference, 30.05.2007

20. 1: request message 5: notify TE (put message in the appropriate port queue) 7: response message 4: message ready 8: send response message 9: write ready Initiation Dispatcher (Reactor) 2: read ready 6: handle message 10: write message Test Adapter Design System under Test (SUT) Test System (TS) SA Message Handlers Thread Pool Advantages: 3: read request • I/O operations are made completely separately by the processing of the message. TE • Concurrent I/O are possible on a SMP system. TTCN-3 User Conference, 30.05.2007

21. Overall Traffic TTCN-3 User Conference, 30.05.2007

22. CPU Monitoring on SuT TTCN-3 User Conference, 30.05.2007

23. Distribution of SAPS TTCN-3 User Conference, 30.05.2007

24. Average Delay for Invite Requests TTCN-3 User Conference, 30.05.2007

25. Conclusions • TTCN-3 specification has a big impact on the performance of the hole test system • Using a component to simulate more than one user • Using a component sender • Using a tree like mechanism for matching the messages • Efficient Adaptation Layer Design has an important role in the overall performance • Smart codec/decoder system • Non-blocking I/O for protocol stack • Optimized runtime for specific needs of the benchmark TTCN-3 User Conference, 30.05.2007

26. Thank you !Time for questions … TTCN-3 User Conference, 30.05.2007