1 / 25

The unresolved problem of system testing Requirement-based testing

Bridging the Gap between logical requirements-based Test Cases and physical Test Data Generation Presentation for Swiss Testing Day by Harry M. Sneed, ANECON, Vienna. The unresolved problem of system testing Requirement-based testing From Requirements to logical Test Cases

frankharrell
Download Presentation

The unresolved problem of system testing Requirement-based testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Bridging the Gap between logical requirements-based Test Cases and physical Test Data GenerationPresentation for Swiss Testing Dayby Harry M. Sneed, ANECON, Vienna • The unresolved problem of system testing • Requirement-based testing • From Requirements to logical Test Cases • From logical Test Cases to a Test Design • From logical Test Cases to Test Data • From logical Test Cases to Test Results • Generating Test Data • Validating Test Results • Can the Gap between the test concept and test execution be bridged???

  2. SWISS-1 The Unresolved Problem of System Testing • At the top of system testing there are logical test cases formulated in natural language based on the requirement specifications. • At the bottom of system testing there are physical test data objects such as HTML pages, XML interfaces and SQL databases. • The problem in system testing is how to go from the logical test case level to the physical test data, i.e. to bridge the gap between test case specification and test data generation.

  3. SWISS-2 The Miracle of System Testing Requirement Document Requirement Analysis Logical Test Cases Test if when amount on stock exceeds minimum stock level a supply order is produced Test Data Miracle <supply_order> <supply_id type = “integer“/> <supply_article type = “integer“/> <supply_name type = “string“/> <supply_amount type = “integer“/> </supply_order> If article_stock<minimum_stock { create supply_order where supply_id = asupply_id + 1; supply_article = article_id; supply_name = article.name; supply_amount = supply.stock * 2; } Design Document Data Analysis Data Schema Objects & Attributes

  4. SWISS-3 From Requirements to Test Data English/German Human Interaction Requirement Document Tester Tester Input Objects SQL XML HTML 2 3 1 Text Analysis Logical Test Cases Object Assignment Database Schemata Interface Schemata GUI Definitions Extracts Logical Test Cases With User Objects & Preconditions Script 4 Test Data Procedures Value Assignment Pre Asserts Human Transactions 5 Existing Data Test Data Generator Value Tables GUI Inputs DB Tables Interface Files HTML / XSL SQL XML/WSDL

  5. SWISS-4 Steps to Test Data Generation The TestAnalyzer extracts logical test cases from the requirement text - one for the object action, object state and object condition. Tester refines the test cases by adding pre conditions to each test case. The precondition refers to the objects used by the system as inputs. Tool displays all logical objects from the test cases and all physical input objects from the data model. The tester assigns logical objects to physical objects. Tool generates an assertion procedure for each physical object. Tool displays attributes of all physical objects. The tester assigns them value domains, specific values, relations or expressions. The tool generates tables of test data. Tool joins generated test values with existing test data to generate system inputs - GUI contents, data base tables and messages. 1 2 3 4 5

  6. SWISS-5 From Requirements to Test Results English/German Human Interaction Requirement Document Data Model Tester Tester SQL XML/WSDL HTML 2 3 1 Text Analysis Logical Test Cases Object Assignment Database Schemata Interface Schemata GUI Definitions Extracts Logical Test Cases Logic Test Cases with User Objects & Post Conditions Script 4 Test Data Procedures Value Assignment Post Asserts Human Transactions 5 Existing Data Test Result Validator Value Tables GUI Inputs DB Tables Interface Files HTML / XSL SQL XML/WSDL

  7. SWISS-6 Steps to Test Result Validation The TestAnalyzer extracts logical test cases from the requirement text - one for the object action, object state and object condition. Tester refines the test cases by adding post conditions to each test case. The precondition refers to the objects used by the system as outputs. Tool displays all logical objects from the test cases and all physical output objects from the data model. The tester assigns logical objects to physical objects. Tool generates an assertion procedure for each physical object. Tool displays attributes of all physical objects. The tester assigns them value domains, specific values, relations or expressions. The tool generates tables of expected values. Tool compares system outputs with previous outputs and with the expected values. 1 2 3 4 5

  8. SWISS-7 Refinement of Test Cases 2

  9. SWISS-8 Assigning Logical Objects to Physical Objects 3 <Article> <Article_id type = “integer“/> <Article_name type = “string“/> <Article_stock type = “integer“/> <Minimum_stock type = “integer“/> <Supply_stock type = “integer“/> </Article> <supply_order> <supply_id type = “integer“/> <supply_article type = “dec“/> <supply_name type = “string“/> <supply_amount type = “integer“/> </supply_order> Output Input

  10. SWISS-9 Assigning Value Domains to Physical Data Fields Article_Ids Begin = 0 Internal = +10 4 Article Integer String Integer Integer Integer Integer Integer String Integer Article_id Article_name Article_stock Minimum_stock Supply_stock Supply_id Supply_article_id Supply_name Supply:amount Article_Names Book Magazine CD_Disk Supply_Order Article_Stock Minimum = 50 Maximum = 9999 Minimum_Stock Constant = 50

  11. SWISS-10 Original Requirements Text [RF 41] Notification can be divided into action and non-action (=information) notification. [RF 42] In case of an “action needed” notification, the system provides a link to the place where the action is needed. [RF 43] Notifications are added to a notification interface, which contains information about The reason of the notification The kind of the notification The type of the notification The date/time of the notification If action needed: A link to the place where the action is needed. [RF 44] The user is able “take” a notification (locked for others), do the action and mark the notification as “finished”. If the action is finished, the user name is stored by the notification. [RF 45] If at least one provider changed his feed data e.g. changing starting time, a notification is added to the notification table (see [RF 43]) so that each user can see the notification. The notification is only added to the notification table if the data is unequal to the current event data (see also [RF 30]).

  12. Requirements extracted from the Requirement Text SWISS-11 <Requirement id = "RF_43"> <RequirementHeader> <RequType>Function</RequType> <RequOwner>your_name</RequOwner> <RequStatus>defined</RequStatus> <RequLabel>RF_43</RequLabel> </RequirementHeader> <RequirementBody> <RequTextLine> Notifications are added to a notification interface, which contains information about</RequTextLine> <RequTextLine>* The reason of the notification</RequTextLine> <RequTextLine>* The kind of the notification</RequTextLine> <RequTextLine>* The type of the notification</RequTextLine> <RequTextLine>* The date/time of the notification</RequTextLine> <RequTextLine>* If action is needed: A link to the place where the action is needed.</RequTextLine> </RequirementBody> </Requirement> <Requirement id = "RF_45"> <RequirementHeader> <RequType>Function</RequType> <RequOwner>your_name</RequOwner> <RequStatus>defined</RequStatus> <RequLabel>RF_45</RequLabel> </RequirementHeader> <RequirementBody> <RequTextLine> If at least one provider changed his feed data e.g. changing starting time, a notification is added to the notification table (see [RF_43]) so that each user</RequTextLine> <RequTextLine>The notification is only added to the notification table if the data is unequal to the current event data (see also [RF_30]).</RequTextLine> <RequTextLine>Provider B has an other starting time than provider A. The user chose Provider A as primary source, so the event time is the time of Provider A.</RequTextLine> <RequTextLine>If Provider B updates his time to the time of the event, no notification is added.</RequTextLine> <RequTextLine>If Provider B sends a time different to the event time, a notification is send.</RequTextLine> </RequirementBody> </Requirement>

  13. SWISS-12 Test Cases extracted from the Requirement Text TestCase-Id BaseType Requirement Relation TestType TestCase Description -------------------------------------------------------------------------------------------------------------------------------------------------------- DFI0163 Require RF_43 perform action Notifications are added to a notification interface, which contains information about: The reason of the notification The kind of the notification The type of the notification The date/time of the notification DFI0164 Require RF_43 checks rule If action is needed a link to the place where the action is needed. -------------------------------------------------------------------------------------------------------------------------------------------------------- TestCase-Id BaseType Requirement Relation TestType TestCase Description -------------------------------------------------------------------------------------------------------------------------------------------------------- DFI0165 Require RF_44 checks rule If the action is finished, the user name is stored in the notification. -------------------------------------------------------------------------------------------------------------------------------------------------------- TestCase-Id BaseType Requirement Relation TestType TestCase Description -------------------------------------------------------------------------------------------------------------------------------------------------------- DFI0166 Require RF_45 checks rule If at least one provider changed his feed data changing starting time, a notification is added to the notification table (see [RF_43]) so that each user can see the notification. DFI0167 Require RF_45 checks rule The notification is only added to the notification table if the data is unequal to the current event data (see also [RF_30]). DFI0168 Require RF_45 checks rule The user choose Provider A as primary source, so the event time is the time of Provider A. DFI0169 Require RF_45 checks rule If Provider B updates his time to the time of the event, no notification is added. DFI0170 Require RF_45 checks rule If Provider B sends a time different to the event time, a notification is sent.

  14. SWISS-13 Test Cases in XML Format with Objects TestCase Id = "DFI0167" > <Requirement Id = "45" > <RequirementType>Require</RequirementType> <RequirementName>RF_45</RequirementName> <Priority>High</Priority> <Responsible>Analyst</Responsible> </Requirement> <TestCaseInfo> <TestCaseType>rule</TestCaseType> <TestGoal>check:The notification is only added to the notification table if the data is unequal to the current event data (see also [RF_30]).</TestGoal> <TestEnvironment>MS-Windows</TestEnvironment> <TestObjekte> <TestObject>notification</TestObject> <TestObject>notification_table</TestObject> <TestObject>event</TestObject> </TestObjects> </TestCaseInfo> <TestCaseConditions> <PreceedingCase>DFI0166</PreceedingCase> <Trigger>Notification is received</Trigger> <PreConditions> <PreCondition> the data is unequal to the current event data(see also [RF_30]). </PreCondition> </PreConditions> <PostConditions> <PostCondition> notification is only added to the notification table </PostCondition> </PostConditions> </TestCaseConditions>

  15. SWISS-14 Physical Objects defined in SQL Database Schema Oracle Data Tables /*==============================================================*/ /* Table: BASEEVENT */ create table DFI.BASEEVENT ( ID NUMBER(18) not null, EVENTNAME varchar(250), STARTDATETIME date not null, CANCELLED date, UPDATEABLE int not null, EVENTDATE date not null, PROGRAM varchar(100), ERRORCODE number, ERRORTEXT varchar(2000)) /*==============================================================*/ /* Table: FEEDEVENT */ create table DFI.FEEDEVENT ( ID NUMBER(18) not null, BASEEVENT_ID NUMBER(18), FEEDEVENTID varchar(100), EVENTNAME varchar(250), STARTDATETIME date, CANCELLED date) /*==============================================================*/ /* Table: NOTIFICATION */ create table DFI.NOTIFICATION ( ID NUMBER(18) not null, EVENTNAME varchar(30), EVENTDATE date not null, EVENTSPONSOR CHAR(10))

  16. SWISS-15 Physical Objects defined in XML Input Data Schema <xs:element name="Competition"> <xs:complexType> <xs:sequence> <xs:element ref="Stages"/> </xs:sequence> <xs:attribute name="id" type="IdType" use="required"/> <xs:attribute name="sponsor" type="xs:string"/> <xs:attribute name="name" type="xs:string"/> <xs:attribute name="startDate" type=„date"/> <xs:attribute name="endDate" type=„date"/> </xs:complexType> </xs:element> <xs:element name=„Round"> <xs:complexType> <xs:attribute name="number" type="IdType" use="required"/> <xs:attribute name="name" type="xs:string"/> <xs:attribute name="startDate" type=„date"/> <xs:attribute name="endDate" type=„date"/> <xs:attribute name="legs" type="IdType" default="1"/> <xs:attribute name=„replaysPossible" type=„boolean" use="required"/> </xs:complexType> </xs:element>

  17. SWISS-16 Assignment of Logical Objects to Physical Objects

  18. SWISS-17 Initial Assignment of Values to Database Attributes file: DFI_Test_02; if ( object = "BASEEVENT" ); assert new.ID = “4711"; assert new.EVENTNAME = “XXXXXXXXXXXXXXXXX"; assert new.STARTDATETIME = “2006-10-07"; assert new.CANCELLED = "2006-10-05"; assert new.UPDATEABLE = "99999999999999"; assert new.EVENTDATE = “2006-10-07"; assert new.PROGRAM = “YYYYYYYYYYYYYYYYYYY"; assert new.ERRORCODE = “77"; assert new.ERRORTEXT = “ZZZZZZZZZZZZZZZZZZ“; endObject; if ( $object = „ComFEEDEVENT" ); assert new.ID = “4712"; assert new.BASEEVENT_ID = BASEEVENT.ID; assert new.FEEDEVENTID = FEEDEVENT.ID; assert new.EVENTNAME = BASEEVENT.EVENTNAME; assert new.STARTDATETIME = “2006-10-07"; assert new.CANCELLED = "2006-10-05"; endObject; if ( $object = “NOTIFICATION" ); assert new.ID = “4713"; assert new.EVENTNAME = “XXXXXXXXXXXXXXXXX“; assert new.EVENTDATE = “2006-10-15"; assert new.EVENTSPONSOR = “YYYYYYYYYYYYYYYYYYYY"; endObject; End DFIDB;

  19. SWISS-18 Specific TestCase related Assignment of Values file: DFIDB; if ( $Test = " DFI0166 " ); if ( $object = "BASEEVENT" ); assert pre.ID = “4711"; assert pre.EVENTNAME = “Manchester/Liverpool Match"; assert pre.STARTDATE = “2006-10-07"; assert pre.CANCELLED = "2006-10-05"; assert pre.UPDATEABLE = „6856"; assert pre.EVENTDATE = “2006-10-07"; assert pre.PROGRAM = “English football league"; assert pre.ERRORCODE = “77"; assert pre.ERRORTEXT = “Match result not verified“; endObject; if ( $object = „Competition" ); assert pre.ID = BASEEVENT.ID; assert pre.Sponsor = “McDonalds“; assert pre.name != BASEEVENT.EVENTNAME; assert pre.StartDate != BASEEVENT.STARTDATE; assert pre.EndDate = “2006-10-07"; endObject; if ($object = “NOTIFICATION" ); assert post.ID = Competition.ID; assert post.EVENTNAME = Competition.name; assert post.EVENTDATE = Competition.StartDate; assert post.EVENTSPONSOR = Competition.Sponsor; endObject; endTest;

  20. <TestProcess id = "DFI_Test_02" name = "Automatic_Data_Feed_Test" type = "Batch"><TestObjectives> <TestObjective>To test the Automatic Data Feed Function</TestObjective> <TestObjective>To test if the XML files from the Providers are processed correctly</TestObjective> <TestObjective>To test if the DFI Database is updated correctly</TestObjective> <TestObjective>To test if the correct Notifications are sent</TestObjective></TestObjectives><TestEnvironment> <Hardware>Sun-Sparc Server</Hardware> </TestEnvironment><TestPrerequisites> <TestPrerequisite>DFI Database must be initialized with Testdata</TestPrerequisite> <TestPrerequisite>One or more XML Files must be generated</TestPrerequisite> <TestPrerequisite>DFI System must be put into operation</TestPrerequisite> <TestPrerequisite>XML Files must be copied into the system queue</TestPrerequisite> </TestPrerequisites><TargetRequirements> <TargetRequirement>RF-01-Existing_Functionality_is_covered</TargetRequirement> <TargetRequirement>RF-45-Notification_is_generated_if_at_least_one_Provider_changes_his_feed_data</TargetRequirement> <TargetRequirement>RF-46-User_can_restrict_data_by_multiple_Filters</TargetRequirement></TargetRequirements><TargetUseCases> <TargetUseCase>UC-01-Import_Data_Feed</TargetUseCase> <TargetUseCase>UC-01-01-Propagate_new_Feed_Event</TargetUseCase></TargetUseCases><TargetObjects> <TargetObject>BaseEvent</TargetObject> <TargetObject>FeedEvent</TargetObject> <TargetObject>Notification</TargetObject> <TargetObject>Rule</TargetObject> <TargetObject>TeamLeague</TargetObject></TargetObjects><TestResults> <TestResult>Feed Files have been parsed and separated</TestResult> <TestResult>Feed Events have been added</TestResult> <TestResult>Feed Events have been updated</TestResult> <TestResult>Event Notifications have been stored</TestResult> <TestResult>Update Notifications have been stored</TestResult> <TestResult>UpdateEvent Notification has been sent</TestResult></TestResults> Assignment of Use Cases to a Test Process SWISS-19

  21. Assignment of Use Case Steps to Process Steps SWISS-20 <ProcessSteps> <sequence> <ProcessStep name = "Generate_XML_Feed_File"> <selection> <ProcessStepAction>Generate an XML feed file</ProcessStepAction> </selection> </ProcessStep> <ProcessStep name = "Load_XML_feed_File"> <selection> <ProcessStepAction>Load the XML feed file</ProcessStepAction> </selection> </ProcessStep> <ProcessStep name = "System_parses_XML_feed_File"> <selection> <ProcessStepAction>System parses the XML Feed file</ProcessStepAction> </selection> </ProcessStep> <ProcessStep name = "System_stores_Parse_Results"> <selection> <ProcessStepAction cond = "if no parsing errors">System stores good Parse results for processing</ProcessStepAction> <ProcessStepAction cond = "if parsing errors">System stores bad Parse results as error file</ProcessStepAction> </selection> </ProcessStep> <ProcessStep name = "Process_Events" cond = "if no parsing errors"> <selection> <ProcessStepAction cond = "if new event and no other event exists">System creates new event</ProcessStepAction> <ProcessStepAction cond = "if new event and no other event exists">System adds Feed Event reference</ProcessStepAction> <ProcessStepAction cond = "if new event and no other event exists">System sets Base Event properties</ProcessStepAction> <ProcessStepAction cond = "if new event and no other event exists">System checks active Rule</ProcessStepAction> <ProcessStepAction cond = "if new event and no other event exists">System checks if active Rule is fulfilled</ProcessStepAction> <ProcessStepAction cond = "if new event and no other event exists and rule not fulfilled ">System sends Create Notification to Bookie</ProcessStepAction> <ProcessStepAction cond = "if new event and no other event exists and rule fulfilled">System updates base event attributes</ProcessStepAction> <ProcessStepAction cond = "if new event and no other event exists and rule fulfilled">System sends non-action Notification to Bookie</ProcessStepAction> </selection> </ProcessStep> </sequence> </ProcessSteps>

  22. SWISS-21 Assignment of Logical Test Cases to a Test Process <TestCases> <TestCase> <TestCaseId>DFI0166</TestCaseId> <TestCaseDescription>If at least one provider changed his feed data e.g. changing starting time, a notification is added to a notification interface </TestCaseDescription> <TestRequirement>RF-45</TestRequirement> <TestUseCase>UC-01-01</TestUseCase> </TestCase> <TestCase> <TestCaseId>DFI0167</TestCaseId> <TestCaseDescription>The notification is only added to the notification table if the data is unequal to the current event data </TestCaseDescription> <TestRequirement>RF-45</TestRequirement> <TestUseCase>UC-01-01</TestUseCase> </TestCase> <TestCase> <TestCaseId>DFI0168</TestCaseId> <TestCaseDescription> The user choose Provider A as primary source, so the event time is the time of Provider A</TestCaseDescription> <TestRequirement>RF-45</TestRequirement> <TestUseCase>UC-01-02</TestUseCase> </TestCase> </TestCases> <TestEndCriteria> <TestEndCriterium>All Feed File variations have been processed</TestEndCriterium> <TestEndCriterium>All invalid Feed data has been recognized</TestEndCriterium> <TestEndCriterium>All target database tables have been updated</TestEndCriterium> <TestEndCriterium>All database attribute values correspond to expected values</TestEndCriterium> <TestEndCriterium>All feed data has been matched</TestEndCriterium> <TestEndCriterium>All unmatched data has lead to a notification</TestEndCriterium> <TestEndCriterium>All update events have been sent to V5</TestEndCriterium> <TestEndCriterium>All notification data values correspond to expected values</TestEndCriterium> <TestEndCriterium>All notifications have been acknowledged by the bookie</TestEndCriterium> </TestEndCriteria>

  23. SWISS-22 Test Process Data Flow Validiere mit WSDLVal Generiere mit XMLGen XML Interface WSDL Interface Data Feed UC-01 UC-16 UC-17 Feed File Update Event XML BaseEvent Delivered Notification Team League Rule FeedEvent Delivered FeedEvent Notification Type Generiere mit CSVGen Validiere mit CSVVal

  24. Test Result Validation against the Post Conditions SWISS-23

  25. SWISS-24 The Solution to System Testing • The solution to the system testing problem lies in the association of objects and events at the logical level to objects and events at the physical level. • The logical objects referred to in the system requirements must be linked to the physical objects of the system architecture – the GUIs, interfaces and database tables. • The logical events referred to in the system requirements, e.g. use cases, must be linked to the physical events of the system – the online transactions and batch processes.

More Related