1 / 36

System Test Evaluation

STV. TVAL. System Test Evaluation. STV. Location of Test Evaluation Overview of ANSI/IEEE Test Documents System Test Results Test Plan Table of Contents Test Case Specification Test Procedure Specification Test Log Contents Test Measurement Test Metrics Test Coverage Measurement

juanap
Download Presentation

System Test Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. STV TVAL System Test Evaluation STV Location of Test Evaluation Overview of ANSI/IEEE Test Documents System Test Results Test Plan Table of Contents Test Case Specification Test Procedure Specification Test Log Contents Test Measurement Test Metrics Test Coverage Measurement Error Coverage Error Density Code Coverage UseCase Coverage Defect Analysis System Test Reports System Test Log File Generation Report File Content Report File Comparison Report File Audit Report Test Incident Report Test Management Reports Error Report Error Frequency Report Error Density Report Test Coverage Report Test Status Report 1 15 2 16 3 17 18 4 19 5 20 6 21 7 8 22 9 23 24 10 11 25 26 12 13 27 14 28

  2. STV TVAL-1 Location of Test Evaluation Init Test planning Test Plan Test designing Test Design Test spec. Test Cases Test execution Test- results Test evaluation Closure Test- docus. Results of theTest Evaluation

  3. STV TVAL-2 Overview of the ANSI/IEEE Test Documents Projectdoc. Testitem Itemdoc. Testplan Testdesignspec. Testdesignspec. Testdesignspec. Testcasespec. Testprocspec. Testitem-Trans- mittal-Report T e s t e x e c u t i o n Document specified by this Standard Testlog Testlog Document not specifiedby this standard Testitem (not specifiedby this standard) Testincident- Report Testincident- Report Process not specifiedby this standard Testsummary- Report

  4. Test STV TVAL-3 System Test Results Test Coverage Report Test- Incident Report Test Validation Report Test Summary Report Test Log Exceptions & Error Messages Function Trace & Object States Code Branches Traversed & Functions executed Soll / Ist Compares & Deviations Test Goals Test Metrics Test Status Defect Statistics

  5. STV TDOC-4 Test Plan Table of Contents According to IEEE-Standard 829Software Test Documentation 1) Test Plan Identification Test Project Description Test Configuration Items Test Goals Test Constraints Test Strategy Test End Criteria Test Interruption Criteria Test Results Test Tasks Test Environment Test Responsibilities Test Work Plan Test Time Plan Test Risks and Contigencies Test Approval 2) 3) 4) 5) 6) 7) 8) 9) 10) 11) 12) 13) 14) 15) 16)

  6. STV TDOC-5 Test Case Specification According to IEEE-Standard 829Software Test Documentation Purpose of the Test CaseTest case IdentificationTest Objects  Requirement Specs System Design User Hand Book System Usage Guide Installation Guide Specification of the InputsSpecification of the OutputsSpecification of Test Environment  Hardware Software (OS/DBS/DCS/MW) Procedural RequirementsDependency on other Test Cases 1) 2) 3) 4) 5) 6) 7) 8)

  7. STV TDOC-6 Test Procedure Specification According to IEEE-Standard 829Software Test Documentation Purpose of the Test ProcedureTest Procedure DescriptionTest Procedural Requirements Test Steps  Test preparation Test execution Test repetition Test evaluation Test closure Test incident reporting 1) 2) 3) 4)

  8. STV TDOC-7 Test Log Contents According to IEEE-Standard 829Software Test Documentation Purpose of the Test Log Test Log Identification Description of Test Environment Test incident List Test execution protocol Test result report Test Environment protocol Test incident report  Test inputs Expected outputs Actual outputs Test anomalies Date and time of incident Test step Test configuartion Program state 1) 2) 3) 4) 5) 6) 7) 8)

  9. STV TDOC-8 Test Measurement Reality existing Errors Functionality States Code Mass Ziel Soll reported Errors Test Coverage Test Effort Test Time Ist / Soll Difference Ist reported Errors Test Coverage Test Effort Test Time

  10. STV TDOC-9 Test Metrics Error Reports Test Runs executed Test cases tested Test Effort ( Tester days ) Test machine hours used Calender days expired 0 Test Counts Test Goals

  11. STV TDOC-10 Test Coverage Measurement Test Coverage Degree Time

  12. STV TDOC-11 Error Coverage 100 % Error Reports submitted Ziel 98 % 72 % Stand Soll Ist 0 Time Found

  13. Lines of Code, Statements, Object-Points, Function-Points, Data-Points, UseCase-Points Codezeilen, Anweisungen, Object-Points, Function-Points , Data-Points Error Density Errors per 1000 Statements Fehler pro 1000 Größeneinheiten 0,003 0,003 System Size STV TDOC-12 Predicted Error Coverage

  14. STV TDOC-13 Code Coverage Report Date: 02.09.1998 Test Coverage Report System-Name: Calender User-Name: Bahr Module: Tag ( r = Missed Node-Nr. Line-Nr. Node-Name. 000 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 00022 00029 00033 00035 00039 00040 00040 00042 00044 00048 00060 00061 00062 00063 00070 00075 00075 00075 00080 00085 Konstruktor_Def/Tag Destruktur_Def/Tag Methode_Def/Tag/get_AKtivitätenliste Logic/return Methode_Def/Tag/add_Aktivität Logic/if Logic/return Konstruktor_New/tasks/Task Logic/return Methode_Def/Tag/can_Aktivität Logic/while Methode_Inv/tasks/get_Start_Zeit Metode_Inv/task/get_Ende_Zeit Methode_Inv/task/get_Aktivität Logic/if Logic/else Logic/if Logic/return Logic/while Logic/return 3 0 8 8 6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 r r r r r r r r r r r r r r r r Nodes-Total: 20 Nodes-Execution: 4 Coverage Ration 20,00%

  15. STV TDOC-14 Use Case Coverage Report Event Input data Test Errors Use-Case Open_Calender With authorized Person With unauthorized Person With an invalid week With a valid week With an invalid day With a valid day With submit With an invalid start time With an invalid end time With a missing description With an invalid project With a valid project With too many activities With an existing activity X X X X X X X X X X X X - - Error message 1 continue Error message 2 continue Error message 3 continue Error message 4 Error message 5 Error message 6 Error message 7 Error message 8 Activity accepted - - 0 0 0 0 0 0 1 0 0 1 1 2 - - Enter_Activity Delete_Activity 13 5

  16. STV TDOC-15 Defect Analysis Defect Count Severe errors Medium errors Minor errors Class Defects Employee Calender Week Day Activity Project 3 3 1 1 2 1 2 1 1 1 1 1 1 1 2 Defect Density Funct.- Points Class Lines Stmts Obj.-Points Employee Calender Week Day Activity Project 0,009 0,012 0,010 0,008 0,011 0,015 0,018 0,024 0,020 0,016 0,022 0,030 0,027 0,036 0,030 0,024 0,033 0,045 0,180 0,240 0,200 0,160 0,220 0,300 Defects Defects Defects Defects Statements Lines Obj.-Points Funct.-Points

  17. STV TDOC-16 System Test Reports System Test LogFile Generation ReportFile Content ReportFile Comparison ReportFile Audit ReportTest Status ReportTest Incident Report

  18. STV TDOC-17.1 System Test Log PROGRAMMIERSPRACHE: COBOL-85 DATUM: 31.07.92PROGRAMMNAME: P3006 SEITE: 001TESTFALL: 001-----------------------------------------------------------------STUBNAME RUFNR. HANDLUNG ASSERTION-VERLETZUNGEN-----------------------------------------------------------------KUNDEN 1 EINGABEARTIKEL 1 EINGABEARTIKEL 2 EINGABEARTIKEL 3 EINGABEVERSAND 1 AUSGABE POSTLEITZAHL SOLL: = 89000 IST: = 89001 RECHNUNGSBETRAG SOLL: > 01000.00 < 09999.99 IST: = 00999.99KUNDEN 2 EINGABEARTIKEL 4 EINGABEARTIKEL 5 EINGABEARTIKEL 6 EINGABEVERSAND 2 AUSGABE LIEFERWEG SOLL: = BAHN IST: = LASTKUNDEN 3 EINGABEARTIKEL 7 EINGABE

  19. STV TDOC-17.2 ARTIKEL 8 EINGABEARTIKEL 9 EINGABEVERSAND 3 AUSGABE STRASSE SOLL: = ROSENSTRASSE 40 IST: = ROSENSTRASSE 4KUNDEN 4 EINGABEARTIKEL 10 EINGABEARTIKEL 11 EINGABEARTIKEL 12 EINGABEVERSAND 4 AUSGABE LAND SOLL: = BAYERN IST: = HESSENBERICHT 1 AUSGABEBERICHT 2 AUSGABEBERICHT 3 AUSGABEBERICHT 4 AUSGABEBERICHT 5 AUSGABE GESAMTBETRAG SOLL: > 010000.00 < 999999.99 IST: = 000863.75-----------------------------------------------------------------ANZAHL TESTFAELLE ==========> 1ANZAHL EINGABEN ==========> 16ANZAHL AUSGABEN ==========> 9ANZAHL GEPRUEFTE ERGEBNISSE = 60ANZAHL FALSCHER ERGEBNISSE = 6ZUVERLAESSIGKEITSGRAD = 0,90

  20. STV TDOC-18.1 File Generation Report DATEIART: AS-400/ISAM DATUM: 31.07.92DATEINAME: KUNDEN SEITE: 001------------------------------------------------------------RECORD-NAME = KUNDENSATZRECORD-ANZAHL = 0400------------------------------------------------------------FELDNAME | PICTURE | INHALT------------------------------------------------------------NACHNAME | X(20) | BRAUNVORNAME | X(20) | FRITZSTRASSE | X(20) | ROSENSTRASSE 56STADT | X(20) | MUENCHENPLZ | 99999 | 89000LAND | X(10) | BAYERNGEBURTSDATUM | 9(6) | 110351GUTHABEN | S9(5).99 | 05055.50SCHULDEN | S9(5).99 | 00045.66BONITAET | 9 | 2BESTELLUNGEN | 9999 | 0088ARTIKELART (1)| X(8) | SCHAMPOOARTIKELART (2)| X(8) | HAARFETTARTIKELART (3)| X(8) | FAERBUNGLIEFERMENGE (1)| 999 | 150LIEFERMENGE (2)| 999 | 175

  21. STV TDOC-18.2 LIEFERMENGE (3)| 999 | 190LIEFERWEG | XXXX | BAHNLIEFERDATUM | 999999 | 150892RECHNUNGSBETRAG | S99,999.99 | -00,000.76------------------------------------------------------------ANZAHL SAETZE GENERIERT ==========> 400ANZAHL FELDER PRO SATZ ==========> 20ANZAHL GENERIERTER DATEN ==========> 8000ANZAHL HAUPTSCHLUESSEL ==========> 1ANZAHL FREMDSCHLUESSEL ==========> 2ANZAHL DATA-POINTS ==========> 28 DATENKOMPLEXITAETSGRAD ==========> 0,150

  22. STV TDOC-19.1 File Content Report DATEIART: AS-400/ISAM DATUM: 31.07.92DATEINAME: KUNDEN SEITE: 001VON-KEY: AAAAAAAAAAAAAAAAAAAABIS-KEY: ZZZZZZZZZZZZZZZZZZZZ------------------------------------------------------------RECORD-NAME = KUNDENSATZRECORD-NR. = 0011RECORD-KEY = BRAUN------------------------------------------------------------FELDNAME | PICTURE | INHALT------------------------------------------------------------NACHNAME | X(20) | BRAUNVORNAME | X(20) | FRITZSTRASSE | X(20) | ROSENSTRASSE 56STADT | X(20) | MUENCHENPLZ | 99999 | 89000LAND | X(10) | BAYERNGEBURTSDATUM | 9(6) | 110351GUTHABEN | S9(5).99 | 05055.50SCHULDEN | S9(5).99 | 00045.66BONITAET | 9 | 2

  23. STV TDOC-19.2 BESTELLUNGEN | 9999 | 0088ARTIKELART (1)| X(8) | SCHAMPOO ARTIKELART (2)| X(8) | HAARFETTARTIKELART (3)| X(8) | FAERBUNGLIEFERMENGE (1)| 999 | 150LIEFERMENGE (2)| 999 | 175 LIEFERMENGE (3)| 999 | 190LIEFERWEG | XXXX | BAHNLIEFERDATUM | 999999 | 150892RECHNUNGSBETRAG | S99,999.99 | -00,000.76------------------------------------------------------------RECORD-NAME = KUNDENSATZ RECORD-NR. = 0012RECORD-KEY = BRAUNIGER ANZAHL SAETZE IN DATEI ==========> 400ANZAHL FELDER PRO SATZ ==========> 20 ANZAHL DATENELEMENTE ==========> 8000

  24. STV TDOC-20.1 File Comparison Report DATEIART: AS-400/ISAM DATUM: 31.07.92DATEINAME: KUNDEN SEITE: 001------------------------------------------------------------RECORD-NAME = KUNDENSATZRECORD-NR. = 0011 STATUS: UNGLEICHRECORD-KEY = BRAUN------------------------------------------------------------FELDNAME | PICTURE | INHALT------------------------------------------------------------VORNAME | X(20) | FRITZ | SOLL | FRANZGEBURTSDATUM | 9(6) | 110351 | SOLL | 100351BONITAET | 9 | 2 | SOLL | 3SCHULDEN | S9(5).99 | 00045.66 | SOLL | 00044.77------------------------------------------------------------RECORD-NAME = KUNDENSATZRECORD-NR. = 0144 STATUS: FEHLTRECORD-KEY = KAUFMANN------------------------------------------------------------RECORD-NAME = KUNDENSATZRECORD-NR. = 0162 STATUS: FEHLTRECORD-KEY = LORENZ------------------------------------------------------------RECORD-NAME = KUNDENSATZRECORD-NR. = 0196 STATUS: UNGLEICHRECORD-KEY = MITTELMEIER------------------------------------------------------------

  25. STV TDOC-20.2 ------------------------------------------------------------FELDNAME | PICTURE | INHALT------------------------------------------------------------VORNAME | X(20) | RESI | SOLL | ROSIGEBURTSDATUM | 9(6) | 210648 | SOLL | 200648GUTHABEN | S9(5).99 | 05055.50 | SOLL | 05055.00------------------------------------------------------------RECORD-NAME = KUNDENSATZRECORD-NR. = 0240 STATUS: UNGLEICHRECORD-KEY = OBERMEIER------------------------------------------------------------FELDNAME | PICTURE | INHALT------------------------------------------------------------RECHNUNGSBETRAG | S99,999.99 | -00,000.76 | SOLL | +00,000.75------------------------------------------------------------RECORD-NAME = KUNDENSATZRECORD-NR. = 0360 STATUS: FEHLTRECORD-KEY = SCHMIDTHUBER------------------------------------------------------------RECORD-NAME = KUNDENSATZRECORD-NR. = 0399 STATUS: FEHLTRECORD-KEY = WERNER------------------------------------------------------------ANZAHL SAETZE IN SOLL-DATEI ==========> 400ANZAHL SAETZE IN IST-DATEI ==========> 396ANZAHL FEHLERHAFTER SAETZE ==========> 8ANZAHL FEHLERHAFTER FELDER ==========> 32KORREKTHEITSGRAD =====================> 0,97

  26. STV TDOC-21.1 File Audit Report DATEIART: AS-400/ISAM DATUM: 31.07.92DATEINAME: KUNDEN SEITE: 001VON-KEY: AAAAAAAAAAAAAAAAAAAABIS-KEY: ZZZZZZZZZZZZZZZZZZZZ------------------------------------------------------------RECORD-KEY ASSERTION-VERLETZUNGEN------------------------------------------------------------BRAUN VORNAME SOLL: = FRANZ IST: = FRITZ GEBURTSDATUM SOLL: = 100351 IST: = 110351 BONITAET SOLL: = 3 IST: = 2 SCHULDEN SOLL: = 00044.77 IST: = 00045.66MITTELMEIER VORNAME SOLL: = ROSI IST: = RESI

  27. STV TDOC-21.2 GEBURTSDATUM SOLL: = 200648 IST: = 210648 GUTHABEN SOLL: = 05055.00 IST: = 05055.55OPPENHEIMER KEY NICHT GEFUNDENSCHOENHUBER GUTHABEN SOLL: => 10000.00 <= 50000.00 IST: = 9999.98WALLMANN KREDIT-LIMIT SOLL: <= 00999.00 IST: = 01000.00------------------------------------------------------------ANZAHL SAETZE GEPRUEFT ==========> 400ANZAHL FEHLERHAFTER SAETZE ==========> 5ANZAHL FEHLERHAFTER DATEN ==========> 9ZUVERLAESSIGKEITSGRAD = 0,988

  28. STV TDOC-22 Test Incident Report VORFALLS- TEST- VORFALLS- VORFALLS- VORFALLS- VORFALLSBESCHREIBUNGNR. ZYKLUS TYP DATUM STATUS003 2 ABEND 02.05.94 BEHOBEN Endlosschleife bei Kundenerfassung004 3 RESULT 04.05.94 OFFEN Falsches Ergebnis im Artikelsatz005 3 TECH 05.05.94 OFFEN Deadlock-Fehler beim Zugriff auf Lieferanten006 4 UI 08.05.94 BEHOBEN System reagiert nicht auf Masiklick007 4 RESULT 08.05.94 OFFEN Falscher Rechnungs- betrag008 5 TECH 09.05.94 OFFEN RPC lieferte Exception- CODE bei Versandauftrag... ... ... ... ... ...

  29. PROD CMGT-25 Validating the Effect of Change on the Data B 2 21 0 D 4 40 0 D 4 40 0 G 7 70 0 F 6 60 0 A 1 10 0 A 1 10 0 C 3 30 0 C 3 30 0 E 5 50 0 E 5 50 0 H 8 80 0 G 7 70 0 B 2 20 0 New Data Old Data Data Validation Compare if objects are present and equal Doesn‘t Match Missing inthe New Missing inthe Old Validate one set of data against another by checking existence and matching content

  30. STV TDOC-23 Test Management Reports Error ReportError Frequency ReportError Density ReportTest Coverage ReportTest Status Report

  31. STV TDOC-24 Error Report Produkt: FIDIS Tester: Erdoes Teilsystem: ZAS Datum: 31.08.1999 Fehlergewicht: Severe Fehlertyp: Ergebnis Fehlerart: Verwechslung Fehlerquelle: Konzept Funktion: Abrechnung Auswirkung: Falsche Fehlermeldung Testfall: Abrechnungsfall_2 (Rechnungserstellung, wenn Kunde Aktien mit Vorzugsrecht kauft) Beschreibung: Bei ungültigem Rechnungsbetrag erscheint die falsche Fehlermeldung

  32. STV TDOC-25.1 Error Frequency Report TESTDOC TEST SUMMARY REPORT(by Test Cycle) DATE: 30.09.1999 FIRM: SDSSYSTEM: FIDIS SYSTEM DEFECTS PAGE: 01 Test Cycle Faults Severe Medium Minor Undefined 1.1.4 0001 0000 0001 0000 00001999KW 0001 0001 0000 0000 000098KW18 0002 0000 0001 0001 000098KW19 0001 0000 0000 0001 000098KW38 0003 0000 0002 0001 000098KW49 0021 0000 0007 0014 000098KW50 0011 0000 0002 0009 000098KW51 0006 0000 0002 0004 000098KW52 0002 0001 0001 0000 0000999999 0124 0034 0045 0045 000099KW01 0005 0001 0000 0004 000099KW02 0001 0000 0001 0000 000099KW03 0006 0000 0002 0004 000099KW04 0003 0000 0001 0002 000099KW05 0003 0002 0000 0001 000099KW06 0001 0000 0000 0001 000099KW07 0001 0001 0000 0000 000099KW08 0095 0008 0011 0076 0000

  33. STV TDOC-25.2 99KW09 0042 0002 0005 0035 000099KW10 0036 0004 0008 0024 000099KW11 0007 0000 0004 0003 000099KW12 0001 0001 0000 0000 000099KW14 0040 0003 0011 0026 000099KW15 0004 0003 0000 0001 000099KW16 0004 0000 0000 0004 000099KW17 0001 0000 0001 0000 000099KW18 0015 0003 0005 0007 000099KW19 0001 0000 0000 0001 000099KW20 0005 0000 0003 0002 000099KW22 0012 0009 0001 0002 000099KW23 0001 0000 0001 0000 000099KW24 0032 0014 0011 0007 000099KW25 0060 0003 0010 0047 000099KW26 0032 0001 0002 0029 0000 Total 0580 0091 0138 00351 0000

  34. STV TDOC-26 Error Density Report TESTDOC TEST SUMMARY REPORT(by Component) DATE: 30.09.1999 FIRM: SDSSYSTEM: FIDIS DEFECT RATES (per 1000) PAGE: 04 Component Lines Functs Obj-Pts Dat-Pts Func-Pts GEA 0.000 0.011 0.000 0.000 0.001 KOS 0.000 0.020 0.000 0.000 0.001 KMP 0.005 0.598 0.009 0.010 0.036 FIS 0.000 0.002 0.000 0.000 0.002 GEA 0.000 0.017 0.001 0.002 0.014 KMP 0.000 0.000 0.000 0.000 0.000 KOS 0.001 0.025 0.002 0.003 0.023 ZAS 0.004 0.151 0.008 0.016 0.148

  35. STV TDOC-27 Test Coverage Report TESTDOC TEST SUMMARY REPORT(by Component) DATE: 30.09.1999 FIRM: SDSSYSTEM: FIDIS TEST COVERAGE RATE PAGE: 06 Component Lines Nodes Hits Misses Ratio AAU 10124 986 745 241 0.755 ARK 5550 987 434 553 0.439 EXT 0063 008 007 001 0.875 GEA 23417 11671 6420 5251 0.550 KMP 178446 68695 32093 36602 0.467 KOS 14436 7693 5041 2652 0.655 PMI 1319 964 803 161 0.832 ZAS 32518 19577 9362 10215 0.478 Total 265873 110581 54905 55676 0.496

  36. STV TDOC-28 Test Status Report PLOENZKE/SES AS-400 TEST STATUS SITE: BUDAPESTAUTHOR: SNEED BERICHT DATE: 02.04.93SUBJEKCT: STATUS OF COBOL PROGRAM TEST IN BUDAPEST PAGE: 1 OF 2 NR NR NR NR NR NR CONF CORR COVRPROGRAM TESTER STATUS LINES STMTS BRCHS FILES FKPT ERRS RATE RATE RATEGFL000 FUELE TESTED 1986 544 292 9 160 11 0,91 1,00 0,91(P3053) (12) 30.10GFL001 FUELE TESTED 4385 1231 483 34 162 9 0,89 0,99 0,73(P3001) SZABO 04.12GFL002 SMIKAL TESTED 5(P3024) SZABO 18.12GFL003 SMIKAL TESTED 455 93 54 3 15 1 0,92 1,00 0,92(P3060) (4) 30.11GFL004 SMIKAL TESTED 1334 417 189 8 55 11 0,96 1,00 0,91(P3061) (8) 19.11GFL005 FUELE TESTED 8(P3093) SZABO 27.11GFL012 FUELE TESTED 1744 589 305 10 95 6 0,94 1,00 0,73(P3067) (7) 03.02GFL013 PAP TESTED 1820 619 320 10 95 5 0,95 1,00 0,75(P3068) (6) 04.02GFL014 FUELE TESTED 2006 703 379 10 95 4 0,95 1,00 0,66(P3069) (5) 27.01GFL015 FUELE TESTED 1994 696 370 10 95 4 0,95 1,00 0,65(P3070) (4) 28.01GFL016 SMIKAL TESTED 1993 679 349 10 95 4 0,95 1,00 0,66(P3071) (5) 03.02GFL017 SMIKAL TESTED 1702 586 292 10 19 3 0,95 1,00 0,75(P3072) (4) 03.02GFL018 SMIKAL TESTED 930 264 124 5 35 1 0,94 1,00 0,91(P3063) (3) 21.01GFL019 PAP TESTED 1189 375 192 6 55 2 0,93 1,00 0,79(P3081) (2) 04.02GFL020 SMIKAL TESTED 1190 373 192 6 55 2 0,93 1,00 0,91(P3082) (3) 04.02

More Related