320 likes | 515 Views
TESTING THE TEST METRICS. Department of Jaka. Anna Tester. Brian Tester. Anthony Developer. What have they done today ?. Fixed three bugs . Added two new functions to prototype Written unit tests . Updated Release Notes. Raised a bug .
E N D
Department of Jaka Anna Tester Brian Tester Anthony Developer Whathavethey done today? • Fixedthreebugs. • Addedtwonewfunctionstoprototype • Writtenunittests. • UpdatedReleaseNotes. • Raised a bug. • Deployedthereleaseforthisfixandretestedthreetimes but case is stillfailing. • HelpedBrianfor his test set upmean time. • Deployedtherelease (bygettingsomehelp) • Didregressiontestingandexecuted 127 caseswhicharealreadyautomated. • Gaveapprovaltoshipment.
Whatmakes test brillant? • Finding maximum number of bugs? • Finding maximum number of critical bugs? • No bugs returned from customer? • Long hours of overtime? ? ? ?
Focus • If metric is a number attached to an idea than as the idea attached to these numbers vary from person to person things get more dangerous. • Testdepartments need more and more explaining their selves which causes bunches of test metrics. Today we are going to analyze and distinguish the productivity of these metrics and try to find the possible fallacies in order to select the most efficient of them.
Todaywearegoingto talk about • whatthetestablewhich is test metricswill be usedfor, whowilluse it and in whatways it will be used? • define themetricgroupthatwill be takenunder test • examinetheexpectedresultswiththeactualresults • comparetheresultswith a controlgroup
Customers A kilo of banana weighs a kilo. How muchtestingarewebuying(orselling) in thisproject? I assureyouthatwearedoing a greatjob here. Seetheresultsattached. Upper Management Senior Manager Itseemsthatweneedreorganizing. Whereto start from? I am themostexcelenttesteryouhave. Does it bring a pay raise? Line Manager Staff
Customer I do not wanttobother but areweshippingtoday? I am at thecustomernow. Do wehaveanyknownbugs I shouldnow? Sales Project Manager Pleasefill in theformsforperformanceassessmentsin ordertolet us driveyourperformancefeedbacks. How shall I knowthesenumbersaremeasuringtheintendedcorrectly? Human Resources Me, Metrictester
Choice of metrics Since in a limited time we cannot test all the metrics defined on earth, we will select a set of random methods used. For example we shall use following: • Tests Passed vs. Failed Ratio
Defect slippage ratio • Prerequisites:Defect tracking system, a pile of testers, a group of users or supports team whom tracks defects during usage at production, definite time period to finalize until when production defects are accumulated. • Execution: Count the number of bugs raised at test execution for a release cycle than until the defined time, count the bugs raised from production. Get the ratio of these two. • Expectation: If this ratio is small than we shall say that we have a proficient test group.
Defect slippage ratio Test Group 1 Test Group 2 Test Group 3
Defect slippage ratio Test results show that; • Sinceseverity is not taken into consideration this results fools its users. Although our first test group missed three critical bugs they look the best performing among all. Additively unfortunate second group looks really bad performing among all three,may be because of a make up defect. • In addition to those since the issues rejected because of no error…etc. are not taken into consideration, which means more false results. Among all the issues raised from production depends on the performance of users at production. If there are few or no users than this metric does not tell much about the performance of testers.
Defect Rejection Ratio • Prerequisites:Defect tracking system, one tester, a group of development team whom solves or rejects defects send from testers, a definite release cycle. • Execution: Count the number of bugs raised at test execution for a release cycle, than group them due to resolution types after that count the rejected bugs. Get the ratio of these two. • Expectation: If this ratio is small than we shall say that we have a proficient test group. If this ratio is big than that means testers are not capable of their product, which causes a big time loss for the development team.
Defect Rejection Ratio Test results show that; • This metric causes misbehaving to unfortunate testers who are dealing with difficult development groups. In this case the time consumed should also be investigated with development team if they have provided enough information to testing team at the beginning. • In our third case although configuration cases consumed all of development team’s time since development feels the responsibility of successful business results they achieved the process cooperatively which does not give any solid information about testers skill. Testers know how generally depends on documentation written by designers or developers and their personal experience with the product. If the product provides detailed logging, configuration items are self-descriptive and do not have duplicates than the tester becomes proficient at his/her work faster than his/her colleagues.
Customer satisfaction index – volume of calls Customer satisfaction index not only depends on the volume of call but also Number of system enhancement requests per year, Number of maintenance fix requests per year, training time per new user, Number of product recalls or fix releases (software vendors), Number of production re-runs (in-house information systems groups). For simplicity we have chosen volume of calls. Since calls to customer services costs a lot, many people do not use it except really urgent cases. So every unique call deserve our emphasis. • Prerequisites:Defect tracking system, a teamof testers, Customerservicessystemthatrecordscustomerscomplaints. • Execution: Count the number of bugs raised at test execution for a release cycle, count the number of callscontainingcomplaintsperproduct. Get the ratio of these two. • Expectation: If this ratio is bigthan we shall say that we have a proficient test group.
Customer satisfaction index – volume of calls Test results show that; • Since product A is as bad tested at product B and C, the risk of product and the attention of subscriber changes the successive rate direction. • Although product C is also critical since it causes company to financially damaged no calls arise from this defect. So it is fallible to depend on the customer reaction when judging test activities.
Test cost (in %) It is a myth that test cost should be a ratio for example one fourth of the development cost. Let us see if this myth is true or not. When we analyze test cost against overall project cost as follows • Prerequisites: Project managementsystemthatrecordsworklog time record of peopleworking on a project, testers, developpers • Execution: Count the number of bugs raised at test execution for a release cycle than until the defined time, count the bugs raised from production. Get the ratio of these two. • Expectation: If this ratio is small than we shall say that we have a proficient test group.
Test cost (in %) Test results show that; • there is no scientific relation with the job done by testers and developers. Testers deal with configuration, test set up, data preparation issues, as well as testing the whole product against any unpredicted errors occur whereas developer deal with the requirement requested may be effecting a few lines of code. • In fact if a developer is good at his work and writes his code with thinking race scenarios, testers workload will be less compared to a tester who does a lot of test set up and retesting. That means in fact there is a reciprocal relationship. If development time is short, it is highly possible that the code is buggy so there needs to be more time for test execution and bug fix period. As it is clearly seen from the test results, we can say that this myth has failed.
Tests Passed vs. Failed ratio “Test does not show absence of bugs, it shows existence”. But“How about giving shipment decisions? ” • Prerequisites: Test caserepository, toolrecordingcaseresults, test team. • Execution: Afteronecycle of test execution, countthenumber of test casespassedandfailed, getthepercentage of thesetwo. • Expectation: If this ratio is small than we areclosertoshipment.
Tests Passed vs. Failed ratio Test results show that; • Shipmentdecision can be madefor a productwhich has%1 pass rate at reports the previous day. • One more month shall be neededto ship the product who had %99 pass rate. • The other 50% mightnot be 50% any more. Thinkaboutreportingtheseresultsto CTO, how reliablethereportswilllook?
Thetruth: Allmetricsarefallable… Whatwillwe do now? “It’s easy to refuse to provide something that isn’t perfect. But that’s not helpful when the perfect isn’t available. When it comes to testers or consultants offering a “better alternative”, every executive has both the right and the responsibility to decide which alternative is the better one for her or his situation. ” HungQuocNguyen RunningLogiGear
Acceptallthatway? “ Itis important (I thinkveryimportant) to pay attentiontothevalidity of ourmetrics. It is importanttoimprovethem, tofindwaystomitigatetherisks of usingthem, andtoadviseourclientsaboutthecharacteristicsandrisks of the data/statisticswesupplytothem. I thinkit’simportanttousemetrics in waysthatdon’tabusepeople. ” Cem Kaner Professor of Software Engineering at Florida Institute of Technology
Control with a controlgroup Test progressreport Stock market report Finance SDLC
Comparisonmetricfrom Finance sector • One of theCreditAuditMetrics: Cash Rate Cash Rate = cashandcashequivalents / short-termdepts Expectation: IfCache Rate is morethantwothancompany is foundto be reliableforcreditassesments. Test results show that; Financial metricsarealsofallable.