1 / 50

White Box Testing

White Box Testing. Agenda The need Benefits Code review Unit tests and automation cppUnit Concepts Memory leaks Quantification Coverage APIs Challenges Best Practices Don’ts Metrics Cyclomatic Complexity. Need for White Box Testing.

Download Presentation

White Box Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. White Box Testing • Agenda • The need • Benefits • Code review • Unit tests and automation • cppUnit Concepts • Memory leaks • Quantification • Coverage • APIs • Challenges • Best Practices • Don’ts • Metrics • Cyclomatic Complexity

  2. Need for White Box Testing • Better to test anything from outside-in as well as inside-out • Most of the functional and performance issues arise due to bad coding • Ultimately the code matters – hence test the code left right and center • Black box tests are used to check cause and effect without getting into internals • White box tests do the same by getting into the internals of every program • Every developer is by default a white box tester • Medical field requires general physicians and surgeons • Certain problems can be identified and solved only by surgeons • A person may look healthy but internally the person may have high bp and sugar; they are not visible externally • So far, the importance of white box testing is not upto the mark • Product companies definitely need white box testing • If your application must scale to a very large extent, white box tests are inevitable

  3. Black Box Cannot Test These • It takes long time to check memory leaks • Any exceptions that happened but unnoticed or unhandled • Which portion of if condition is evaluated • What path the program took to achieve the end result • For every request what is the memory and cpu time taken • Is there any dead and unused code • Is there any extra code that is not needed • What are the potential breaking points in code • Is the code compliant to best practices • In large business apps, the data paths will be too large to test; it will require senior business analysts; if done thru white box mode, normal programmers can bring out great bugs

  4. Diagnostics tools • IBM Rational Purifyplus • JProbe • Parasoft tools • HP Mercury Diagnostics • Dynatrace • .Net profiler • AppPerfect • Netbeans • AQTime • All tools work on instrumentation and trace options • Some tools help in analyzing system memory space as well • Usually the system cpu executions are privileged, and hence we may not get access to those

  5. Business Benefits • Ensures that the core building blocks are intact, when adding new features and modules • Go to market with confidence on product stability • Provides greater visibility in terms of technical and internal compliance of the product • Vulnerability testing always needs a greater amount of white box testing • Resource utilization reduction is key for marketing. Products that use less memory and bandwidth are the most welcome ones • Keeps business leaders informed on the underlying strengths and weaknesses of the application platform

  6. Understand Design • Honestly speaking, how many of us use proper design before coding? • Can the same happen to real estate industry without a proper floor plan or for pharma industry without proper process and formula for drugs? • Design has 4 parts – UI, database, Interface and Logic design • UI design is key aspect when it comes to usability and compliance • Database design ensures scalability and storage • Interface design is key for plug and play during integration • Logic is important for the business flow of the application • Logic design can be a flow chart or text algorithm or pseudo code • Most of the modelers do have pictorial representation of design • Ideally the code must reflect the translation of pseudo code • Traceability of design elements to code is one of the key areas we need to look into • A change in design may impact several parts of code

  7. Code Review • Most of the times, this is ignored or not done to its true spirit • Code review must have design and code side by side • Code review can save upto 30% of testing cost at a later point in time • Usual problems happen due to cut and paste of code where it is not required • When there is a transition from one developer to another, code goes thru challenges • Quick fixes on big projects cause enormous amount of turbulence on production systems • Ideally senior people must review code • Spend at least 10% of coding time on code review

  8. Code Review Checklist • Just try these 10 critical points in daily life • No hard coding • Ensure loop termination conditions • No object creation inside loops • Close every object that you open • Give comments to every code block • Use database connection only when you need • Remove unused variables and code portions • Follow a consistent naming convention • Try to reduce overloading methods very often • Check SQL or file status soon after every disk operation

  9. Unit Testing • The smallest piece of code that can be independently run and tested is a unit • Unit can be a page or a function or a method within a class or a whole program itself • It is the developer’s responsibility to unit test the code • Unit test must focus on data type of every parameter • Focus on data format of every parameter • Focus on boundary values of every parameter • For any given code, what deliverable do we provide as part of unit testing? • Most of the times it is just the trust on the developer • Developer must spend at least 30% of coding time on unit testing alone

  10. How to draw boundaries • What is the line that demarcates unit and other tests? • One expected output out of a function • No chaining • When to stop unit tests? • When a set of parameter combinations are tested, stop the same • Boundaries and equivalence partitions must be tested before exit

  11. Unit Testing Checklist • Are the naming conventions followed? • Are the comments done properly? • Is there any hard coding? • Test case for custom exceptions. • Test case for system exceptions. • Test case for “body” of an if condition. • Test case for “body” of an else condition. • Test case for every loop termination. • Test case for every recursion termination. • Test case for pointers release (for memory leaks). • Test case for every procedure entry and exit. • Test case for every parameter validation for procedures. • Test case for resource release (Closing DB connections, releasing objects etc.)

  12. Unit Test Matrix • A single test must make a call to a function with some parameters combination and expect return value(s) • If a function has got multiple parameters that are of different types, then combinations of all parameters are possible and it will become a Cartesian product table • As the first step, a matrix needs to be prepared with all such combinations • Then the matrix must be optimized to reflect the practicality of the inputs • Possibilities of return paths must also be documented • The samples may go in 100s but not to worry on • Since automated unit tests run extremely fast without human inputs, 1000s of tests can be executed in minutes

  13. Automate Unit Tests • It is not practically possible to give all combinations of the inputs to programs and run • To reduce time, we need to write a program, that calls the program under test and pass parameters • Ideally the test program must be in the same language as the program being tested • This will help in maintaining the homogeneity of the data type of parameters passed • Since developers know the programming language, it is easy for them to write the test program as well • These are called test suites or test scripts • The investment on the unit test automation is one time as long as changes are minimal • The person who automates test for a program must know the program well

  14. The core of unit automation • Class A • Method m1(p1, p2) • Method m2(p3, p4) • TestClass TA • Testcase tm1 • ta = new Class A • rc = ta.m1(4, 5) • If (rc == exp_result) • Pass • Else • Fail • // destroy ta

  15. Level 1 Automation • Crude method – quick solution for short term projects • Write wrapper callers • Hard code parameters • Cut and paste test code with no hesitation • Spend little time on test creation • Test code can be throw-away soon • Simple logs, more manual result analysis required

  16. Level 2 Automation • Parameters to tests must be separated from test code • You can use csv or xml or any format for test data • Useful to run same test on larger test data sets • When a mixture of customers use the product, the data set also will change largely – this will be helpful in that case • Use your own test logs with more details

  17. Level 3 Automation • Use test tools such as cppUnit or jUnit etc. • Build auto recovery options if tests fail in between • Ensure every test has start-run-cleanup structure • Use this on long term test engagements • When tests are to be reusable, this is the best option • If you know cppUnit, the concepts are exactly the same in nUnit, jUnit etc.

  18. Stubs and Drivers • When the program under test needs some pre-built data or state, we need to mimic the same • This is a short-cut approach, but it serves the purpose • Stub is the one that mimics a called function • Driver is the one that mimics a caller function • When we use stubs and drivers, it is a must that we ensure that the data created by them are destroyed soon after the test

  19. Testing APIs • APIs do exchange of data from one piece to another • APIs are usually consumed by many independent programs • APIs in general are meant to provide a window of information to 3rd parties • One change in API will affect all consumers at once • Most of the current day apps provide APIs • We can send tweets from our application using APIs • We can send emails from Amazon SES thru APIs • API calls must be assumed to be context neutral • Some APIs do require authentication mechanism • APIs fall under external interfaces

  20. Wrapper Tools for APIs • A set of business critical APIs are now being used in all EAI scenarios • FIX, SWIFT, News feeds, HL7, HIPAA, Part 11 standards etc • A variety of commercial tools are available in each category • When messages are sent, these tools validate the structures of XML or EDI against the templates • Also, there are different versions of these APIs exist like html 5 etc.

  21. cppUnit • Write test wrappers in cpp for cpp programs • If you are familiar with cpp programming, you may not require cppUnit ! • There are many versions of cppUnit itself as it is open source – many people modified it for their convenience • You can write your own test library and framework • A set of pre-built classes and methods are already made available to minimize test coding time • Testcase, Testsuite, Assert are the important aspects in cppUnit • Testcase is the basic block – every single test • Testsuite contains multiple tests ( a batch) • Assert is the actual verification point where actual result is compared with expected result

  22. cppUnit • Fixture – known set of objects for test • setUp is the place where we create testcases • tearDown is the place where we remove all created tests • Commonly used asserts • Assert • Assert_message • Assert_equal • Assert_throw • Assert_equal_message

  23. Instrumentation • White box tools and profiler tools use one single concept called Instrumentation • Source code is compiled and built as binary • These tools understand the format of these binary files – where function entry points are made, how they exit, where stack is maintained etc. • Before profiling, they instrument the binary – inject their own log codes in the binary locations • During the execution, the instrumented code is run and every detail is logged using these injected code (such as entered function x, executed loop y, exited function y, destroyed object o etc.) • Using these logs, the tools provide internal details • This is very similar to sending a probe inside our body and doing a scan of that probe

  24. Overheads • When profilers run along with the instrumented programs, it adds load to the machine • This is a necessary overhead we need to take • The performance overheads are negligible when compared to actual object size • The statistics that are collected out of the programs will vary from time to time; but if the variance is beyond 5%, then the tool has real issues • Also, do not run 2 competing tools on the same machine. The effects are unpredictable

  25. Memory Leaks • When a memory pointer is declared and initialized and it is not released, it is a leak • Even in managed code sections, the leaks are there for a time till the garbage collector acts • Leaked memory locations are useless and vulnerable • One program leaking 10 bytes of memory per call – can bring down the system in a day when 1000s of users use the same for 100s of transaction • This is very important when we deal with device drivers and embedded systems

  26. Memory Profiling • This happens at runtime • Profiling is the basic block of performance engineering • Concept of instrumentation and tracing happens • Call graph will tell what is the path the program touched in terms of functions • Statistical profilers talk about memory and cpu • Memory consumed at that instance by variables and functions • Memory released at any given point of time

  27. Memory Quantification • What is the consumption of memory and cpu is quantified • Usual quantification happens in basic unit of measurement • UOM can be KB or machine cycles • How many cpu cycles are spent in this method when this is running • How many cpus cycles are spent to initialize the class

  28. Memory Usage

  29. Buffer Overflow • When arrays or memory blocks are declared and we try to access beyond those boundaries it is called overflow • When program variable space is packed, the overflows can cause damage to program counters or other stack areas • When stack is compromised, the result is program abort • Dynamic arrays are also vulnerable to overflow when people do not keep track of the max allocations • Buffer overflow can result on network buffers also when the response buffer is underestimated • Protection of buffer overflow is usually made by providing padding space to every memory space – but this needs to be done against memory optimization

  30. Object Issues • Objects require space and the methods need cpu cycles • When objects are declared and not destroyed, they also cause leaks • Unused objects are vulnerable areas for attack • Closure and nullifying objects are essential for optimized use of memory • Creating a series of objects in a multi user environment can cause spiral problems

  31. Function-wise Quantification • Every function has variables and statements • Parameters and variables take space from symbol table as well as stack • Statements do take cpu cycles based on how many times they get executed • When one function calls another, there is a huge amount of context switch happening at heap and program registry levels • How many times these context switches happen will also determine the complexity of the programs

  32. Test case for coverage • A path is defined as a route thru which a specific functionality is achieved by calling a series of steps • If programs flow from top to bottom, it is easy • The more conditions, the more complex it is • Coverage must ultimately make up to 100% of the tested function • Purecoverage will help in achieving this • Covering statements based on design is the best method • But when design is not in sync with program, we need to apply simple logic to hit every statement that is present in a program

  33. Coverage Output

  34. Coverage for conditions • a < b && c > d • In the above c>d is evaluated only when a < b comes to true • a < b || c > d • In the above c>d is evaluated only when a < b comes to false • Though part of condition is not evaluated by compiler as part of optimization, they may be later used in the program • Conditions need not be derived based on business case • Most of the time, the conditions are based on parameter values of the functions to determine the path to be taken

  35. Coverage for blocks • This depends on how many conditions we have in the code • Write one test for every relational operation of the condition • Definitely test db, IO and memory exceptions • Write one condition for every catch block • In case the exception cannot be easily triggered, have conditional compiled statements to raise such exceptions • Most of the time, people ignore exceptions thinking that it will be taken care – remember, it will be caught, but how to treat the exception is up to us

  36. Coverage goals • Coverage must have 100% at the end of all tests for every function • There must be 1 test for every relational operation in conditions • There must be 1 test for every loop termination • There must be 1 test for every exception • There must be 1 test for every test exit point (if method has multiple return points) • There must be one test for successful sql operation • There must be one test for unsuccessful sql operation

  37. Metrics Collection • Average number of unit test cases per page/component/any specific unit in the project • Total number of unit test cases • Total number of unit test cases failed in test pass-1 • Total number of unit test cases failed in test pass-2 • Total number of unit test cases failed in test pass-n • Total number of memory leaks found • Percentage code coverage as per the tool statistics • Execution time for each test run • Total test execution time for all units • Ratio of # of defects found in unit testing to # of defects found in code review • Defect distribution across units (asp related, db related, parameter related etc.)

  38. Metrics Analysis • Trend analysis helps in seeing whether the error rate moves up or down • See pattern of failures and find out the corrective action • Metrics vary a lot from developer to developer; hence find out error rate pattern across developers • Analyze failure pattern and time of failure • It is not easy to catch all errors in one round of testing; hence do more rounds; more often

  39. Configuration Management • Since we develop test code, we need to check in these code sections to VSS or subversion or PVCS • Maintain a uniform x.y version code to all test cases and suites • Check in the test data along with test code as well • Have one single person who can oversee all the CM operations • Never allow people to have local copies of test code

  40. Best Practices • Keep adding more tests every day • More data preparation is the key • Test often - even twice a day • Never ignore warnings from IDEs • Run unit tests on a clean test bed • Review unit test code as well • Run regression unit tests daily • Run unit tests from other developer’s machine

  41. Best Practices • Run unit tests by a different developer • Make unit test code simple • Never try to have if then else in a test code • Do not have loops in a test code • Run critical tests first • Run all unit tests in a single session; never stagger

  42. Don’ts • Do not test entire functionality in one test • One test must satisfy just one condition • Never underestimate any parameter combination • When skype or sony programs can come down just by 1 bug, our program too can come down • If certain validations are performed in other layers, do not do in unit tests • Never run tests on a test machine without clean up

  43. Don’ts • Never miss the null value test for any parameter • Never miss any negative value tests for database connection or query tests • Never miss trying to access an object or record that does not exist, because these are the places where exceptions get triggered • Do not miss to pass a wrong xml structure to web services

  44. Challenges • White box testers must be developers • Hence companies use them to do core development work and ignore writing white box tests • White box tests need tools and sometimes they are a little expensive; hence companies avoid this route • Main challenge comes in not documenting design. Since design is the basis, and if that is not present or incomplete, white box tests do not yield good results • Project planning never considers the time for white box testing; especially in service companies, this is out of scope! • When changes are made to the programs being tested, the changes are not documented well and hence unit tests go out of gear

  45. Challenges • When same developer runs the unit tests, there is a chance to ignore issues – so have developer A testing the code of developer B • Initial unit test automation code takes some time; hence do not expect the ROI on day 1 • Results can be seen only after one or 2 quarters. Many management teams do not have patience to wait for such a long time • People try to implement logic in different layers that they are designated for; instead of coding some logic in Java, they code in stored procedure and call that from java – this makes profiling and coverage difficult

  46. Cyclomatic Complexity • This is to arrive at the testing complexity of code • Nodes, edges are the primary factors • Higher the complexity, higher the number of tests we need • This is based on graph theory • Many physical problems can be solved by graph theory • e.g. traveling salesman theory • M = E - N + 2P • E - number of edges (lines) • N - number of nodes (rounds) • P is the connected components (start and destination points) • The more if conditions and branches, the nodes and edges increase • Call graph of all methods will show the independent connected components

  47. Business Flow Tests • When we do white box tests, we test individual components • Hence it comes to unit tests • Divide flow into multiple components • Test them one by one • Aim at one transaction at a time • Club white box tools with functional test tool • Since business transactions need database setup, ensure you have test stubs that pre-populate records in tables

  48. How to speed-up • Let a senior create a sample test for every function • Let a senior prepare unit test matrix • For a few tests, senior and junior developers must do pair test automation • Then junior takes care of rest of the unit tests for automation • Senior reviews the unit test code • Lead runs all tests every day

  49. Get ready to take blame • Unit tests are code; hence they may also have bugs. Developers may start blaming the tests themselves • Tests may be redundant if not properly reviewed • Sequence of unit tests must be carefully done. Always use bottom up approach on unit tests – test the smallest one first • Run unit tests soon after build is released, and before BVT starts by the black box team

  50. Never Give Up • Teams give up white box tests after a quarter or so • This is because, they do not see visible returns • Remember – these things fail is a feel-good factor on testing; at the same time, these things work is more important as well • Most of the times, great systems are brought down due to missing white box tests and not black box functionality • It may take years to find out, one great bug – but it is essential for big time products • If you are in product company, be patient and white box will pay back

More Related