Testing Programs for Transportation Management Systems – Practical Considerations

1. Testing Programs for Transportation Management Systems – Practical Considerations Testing is an important part of the deployment of a TMS. The purpose of testing is two-fold. First, testing is about verifying that what was specified is what was delivered - it verifies that the product (system) meets the functional, performance, design, and implementation requirements identified in the procurement specifications. Therefore, a good testing program requires that there are well-written requirements for both the components and the overall system. Without testable requirements, there is no basis for a test program. Secondly, testing is about managing risk for both the acquiring agency and the system’s vendor/developer/integrator. The test program that evolves from the overarching systems engineering process, if properly structured and administered, allows managing the programmatic and technical risks to help assure the success of the project. The testing program is used to identify when the work has been “completed” so that the contract can be closed, the vendor paid, and the agency can transition the system to the warranty and maintenance phase of the project. Meaning of Requirements Statements Terms The language, syntax, and structure of you requirements statements are extremely important as they directly affect the quality and thoroughness of the test program that is based on them. Certain terms used in requirements statements have specific contractual implications. • “Shall” is used to confer a requirement on the provider of the product or service and is typically understood to mean at the time of delivery. • “Will” is used to confer a requirement on the receiver (the accepting agency) of the product or service when that product or service is delivered. “Will” is also used to imply a future requirement on the provider that should be clear from the context of the statement. • “May” is a conditional term and implies that either the provider or the receiver has the option to meet the stated requirement. “May” statements are generally not testable unless additional conditions are included to indicate what is expected if the provider or receiver elects that option. • “Should” falls into same category as “may” and is considered an optional requirement that may or may not be included in the system depending on the provider’s perspective. • “Must” is used to add additional emphasis to the requirement statement that can be directed at either the provider or receiver, but more typically the provider. “Must” is typically used in a requirement that has specific legal or contractual ramifications such as may be invoked by requiring a particular State statute or governing regulation be strictly adhered to in the performance of the work. In the contract specifications, it has the same basic meaning as “shall.” From a contracting perspective, only requirements with MUST and SHALL statements are likely to be provided by the contractor. All other requirements should be considered part of a “wish list” and will not be part of the testing program.

2. Writing Testable Requirements – Do’s and Don’ts The requirements statements contained within the procurement specifications are the basis for test and acceptance. Poor or missing requirements statements result in requirements that cannot be verified and products or services that don’t meet expectations. Requirements statements should be written as clear, unambiguous, declarative sentences. Proper grammatical sentence structure is as important as is use of “shall” and “must,” particularly in defining who is the responsible party for providing the product or service and who will be accepting delivery of that product or service. The following are some do’s and don’ts for writing and reviewing testable requirements. Do’s – • Write the requirement in simple, understandable, concise terms; be short and to the point. If complex technical terminology is necessary, make sure those terms are defined or well understood by the provider as well as the receiver. • For each [individual] requirement there should be one shall or must statement. If the requirements are complex, then they should be subdivided into a string of individual requirements to the greatest extent possible. • Write the requirement as a positive statement. If something is not desired, try to phrase the requirement to state what is desired. • Have a test method, such as inspection, certificate of compliance, analysis, demonstration or test, and pass/fail criteria in mind when writing or reviewing the requirement. If you can’t figure out how to verify the requirement or what criteria constitutes acceptance, you can’t expect the provider to demonstrate compliance with the requirement. • Consider the complexity, technical expertise, and expense of the testing that may be necessary to verify the requirement — simplifying the requirement may result in the same end product or service, but at a reduced test expense. • When preparing the requirements statement, make sure that the requirements will have the same interpretation regardless of the background and experience of the reader. Add clarifying information when necessary to ensure a common understanding by readers with radically different backgrounds. Don’ts – • Avoid the use of “may” and “should” in the requirement statement unless you specifically want to give the provider an option in how that requirement can be met, or give the receiver an acceptance option or an “out.” • Avoid negative requirements. Positive statements are better, because they can be definitively measured for acceptance testing. • Don’t mix dissimilar or unrelated requirements in the same statement. This practice complicates requirements traceability and verification testing. Unrelated requirements will usually be verified at different times, under different test conditions, and using different test procedures or methods. Additional Resources The Testing Handbook for Transportation Management Systems is intended to provide direction, guidance, and recommended practices for test planning, test procedures, and test execution for the acquisition, operation, and maintenance of transportation management systems and ITS devices. For a non-technical audience, the TMS Testing Primer identifies key aspects of testing, identifies the benefits of developing and using testing programs, and profiles successful practices of existing programs. The material is available on the Federal Highway Administration website at http://ops.fhwa.dot.gov/publications/publications.htm/. • Primer, FHWA-OP-07-089, EDL# 14354 • Handbook, FHWA-OP-07-088, EDL# 14353 Federal Highway Administration U.S. Department of Transportation 400 7th Street, S.W. (HOP) Washington, DC 20590 Toll-Free “Help Line” 866-367-7487 www.ops.fhwa.dot.gov Publication No.: FHWA-OP-0X-XXX EDL Document No.: XXXXX February 2007