Evaluation of Current Requirements Analysis Tools Capabilities for IV&V in the Requirements Analysis Phase

1. Evaluation of Current Requirements Analysis Tools Capabilities for IV&V in the Requirements Analysis Phase SAS 2007 Executive Galaxy Global Corporation Presented by: Valerie Jones & Jennifer Murray NASA POC: Jeffrey Northey SAS_07_EvalRaTools_RAPhase_Jones2

2. Problem Statement • A Requirement Analysis tool that performs comprehensive automated analysis does not exist • Existing tools available for use do not provide guidance or automated assistance in identifying potentially problematic requirements • Requirements analysis is a long and tedious process. • An automated tool could decrease the time required during the requirements analysis phase as well as improve the process.

3. Tool Evaluation Study • Tools Evaluated: • NASA E-Smart/ARM • Lexior • QuARS • Requirements Assistant • SAT • TEKChecker • Tools evaluated on a predefined set of technical and quality attributes. • Model problem defined to do comparison between manual analysis and analysis with automated assistance

4. Tool Study Results • Two evaluation studies were performed • Results indicated automated analysis tools correctly identified technical issues consistent with manual analysis. • Requirements Assistant performed best technically in both studies • Analysis results were consistent with manual analysis and additional key issues identified • Key issues identified including: missing requirements and inconsistencies in requirements

5. RA Technical Results • Comparing to original 21 TIMs submitted for project. • 12/21 were found correctly. • 6/21 were flagged but for other reasons. • 5/21 were not flagged at all. • In addition to matching TIM’s generated, tool also found missed defects that were not reported originally. • RA was the only tool to identify missing and inconsistent requirements

6. Technical Results (Project 2) • Requirements Assistant (RA) • Correctly Identified 2 RIDs. • Could possible collect additional 2 RIDs if entire document was analyzed. • One was found via traceability. (Outside of tools capabilities) • Additional 13 issues identified and presented to project manager

7. RA Differences • RA was the only tool evaluated that takes domain information specific to the system into consideration • Allows for creation of a domain corpus to define known information about the system • Input files allow tool to identify missing or inconsistent requirements • RA analyzes starting at the sentence level, paragraph level then entire document as a whole. • Utilizes knowledgebase of rules developed of 25 years of experience on various systems

8. File Definitions • Milfile - This file contains the aspects that the reviewer wants to address. • Milsub - an entry, that should be considered when the word “system” is found in a requirement. • Actor - an actor in the system (not necessarily humans). • Actuator - a list of hardware that the actor(s) work on. • Quant – Units of measure • Function - An array of functions that the system has, or should have (in the reviewer’s opinion). • Gloss - This contains the glossary of terms in the requirements document. (if provided). The file only contains the glossary terms, not their definition. • List_Sensor – A list of sensors known in the system.

9. Tool Pre-Processing • Input document must be in standard text format • Input files are currently entered manually • Input files are not required but increase the tools ability to correctly identify errors

10. Tool Output • Examples – Poorly written requirements1. Appropriate system standards shall be used where necessary.Remarks of The Requirements Assistant™:a. appropriate – What is appropriate?b. standards – Specifically which standards are required?c. where necessary – Under what conditions will these standards be applied?2. It is required that the red light be illuminated once temperatures exceed about 25 degrees. Remarks of The Requirements Assistant™:a. It is required – Is this a requirement?b. Illuminated – Is this for a certain duration?c. Temperatures – temperatures of..?d. About – accuracy?e. Degrees – Celsius, Fahrenheit?f. Is there also a requirement how/when the red light will be switched off?

11. Examples cont. • Off-line tooling allowing the end-user to manipulate … in order to… compensate for … stability.Remarks of The Requirements Assistant™:a. manipulate – exactly what is manipulated?b. In order to – this is not a requirement but a goal.c. Compensate for – how much? • Remarks of The Requirements Assistant™:No requirements on reliability, safety and EMC were found in the set of requirements.

12. Various Output Previews • The Requirements Assistant™ provides three different previews of its analysis results: View by quality attributes: a. Consistency b. Completeness c. Testability d. Accuracy e. Readability

13. Various Output Options cont. View by sentence: • Poor words (in bold): The FSW will send appropriate error messages, and set the fault condition to “failed”, should the alive and Idle discrete lines fail to signal ready ” and “Idle” within a parameterized amount of time [General Remark: 4 Upper limit? General Remark: and, and, and, complex sentence ] • The FSW will send commands, and receive health & status and science telemetry information from most XXX instrumentation through this interface.General Remark: Quantity-like in paragraph most in sentenceNon Requirement ["will"]: null • View by topic • DELIVERY AND RETURN "The XXX shall deliver at least 2850 kg (6,283 lbm) (gross) of cargo. • DELIVERY AND RETURN "The XXX shall return at least 2,858 kg (6,283 lbm) of cargo.

14. Various Output Options cont. • View by concept: Cargo Processing Concept: • DELIVERY AND RETURN "The XXX shall deliver at least 2850 kg (6,283 lbm) (gross) of cargo. • DELIVERY AND RETURN "The XXX shall return at least 2,858 kg (6,283 lbm) of cargo.

15. Benefits of RA • Leads analyst to identify key issues outside capabilities of other tools evaluated • Could allow the analyst to prioritize analysis based on identification of potentially problematic requirements first. • Better results for both novice and experienced analysts • Novice analyst identified key issues with a total analysis time of 3 days compared to a team of analysts working for 3 weeks a (50% decrease in analysis time). • Developers would have positive ROI if using this tool for tracking requirements across stages of development • Can assist in Requirements Validation with UML modeling. • Knowledge base behind the tool allows for very versatile use for various aspects of software development to stay consistent

16. Recommendations • Proposed facility adopt Requirements Assistant (RA) for further use. • RA requires additional upgrades before it can be used facility wide. • Upgrades • Improve user interface for analyst efficiency • Increase speed • UML support • Domain term library

17. Proof of Concept • Current tasking is to evaluate RA capabilities for requirements validation and verification to UML modeling of systems. • Working with NASA IV&V Facility project performing requirements validation and verification of 4 System Reference Models utilizing RA capabilities. • Initial results from first model analyzed were provided to the project • Project implemented additional updates based on findings • Through the POC we will work to develop an appropriate user interface for the tool and the IV&V analyst.

18. Proof of Concept Model

19. What’s Next? • Continue the Proof of Concept Study for Requirements Validation • Recommended updates to Requirements Assistant for user interface development. • Evaluating Eclipse platform • Develop additional requirements. • Proposal submitted for Research Infusion.