Software Quality Assurance in Neuroinformatics

1. Software Quality Assurance in Neuroinformatics H Jeremy Bockholt 20080630 NITRC Grantee Meeting

2. Outline • Background and Examples • Software Process • Software Quality Assurance • Recommendations

3. Background • Neuroinformatics is an emerging discipline which attempts to integrate neuroscientific information from the level of the genome to the level of human behavior. • Neuroinformatics produces digital capabilities for a web-based information management system in the form of interoperable databases and associated data management tools. • Neuroinformatics tools include: • software for querying and data mining, data manipulation and analysis, scientific visualization, biological modeling and simulation, and electronic communication and collaboration

4. Examples of Neuroinformatics Tools • MRN has over 5500 human subjects, with 6500 scanning sessions using MR, EEG, MEG • We are now actively sequencing 1,000,000 SNPs arrays on subjects as well as collecting vast amounts of clinical, neuropsychological, behavioral, and physiological data • Using MRN tools, over 200 users at 7 sites collaborate on 150 active research studies with access to nearly 200 TB of data and growing… • BIRN has a vast set of multisite tools with even larger user base, set of data and tools and has adopted a federated model • XNAT multi-site tool for collaborative research data management • Analysis tools such as Slicer, FSL, AFNI, SPM, BRAINS, Freesurfer, etc.

5. Goal • Just like we are converging on standards and recommendations for data storage, data exchange, data access, we should explore the possibility of having recommendations for software process and software quality assurance for the field of neuroinformatics

6. Software Process • What tools and set of procedures and policies should be followed to create software. • Source Code Revision Control • Compilation • Testing • Documentation • Enhancement Tracking • Project Management • Communication and Collaboration

7. Software Process Challenges • The largest challenge in neuroinformatics is that usually there is a resistant to following a software process as the perception is that it take too much time • It is common for developers and users can be at separate sites • Many of our neuroinformatics developers are not software engineers • Time constraints and funding constraints sometimes limit neuroinformatics tool development just to functional prototypes that have limited or no testing or documentation associated with them

8. The NA-MIC Software Process • NA-MIC has a publication on software process and follows an excellent example of a software process optimized for Neuroinformatics • Adaptation of the distributed, open source software process • Targeted to the data-driven approach. • Utilize eXtreme Programming techniques • generating user stories, using simple design, writing tests first, refactoring, using continuous integration, and using coding standards are employed • some aspects such as having on-site customers, and pair-programming are not used • Suggests using CVS, DART, DOXYGEN, Bugzilla, Cmake, to support software development process • Encourages software developers to share code and tools and encourages participation in two programming weeks per year to permit face to face time for rapid development

9. Software Quality Assurance • The purpose of a software quality assurance plan is to define the formal processes and procedures for software development, a release review process, software development standards, and testing standards for a organization to follow across all software projects. • The plan should formally define communication strategies, Software Testing Plan, Software Measurements (calibration and user satisfaction), Risk Assessment, Corrective action process, and Training for developers • Software Quality should be measured and tracked at each development iteration

10. Software Quality Challenges • Most Neuroinformatics software development is for users at multiple sites, much of the software is deployed into multi-site research consortia • Most Neuroinformatics software development occurs under limited resources, shortened development cycles, large time constraints • A formal quality plan seems to be rarely followed, • Documentation, training, and thorough testing are typically cut or truncated to meet the constraints of budget and time • While the field is starting to share tools and source code (thanks to NITRC), we’re not sharing software process and strategies for maintaining quality

11. A Software Quality Plan for Neuroinformatics • We have developed a software quality plan that has been optimized to avoid some of the challenges found in the field of neuroinformatics. • Some of the highlights are using DART and Bugzilla to automatically perform and track software quality metrics • programmer productivity (bugs/module) • module defect density (code size/programming time) • requirements stability (number of initial requirements / total number of requirements) • system spoilage (effort fixing bugs / total project effort) • Conduct user satisfaction surveys at the onset and during use of each software product release • We have also generated recommendations on software development standards (IEEE) and specifically recommend following General Principles of Software Validation (US FDA, 2002) for neuroinformatics tools

12. Future Work • How can a software process be shared? • How can a software quality plan be shared? • Can the community decide on a preferred software process and software quality plan • We should publish a lessons learned manuscript on software quality in multi-site neuroinformatics

13. Conclusions • NITRC is a well-positioned to guide the field on software process and software quality • Having tool developers share their software process and quality plans would assist adopters of the tools better assess tools for integration with other components