1 / 21

Removing False Dependencies to Speedup Software Build Processes

Removing False Dependencies to Speedup Software Build Processes. Yijun Yu Homy Dayani-Fard John Mylopoulos. IBM Toronto Lab. University of Toronto. Background.

kieran-bradshaw
Download Presentation

Removing False Dependencies to Speedup Software Build Processes

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Removing False Dependencies to Speedup Software Build Processes Yijun YuHomy Dayani-FardJohn Mylopoulos IBM Toronto Lab University of Toronto Yu, Dayani-Fard, Mylopoulos CASCON 2003

  2. Background • Phenomenon/Problem: large C/C++ software takes much time to be built and rebuilt; in the development processes, debugging, testing, integration, releasing all have to wait for the binary image… • Observation from IBM Toronto lab: it takes > 7 hours to fully compile a system of > 10 million LOC. • Time is money Cost = Single build time x Number of People x Build freq. • … and more • Why? Excessive compile-time dependency among headers and implementations. Well-known. • Light weight solution?makedepend cleanup target dependencies (~0%), removing unused header inclusions and shorten the inclusion chain (estimated by KLOCwork, 37%) Yu, Dayani-Fard, Mylopoulos CASCON 2003

  3. Our approach • Finding false dependencies among finest grain program units • Removing them • Recent case studies (3 months after reported in the CASCON paper) • VIM 6.2, 269 KLOC in C (complete result) • A large commercial software package, 1580 KLOC in C++ (new result) • Future work and collaborations Yu, Dayani-Fard, Mylopoulos CASCON 2003

  4. 1. False dependencies • A PU Xdepends on a PU Y if X is using Y and Y occurs before X • We consider a program unit (PU) as a declaration or a definition at program or file scope. • Examples: function, variable, typedef, union, struct, class, template, enum, macro • Anti examples: data field, member function, auto variables, blocks, etc. • False dependency: dependency at inclusion level, no dependency at program unit level Yu, Dayani-Fard, Mylopoulos CASCON 2003

  5. An analogy • Chemical properties are kept in molecule structures, physical properties are kept in atom structures • Moving molecules doesn’t change the smell of the code, but replacing atoms in the molecule does! • Risk of change • 2H2 + O2 = 2H2O + heat • E = m c 2leads to a nuclear bomb! Yu, Dayani-Fard, Mylopoulos CASCON 2003

  6. #include “foo.h”int main() { foo(); } void foo(); void bar(); #include foo.h main.c Fileboundary true foo DeclareUnit false main DefineUnit bar remove Yu, Dayani-Fard, Mylopoulos CASCON 2003

  7. Negative impacts • Larger preprocessed image • Longer compilation time • False couplings among program units, components and packages Yu, Dayani-Fard, Mylopoulos CASCON 2003

  8. 2. Extracting and removing false dependencies asg2dep: Extracting PU dependencies from abstract syntax graph • based on parser like Datrix, KLOCwork • dependency graph + locations header: Partitioning the dependency graph (algorithm in the paper) and generating code • Removing unused declarations • Grouping declarations used by the same definitions into the same header files • Location is output as a #line directive Regression test and measurements Yu, Dayani-Fard, Mylopoulos CASCON 2003

  9. header.h a b c d e f g h i A D H E B MAIN F I C G 1.c 2.c 3.c main.c Example (original) Yu, Dayani-Fard, Mylopoulos CASCON 2003

  10. 1.c header.h 2.c 3.c main.c E A e H F a B h f MAIN D b d i I g c C G Example (restructured) Yu, Dayani-Fard, Mylopoulos CASCON 2003

  11. 3. Case study: VIM 6.2include relations Yu, Dayani-Fard, Mylopoulos CASCON 2003

  12. Program units dependencies Yu, Dayani-Fard, Mylopoulos CASCON 2003

  13. Buffer419 dcl. gui_gtk if_cscope charset154 dcl. Preprocessed image size 8.81 MB --> 4.78 MB -45.76% Yu, Dayani-Fard, Mylopoulos CASCON 2003

  14. Build time 96.37 seconds --> 42.26 seconds -56.15% Yu, Dayani-Fard, Mylopoulos CASCON 2003

  15. Synch-and-Stabilize • Continually synchronize parallel teams • Periodically stabilize the product in increments versus once at the end • Also known as: • Milestone, daily build, nightly build, zero-defect process Yu, Dayani-Fard, Mylopoulos CASCON 2003

  16. Case study 2. A C++ package in IBMpreprocessed image size 361.2 MB--> 179.1 MB -50.42% Yu, Dayani-Fard, Mylopoulos CASCON 2003

  17. Build time 14.02 minutes --> 4.44 minutes -68.31% Yu, Dayani-Fard, Mylopoulos CASCON 2003

  18. Cumulative build time of directly changed .C files 10.68 minutes --> 3.81 minutes Yu, Dayani-Fard, Mylopoulos CASCON 2003

  19. Cumulative build time of recompiled .C files 7.78 hours --> 2.46 hours Yu, Dayani-Fard, Mylopoulos CASCON 2003

  20. 4. Summary • False dependencies can lower productivity, thus are harmful to both managers and developers • Removing false dependencies in C++ requires supporting tools • Case studies have shown significant improvements (demo: VIM results can be downloaded from http://www.cs.toronto.edu/~yijun) Yu, Dayani-Fard, Mylopoulos CASCON 2003

  21. 5. Future work and possible collaborations • Integrate with IBM development on demand • Leverage KLOCwork’s analysis tools • Potential collaborations with CSER and CASCON members (give the grass, milk the cow) • UVictoria and UAlberta: visualize dependencies to assure programmers that there is no MAGIC behind the scene • UWaterloo: speedup the parser to meet the everyday need in the restructuring: Datrix + ALV? Bring 45 minutes per file down to 45 /184 minute • UWaterloo and UQueens: Quantitative goal modelling • UWindsor: web-services compositions and repackaging • UGhent: Link-time optimizations: Diablo: shrink VIM memory footprint to 1/3 Yu, Dayani-Fard, Mylopoulos CASCON 2003

More Related