1 / 26

SCons: Big Signature Refactoring

SCons: Big Signature Refactoring. 18 April 2007. Build tools do TWO things. Build things in the correct order NOT build what’s up to date. Building things in the correct order. Simplest way to get correct order is to just write down the commands in a script

keefe
Download Presentation

SCons: Big Signature Refactoring

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SCons:Big Signature Refactoring 18 April 2007

  2. Build tools do TWO things • Build things in the correct order • NOT build what’s up to date

  3. Building things in the correct order • Simplest way to get correct order is to just write down the commands in a script • If our dependencies never changed, we’d just script it and be done with it • Dependency management is necessary because our software changes

  4. Not rebuilding what’s up-to-date • Simplest way to get up-to-date software is to build from scratch every time • If our tools were infinitely fast, we’d do build from scratch • Deciding what’s up-to-date is necessary because our tools take time

  5. How does Make decide what’s up to date? • Target file • Source file(s) • Is timestamp(target) > timestamp(source) for all source files, it’s up to date

  6. What’s wrong with the way Make does it? • Contents can change without modifying timestamp (incorrectly not rebuilding) • Timestamps can change without modifying the contents (unnecessary rebuilds) • This can be positive: touch the source file and rebuild ensures a target is up-to-date • Can’t handle timestamps rolling back in time

  7. What’s right with the way Make does it? • It uses file metadata (the timestamp) to approximate if the file contents have changed • It’s a cheap test

  8. What file metadata can we use? • Timestamp (only metadata Make uses) • Size • Content checksum

  9. Way SCons currently does it • Metadata stored in a .sconsign file • Used to be one per directory • Still can be configured that way • sconsign script to dump metadata info • Contrast ClearCase, which stores metadata from a custom file system • Use the metadata for more sophisticated decisions

  10. Example Program(‘foo.c’) $ scons -Q gcc –o foo.o –c foo.c gcc –o foo foo.o $ sconsign .sconsign === .: foo: 8f72e133e001cb380a13bcb6a16fb16f None 1176861920 6762 foo.o: e61afae6ccfe99a63b0b4c15f18422f6 foo.o: e61afae6ccfe99a63b0b4c15f18422f6 None 1176861920 1488 foo.c: b489a8c34c318fc60c8dac54fd58b791 foo.h: c864c870c5c6f984fca5b0ebd7361a7d

  11. More readable output $ sconsign –-verbose .sconsign === .: foo: bsig: 8f72e133e001cb380a13bcb6a16fb16f csig: None timestamp: 1176861920 size: 6762 implicit: foo.o: e61afae6ccfe99a63b0b4c15f18422f6 foo.o: bsig: e61afae6ccfe99a63b0b4c15f18422f6 csig: None timestamp: 1176861920 size: 1488 implicit: foo.c: b489a8c34c318fc60c8dac54fd58b791 foo.h: c864c870c5c6f984fca5b0ebd7361a7d

  12. Readable timestamps, too $ sconsign –-verbose –-readable .sconsign === .: foo: bsig: 8f72e133e001cb380a13bcb6a16fb16f csig: None timestamp: 'Tue Apr 17 19:15:50 2007' size: 6762 implicit: foo.o: e61afae6ccfe99a63b0b4c15f18422f6 foo.o: bsig: e61afae6ccfe99a63b0b4c15f18422f6 csig: None timestamp: 'Tue Apr 17 19:15:50 2007' size: 1488 implicit: foo.c: b489a8c34c318fc60c8dac54fd58b791 foo.h: c864c870c5c6f984fca5b0ebd7361a7d

  13. SourceSignatures(‘MD5’)Program(‘foo.c’) $ sconsign –-verbose .sconsign === .: foo: bsig: 8f72e133e001cb380a13bcb6a16fb16f csig: None timestamp: 1176861920 size: 6762 implicit: foo.o: e61afae6ccfe99a63b0b4c15… foo.o: bsig: e61afae6ccfe99a63b0b4c15f18422f6 csig: None timestamp: 1176861920 size: 1488 implicit: foo.c: b489a8c34c318fc60c8dac54… foo.h: c864c870c5c6f984fca5b0eb… foo foo.o foo.c foo.h

  14. “Build signatures” bsig(foo.o) = md5( sig(foo.c) + sig(foo.h) + sig(cmd_line) ) bsig(foo) = md5( sig(foo.o) + sig(cmd_line) ) $ sconsign –-verbose .sconsign === .: foo: bsig: 8f72e133e001cb380a13bcb6a16fb16f csig: None timestamp: 1176861920 size: 6762 implicit: foo.o: e61afae6ccfe99a63b0b4c15… foo.o: bsig: e61afae6ccfe99a63b0b4c15f18422f6 csig: None timestamp: 1176861920 size: 1488 implicit: foo.c: b489a8c34c318fc60c8dac54… foo.h: c864c870c5c6f984fca5b0eb…

  15. SourceSignatures(‘MD5’)TargetSignatures(‘content’)Program(‘foo.c’)SourceSignatures(‘MD5’)TargetSignatures(‘content’)Program(‘foo.c’) $ sconsign –-verbose .sconsign === .: foo: bsig: 27d34d21414965ce2f9cc5d8f3e3fbbb csig: None timestamp: 1176861920 size: 6762 implicit: foo.o: c7710864231f22aad3e8c1bc… foo.o: bsig: e61afae6ccfe99a63b0b4c15f18422f6 csig: None timestamp: 1176861920 size: 1488 implicit: foo.c: b489a8c34c318fc60c8dac54… foo.h: c864c870c5c6f984fca5b0eb… foo foo.o foo.c foo.h

  16. SourceSignatures(‘timestamp’)Program(‘foo.c’) $ sconsign –-verbose .sconsign === .: foo: bsig: 1176915205 csig: None timestamp: 1176915335 size: 6762 implicit: foo.o: 1176915205 foo.o: bsig: 1176915205 csig: None timestamp: 1176915335 size: 1488 implicit: foo.c: 1176915205 foo.h: 1176915196 foo foo.o foo.c foo.h

  17. SourceSignatures(‘timestamp’)TargetSignatures(‘content’)Program(‘foo.c’)SourceSignatures(‘timestamp’)TargetSignatures(‘content’)Program(‘foo.c’) $ sconsign –-verbose .sconsign === .: foo: bsig: 1176915335 csig: None timestamp: 1176915335 size: 6762 implicit: foo.o: 1176915335 foo.o: bsig: 1176915205 csig: None timestamp: 1176915335 size: 1488 implicit: foo.c: 1176915205 foo.h: 1176915196 foo foo.o foo.c foo.h

  18. Problems with how SCons does it now • Can’t switch between content and timestamps • Can’t mix content + timestamps in same config • Example: want to use content for all input files except one really big file where you want to use timestamps • Stores information about last build decision, not just metadata about the current state of file • Must have complete dependency graph to make the same signature as last time • Can’t build only part of the DAG • Can’t use dependency output from tools • Example: gcc –Md output

  19. New .sconsign format $ sconsign .sconsign === .: SConstruct: 059bf2bda6723d166c5dab7d54a0ca13 1176861882 17 foo: 5701724287c3d3847516781876f56d87 1176862330 6762 foo.o: cc74a5b5cd4b174a59b58495cd2ef1f9 1176862330 1488 c4245ece9e7108d276b3c8eb7662d921 [$LINK -o $TARGET ...] foo.c: b489a8c34c318fc60c8dac54fd58b791 1176861903 55 foo.h: c864c870c5c6f984fca5b0ebd7361a7d 1176861911 19 foo.o: cc74a5b5cd4b174a59b58495cd2ef1f9 1176862330 1488 foo.c: b489a8c34c318fc60c8dac54fd58b791 117686190355 foo.h: c864c870c5c6f984fca5b0ebd7361a7d 117686191119 d055c09cba5c626f5e38f2f17c29c6fa [$CC -o $TARGET -c...]

  20. === .: SConstruct: csig: 059bf2bda6723d166c5dab7d54a0ca13 timestamp: 1176861882 size: 17 foo: csig: 5701724287c3d3847516781876f56d87 timestamp: 1176862330 size: 6762 implicit: foo.o: csig: cc74a5b5cd4b174a59b58495cd2ef1f9 timestamp: 1176862330 size: 1488 action: c4245ece9e7108d276b3c8eb7662d921 [$LINK –o $TARGET $LINKFLAGS $SOURCES ...] foo.c: csig: b489a8c34c318fc60c8dac54fd58b791 timestamp: 1176861903 size: 55 foo.h: csig: c864c870c5c6f984fca5b0ebd7361a7d timestamp: 1176861911 size: 19 foo.o: csig: cc74a5b5cd4b174a59b58495cd2ef1f9 timestamp: 1176862330 size: 1488 implicit: foo.c: csig: b489a8c34c318fc60c8dac54fd58b791 timestamp: 1176861903 size: 55 foo.h: csig: c864c870c5c6f984fca5b0ebd7361a7d timestamp: 1176861911 size: 19 action: d055c09cba5c626f5e38f2f17c29c6fa [$CC –o $TARGET –c $CFLAGS $CCFLAGS ...]

  21. New .sconsign format $ sconsign .sconsign === .: SConstruct: 059bf2bda6723d166c5dab7d54a0ca13 1176861882 17 foo: 5701724287c3d3847516781876f56d87 1176862330 6762 foo.o: cc74a5b5cd4b174a59b58495cd2ef1f9 1176862330 1488 c4245ece9e7108d276b3c8eb7662d921 [$LINK -o $TARGET ...] foo.c: b489a8c34c318fc60c8dac54fd58b791 1176861903 55 foo.h: c864c870c5c6f984fca5b0ebd7361a7d 1176861911 19 foo.o: cc74a5b5cd4b174a59b58495cd2ef1f9 1176862330 1488 foo.c: b489a8c34c318fc60c8dac54fd58b791 1176861903 55 foo.h: c864c870c5c6f984fca5b0ebd7361a7d 1176861911 19 d055c09cba5c626f5e38f2f17c29c6fa [$CC -o $TARGET -c...]

  22. New .sconsign format • Every file entry is consistent: foo.c: b489a8c34c318fc60c8dac54fd58b791 1176861903 55 • Content signature (if read, None if not) • Timestamp • Length • Just stores file state at last time used • Source files are explicitly stored • Can be used for caching checksums • Actions and their signatures are explicitly stored

  23. No build signatures! • Up-to-date decision is done by comparing current metadata of each input file with information last time target was built • Each decision can be independent: • Example: “Rebuild this target file if:” • Any input text file has different content than last time • Any input graphic file has a different timestamp than last time

  24. Supporting Slides

  25. Build signature boils down states of source+dependency files at time target was built • Not complete state, just our signature calc

  26. foo foo.o foo.c foo.h

More Related