Smoothing Software Project Scripting

1. Smoothing Software Project Scripting • Mark Ramm • reprising his award-winning role as • Kevin Dangoor • BlueSkyOnMars.com

2. Python is not compiled (actually, it is, but that’s not important right now)

3. Python projects certainly do not need • Redistributable files to move around • Deployment to other machines • Interaction with source control systems • Documentation that’s built from the source code • Documentation that’s built from a format other than its final display format

5. If only there was some scripting language that we already knew.

6. Waitasec. Isn’t Python a scripting language?

7. Project-related scripts need... • Command line argument handling • Configuration • Often work with files • Sometimes need to use distutils/setuptools, but wish they could do a little more • Need to work with other common tools (Sphinx, svn, virtualenv)

8. Command-line argument handling • bin/start-server --port 8675309

9. Configuration • [messages] • greeting=Hello • [thing1] • who=World • message=${messages.greeting}, ${who}

10. Lots of working with files • import os • if not os.path.exists(“foo”): • os.mkdir(“foo”) • if not os.path.exists(os.path.join(“foo, “bar”)): • open(os.path.join(“foo”, “bar”), “w”).write(“Hi”)

11. Working with distutils/setuptools • #!/bin/sh • sphinx-build (blah blah blah) • python setup.py sdist upload

12. Other common tools • sphinx-build ... • subprocess.Popen(“svn info”... • # and do something with the output • virtualenv.create_bootstrap_script(“# more code”)

13. Do you really want separate scripts for everything? • /usr/local/bin/git /usr/local/bin/git-merge-resolve • /usr/local/bin/git-add /usr/local/bin/git-merge-stupid • /usr/local/bin/git-add--interactive /usr/local/bin/git-merge-subtree • /usr/local/bin/git-am /usr/local/bin/git-merge-tree • /usr/local/bin/git-annotate /usr/local/bin/git-mergetool • /usr/local/bin/git-apply /usr/local/bin/git-mktag • /usr/local/bin/git-archimport /usr/local/bin/git-mktree • /usr/local/bin/git-archive /usr/local/bin/git-mv • /usr/local/bin/git-bisect /usr/local/bin/git-name-rev • /usr/local/bin/git-blame /usr/local/bin/git-pack-objects • /usr/local/bin/git-branch /usr/local/bin/git-pack-redundant • /usr/local/bin/git-bundle /usr/local/bin/git-pack-refs • /usr/local/bin/git-cat-file /usr/local/bin/git-parse-remote • /usr/local/bin/git-check-attr /usr/local/bin/git-patch-id • /usr/local/bin/git-check-ref-format /usr/local/bin/git-peek-remote • /usr/local/bin/git-checkout /usr/local/bin/git-prune • /usr/local/bin/git-checkout-index /usr/local/bin/git-prune-packed • /usr/local/bin/git-cherry /usr/local/bin/git-pull • /usr/local/bin/git-cherry-pick /usr/local/bin/git-push • /usr/local/bin/git-citool /usr/local/bin/git-quiltimport • /usr/local/bin/git-clean /usr/local/bin/git-read-tree • /usr/local/bin/git-clone /usr/local/bin/git-rebase • /usr/local/bin/git-commit /usr/local/bin/git-rebase--interactive • /usr/local/bin/git-commit-tree /usr/local/bin/git-receive-pack • /usr/local/bin/git-config /usr/local/bin/git-reflog • /usr/local/bin/git-convert-objects /usr/local/bin/git-relink • /usr/local/bin/git-count-objects /usr/local/bin/git-remote • /usr/local/bin/git-cvsexportcommit /usr/local/bin/git-repack • /usr/local/bin/git-cvsimport /usr/local/bin/git-repo-config • /usr/local/bin/git-cvsserver /usr/local/bin/git-request-pull • /usr/local/bin/git-daemon /usr/local/bin/git-rerere • /usr/local/bin/git-describe /usr/local/bin/git-reset • /usr/local/bin/git-diff /usr/local/bin/git-rev-list • /usr/local/bin/git-diff-files /usr/local/bin/git-rev-parse • /usr/local/bin/git-diff-index /usr/local/bin/git-revert • /usr/local/bin/git-diff-tree /usr/local/bin/git-rm • /usr/local/bin/git-fast-import /usr/local/bin/git-runstatus • /usr/local/bin/git-fetch /usr/local/bin/git-send-email • /usr/local/bin/git-fetch--tool /usr/local/bin/git-send-pack • /usr/local/bin/git-fetch-pack /usr/local/bin/git-sh-setup • /usr/local/bin/git-filter-branch /usr/local/bin/git-shell • /usr/local/bin/git-fmt-merge-msg /usr/local/bin/git-shortlog • /usr/local/bin/git-for-each-ref /usr/local/bin/git-show • /usr/local/bin/git-format-patch /usr/local/bin/git-show-branch • /usr/local/bin/git-fsck /usr/local/bin/git-show-index • /usr/local/bin/git-fsck-objects /usr/local/bin/git-show-ref • /usr/local/bin/git-gc /usr/local/bin/git-ssh-fetch • /usr/local/bin/git-get-tar-commit-id /usr/local/bin/git-ssh-pull • /usr/local/bin/git-grep /usr/local/bin/git-ssh-push • /usr/local/bin/git-gui /usr/local/bin/git-ssh-upload • /usr/local/bin/git-hash-object /usr/local/bin/git-stash • /usr/local/bin/git-http-fetch /usr/local/bin/git-status • /usr/local/bin/git-imap-send /usr/local/bin/git-stripspace • /usr/local/bin/git-index-pack /usr/local/bin/git-submodule • /usr/local/bin/git-init /usr/local/bin/git-svn • /usr/local/bin/git-init-db /usr/local/bin/git-svnimport • /usr/local/bin/git-instaweb /usr/local/bin/git-symbolic-ref • /usr/local/bin/git-local-fetch /usr/local/bin/git-tag • /usr/local/bin/git-log /usr/local/bin/git-tar-tree • /usr/local/bin/git-lost-found /usr/local/bin/git-unpack-file • /usr/local/bin/git-ls-files /usr/local/bin/git-unpack-objects • /usr/local/bin/git-ls-remote /usr/local/bin/git-update-index • /usr/local/bin/git-ls-tree /usr/local/bin/git-update-ref • /usr/local/bin/git-mailinfo /usr/local/bin/git-update-server-info • /usr/local/bin/git-mailsplit /usr/local/bin/git-upload-archive • /usr/local/bin/git-merge /usr/local/bin/git-upload-pack • /usr/local/bin/git-merge-base /usr/local/bin/git-var • /usr/local/bin/git-merge-file /usr/local/bin/git-verify-pack • /usr/local/bin/git-merge-index /usr/local/bin/git-verify-tag • /usr/local/bin/git-merge-octopus /usr/local/bin/git-whatchanged • /usr/local/bin/git-merge-one-file /usr/local/bin/git-write-tree • /usr/local/bin/git-merge-ours /usr/local/bin/gitk • /usr/local/bin/git-merge-recursive

15. easy_install Paver

16. paver paverdocs

17. pavement.py

18. Why Python build files? • You already know Python • The language rules are well-defined • The language rules are well-documented • Python is powerful, so you’ll never be left hanging or need an escape hatch

19. Configuration • options( • setup = setup_meta, • minilib=Bunch( • extra_files=['doctools', 'virtual'] • ), • sphinx=Bunch( • builddir="build", • sourcedir="source" • ), • virtualenv=Bunch( • packages_to_install=["nose", "sphinx", "docutils", "virtualenv"], • install_paver=False, • script_name='bootstrap.py', • paver_command_line=None • ) • )

20. Dynamic config values • >>> from paver.defaults import * • >>> import time • >>> options(current=lambda: time.time()) • >>> options.current • 1216726815.0027969

21. Namespace searching • >>> options( • ... setup=Bunch(version="1.0"), • ... sphinx=Bunch(builddir="docbuild") • ... ) • >>> options.version • '1.0'

22. Namespace searching (continued) • >>> options( • ... setup=Bunch(version="1.0"), • ... sphinx=Bunch(builddir="docbuild") • ... ) • >>> options.order('sphinx') • >>> options.version • Traceback (most recent call last): • File "<stdin>", line 1, in <module> • File "/Users/admin/projects/paver/paver/runtime.py", line 31, in __getattr__ • raise AttributeError(name) • AttributeError: version

23. Configuration is still standard Python • You can treat options like a normal, nested dictionary • The only unusual thing would be that callables are called.

25. Tasks • @task • def clean(): • """Cleans up this paver directory. Removes the virtualenv traces and • the build directory.""" • pass

26. paver help • paver help • ---> paver.tasks.help • Usage: paver [global options] taskname [task options] [taskname [taskoptions]] • Options: • --version show program's version number and exit • -n, --dry-run don't actually do anything • -v, --verbose display all logging output • -q, --quiet display only errors • -i, --interactive enable prompting • -f FILE, --file=FILE read tasks from FILE [pavement.py] • -h, --help display this help information • Tasks from paver.misctasks: • generate_setup - Generates a setup

27. paver help <taskname> • Usage: paver paver.misctasks.minilib [options] • Options: • -h, --help display this help information • Create a Paver mini library that contains enough for a simple • pavement.py to be installed using a generated setup.py. This • is a good temporary measure until more people have deployed paver. • The output file is 'paver-minilib.zip' in the current directory. • Options: • extra_files • list of other paver modules to include (don't include the .py • extension). By default, the following modules are included: • defaults, path, release, setuputils, misctasks, options, • tasks, easy

28. @needs • @task • @needs("uncog") • def commit(): • """Removes the generated code from the docs and then commits to bzr.""" • pass

29. @cmdopts • @task • @cmdopts([("username=", "u", "Username for remote server"), • ("server=", "s", "Server to deploy to")]) • def deploy(): • """Copy the Paver website up.""" • pass

30. paver <taskname> • $ paver generate_setup minilib • ---> generate_setup • Write setup.py • ---> minilib • Generate paver-minilib.zip

31. Paver and Distutils/Setuptools

32. python setup.py install

33. Paver embraces and extends Distutils (don’t worry, it’s not evil)

34. paver install

35. setup.py example • from distutils.core import setup • setup(name='foo', version='1.0', py_modules=['foo'], )

36. Upgrading to Paver • from paver.setuputils import setupsetup(name='foo', version='1.0', py_modules=['foo'], )

37. Keeping it simple for users • $ paver generate_setup minilib • ---> generate_setup • Write setup.py • ---> minilib • Generate paver-minilib.zip

38. The Paver Standard Library (is actually newer and less musty)

39. The Paver Standard Library • Easy scripting (paver.easy) • Distutils integration (paver.setuputils) • File handling (paver.path) • Documentation tools (paver.doctools) • Subversion (paver.svn) • SSH (paver.ssh) • Virtualenv (paver.virtual) • Miscellaneous Tasks (paver.misctasks)

40. paver.easy • from paver.easy import * • # display text if verbose is set • debug(“Hi there. Feeling chatty today?”) • # display text if quiet is not set • info(“Glad we don’t have to keep quiet”) • # display text regardless of setting • error(“HA! I’M SHOUTING AND YOU CAN’T STOP ME!”)

41. paver.easy (continued) • # run a command, as long as dry-run is off • # capture the output into myval • myval = sh(“cat /tmp/foo”, capture=True) • # run a function (delete_all(‘/’) to be exact). • # if dry-run is set, then • # just print the message instead • dry(“Delete everything”, delete_all, ‘/’)

42. paver.path • Jason Orendorff’s path.py module (also available separately) • It’s a subclass of string! • Fun use of operator overloading • Lots of great methods • Makes working with files/directories fun and easy

43. paver.path examples • p = path("docs")tmpdir = p /"tmp"tmpdir.mkdir()fn = tmpdir /"myfile.txt"fn.write_text("Hi there!")

44. paver.doctools • # add this to use • import paver.doctools

45. paver.doctools – Sphinx • paver.doctools.html() • ¶ • Build HTML documentation using Sphinx. This uses the following options in a “sphinx” section of the options. • docroot • the root under which Sphinx will be working. Default: docs • builddir • directory under the docroot where the resulting files are put. default: build • sourcedir • directory under the docroot for the source files default: (empty string) • paver.doctools.doc_clean()¶ • Clean (delete) the built docs. Specifically, this deletes the build directory under the docroot. See the html task for the options list.

46. paver.doctools – Cog & SectionedFile • Solutions to common problems of creating high-quality docs • Ideally, your code samples will be in convenient runnable code files and have unit tests. • But you also want nice, minimal code samples in your text as you’re writing. • Ned Batchelder’s Cog (included with Paver) and paver.doctools.SectionedFile make these easy. • Not Sphinx-specific at all

47. paver.doctools sample code • # mysample.py# [[[section mysample]]]defsample_func():print "To sample, or not to sample?"# [[[endsection]]]

48. paver.doctools in docs • And then when you want to print the sample string, you just call:: • # [[[cog include(“code/mysample.py”, “mysample”)]]] • # [[[end]]]

49. paver.doctools in docs (2) • And then when you want to print the sample string, you just call:: • # [[[cog include(“code/mysample.py”, “mysample”)]]] • def sample_func(): • print "To sample, or not to sample?" • # [[[end]]]

50. paver.doctools uncog before commit • @task • @needs("uncog") • def commit(): • """Removes the generated code from the docs and then commits to bzr.""" • sh("bzr commit")