Extending Python with C (Part I – the Basics) June 2002

1. Extending Python with C(Part I – the Basics)June 2002 Brian Quinlan brian@sweetapp.com

2. Why Mess with C? • There is a preexisting library, available in C, that you would like to access from Python (e.g. pyexpat, math, zlib) • Solving the problem in Python would be too CPU or memory inefficient (e.g. Numeric, PIL) • Only way to create new fundament types (doesn’t seem too important to me; save it for another talk)

3. Jumpin’ In • Create our own extension module containing an eclectic set of functions • We’ll call it: Vanpy • Basic steps: • Do module initialization • Write functions • Write build script • Testing

4. A module is just a dictionary foo.py magic_number = 5 def divide(x,y): return x / y >>> import foo >>> foo.__dict__ {'__doc__': None, 'magic_number': 5, 'divide': <function divide at 007B1484>, …} >>> foo.__dict__['divide'](8,4) 2

5. Module Initialization void initVanpy(void) { PyObject * module; module = Py_InitModule3("Vanpy", module_functions, module_doc); PyModule_AddStringConstant(module, "__version__", "0.0.1"); }

6. A VERY Simple Function static PyObject * is_even(PyObject * self, PyObject * args) { long number; if (!PyArg_ParseTuple(args, "i", &number)) return NULL; return PyBuildValue("i", number % 2 == 0) } static PyMethodDef module_functions[] = { {"is_even", is_even, METH_VARARGS, is_even_doc}, {NULL, NULL} }

7. A Simple Function static PyObject * classify_characters(PyObject * self, PyObject * args) { char * string; long length; long alpha, other = 0; if (!PyArg_ParseTuple(args, "s#", &string, &length)) return NULL; for (int i = 0; i < length; ++i) { isalnum(string[i]) ? ++alpha : ++other } return PyBuildValue("(iif)", alpha, other, ((float) alpha) / length) }

8. Objects and Types typedef struct { struct _typeobject *ob_type; int ob_refcnt; /* The rest depends on the type */ } PyObject; typedef struct _typeobject { char *tp_name; /* Stuff to discuss in another talk */ }

9. 1 Hello a Reference Counting >>> a = “Hello” PyObject * a = Py_BuildValue(“s”, “Hello”)

10. 1 2 Hello Hello a Reference Counting >>> a = “Hello” PyObject * a = Py_BuildValue(“s”, “Hello”) >>> b = a PyObject * b = a Py_INCREF(b) a b

11. 1 2 1 Hello Hello Hello a Reference Counting >>> a = “Hello” PyObject * a = Py_BuildValue(“s”, “Hello”) >>> b = a PyObject * b = a Py_INCREF(b) >>> del a Py_DECREF(a) a b b

12. 1 2 1 0 Hello Hello Hello Hello a Reference Counting >>> a = “Hello” PyObject * a = Py_BuildValue(“s”, “Hello”) >>> b = a PyObject * b = a Py_INCREF(b) >>> del a Py_DECREF(a) >>> del b Py_DECREF(b) a b b

13. Reference Counting Rules • When returning a PyObject from an extension function or method, always increment the object’s reference count • USUALLY, when calling a Python API function, you need not increment the reference counts of PyObject arguments (only 2 important exceptions) • USUALLY, when a PyObject is returned from a Python API function, you must decrement the object’s reference count when you are done with it (> 4 important exceptions)

14. A “Realistic” Example static PyObject * password = NULL; static PyObject * set_password(PyObject * self, PyObject * args) { PyObject * new_password; if (!PyArg_ParseTuple(args, "O", &new_password)) return NULL; if (password != NULL) /* or Py_XDECREF(password) */ Py_DECREF(password); password = new_password; Py_INCREF(password); Py_INCREF(Py_None); return Py_None; }

15. A “Realistic” Example static PyObject * get_password(PyObject * self, PyObject * args) { if (!PyArg_ParseTuple(args, "")) return NULL; if (password == NULL) { Py_INCREF(Py_None); return Py_None; } else { Py_INCREF(password); return password; } }

16. Exceptions • Each Python thread has 3 exception variables associated with it: • The type of the exception (e.g. ValueError) • The value of the exception (e.g. “0 denominator”) • A traceback object (e.g. file “foo.py”, line 1, in “divide”) def divide(x,y): raise ValueError, “0 denominator” static PyObject * divide(PyObject * self, PyObject * args) { PyErr_SetString(PyExc_ValueError, “0 denominitor”); return NULL; }

17. Passwords Revised static PyObject * set_password(PyObject * self, PyObject * args) { PyObject * new_password; if (!PyArg_ParseTuple(args, "O", &new_password)) return NULL; if (!PyString_Check(new_password) && !PyUnicode_Check(new_password)) { PyErr_Format(PyExc_TypeError, "expected string or " "Unicode , %80s found", new_password->ob_type->tp_name); return NULL; } ...

18. More Exceptions • If you want to ignore an error, use PyErr_Clear to clear the current exception • You can check to see if an exception has been set using PyErr_Occurred (this is not usually need since the API return value indicates failure) • It is an error to return NULL without setting an exception

19. Custom Exceptions I static PyObject * WeatherError; void initVanpy(void) { PyObject * module; PyObject * module_dict; module = Py_InitModule3("Vanpy", module_methods, module_doc); WeatherError = PyErr_NewException("Vanpy.WeatherError", NULL, NULL); module_dict = PyModule_GetDict(module); PyDict_SetItemString( module_dict, "WeatherError", WeatherError); }

20. Custom Exceptions II static PyObject * goto_beach(PyObject * self, PyObject * args) { if (!PyArg_ParseTuple(args, "")) return NULL; if (rand() % 2) { PyObject * e = Py_BuildValue("(si)", "too cold", 5); if (e == NULL) return NULL; PyErr_SetObject(WeatherError, e); Py_DECREF(e); } else { PyErr_SetString(WeatherError, "too rainy"); } return NULL; }

21. Building setup.py #!/usr/bin/env python from distutils.core import setup, Extension setup( name = "Vanpy", version = "0.0.1", description = "An eclectic set of functions", author = "Brian Quinlan", author_email = "brian@sweetapp.com", ext_modules = [ Extension("Vanpy", ["Vanpy.c"]) ] )

22. Python API organization • Exception handling • Memory Management • Threads • Utilities • OS Utilities • Process Control • Importing Modules • Data Marshalling • Parsing Arguments & Building Values • Abstract Object Support • Concrete Object Support

23. Abstract Object Support • Applies to classes of objects, not specific types • Divided into protocols: • Object • Number • Sequence • Mapping • Iterator • Buffer

24. Object Protocol A sampling of functions in the object protocol: int PyObject_HasAttrString(PyObject *o, char *attr_name) int PyObject_Cmp(PyObject *o1, PyObject *o2, int *result) PyObject* PyObject_Repr(PyObject *o) PyObject* PyObject_Str(PyObject *o) int PyObject_IsInstance(PyObject *inst, PyObject *cls) PyObject* PyObject_CallFunction(PyObject *callable, char *format, ...) PyObject* PyObject_CallMethod(PyObject *o, char *method, char *format, ...) int PyObject_IsTrue(PyObject *o)

25. Number Protocol A sampling of functions in the number protocol: int PyNumber_Check(PyObject *o) PyObject* PyNumber_Add(PyObject *o1, PyObject *o2) PyObject* PyNumber_Subtract(PyObject *o1, PyObject *o2) PyObject* PyNumber_Multiply(PyObject *o1, PyObject *o2) PyObject* PyNumber_Negative(PyObject *o) PyObject* PyNumber_Absolute(PyObject *o) PyObject* PyNumber_Lshift(PyObject *o1, PyObject *o2) PyObject* PyNumber_And(PyObject *o1, PyObject *o2) PyObject* PyNumber_Int(PyObject *o) PyObject* PyNumber_Long(PyObject *o) PyObject* PyNumber_Float(PyObject *o)

26. Sequence Protocol A sampling of functions in the sequence protocol: int PySequence_Check(PyObject *o) int PySequence_Length(PyObject *o) PyObject* PySequence_Repeat(PyObject *o, int count) PyObject* PySequence_GetItem(PyObject *o, int i) PyObject* PySequence_GetSlice(PyObject *o, int i1, int i2) int PySequence_SetItem(PyObject *o, int i, PyObject *v) int PySequence_DelItem(PyObject *o, int i) PyObject* PySequence_Tuple(PyObject *o) int PySequence_Contains(PyObject *o, PyObject *value) int PySequence_Count(PyObject *o, PyObject *value)

27. Concrete Object Support • Applies to specific object types • Different set of functions for every built-in Python type • There any many Python object types: plain int, long int, float, complex, string, unicode, buffer, tuple, list, dictionary, file, instance, method, module, iterator, slice, weak reference, etc. • Reference counting rules are less intuitive

28. Int Objects This is the complete list of int functions: int PyInt_Check(PyObject* o) int PyInt_CheckExact(PyObject* o) PyObject* PyInt_FromLong(long ival) long PyInt_AsLong(PyObject *io) long PyInt_AS_LONG(PyObject *io) long PyInt_GetMax()

29. String Objects A sampling of string functions: int PyString_Check(PyObject *o) int PyString_CheckExact(PyObject *o) PyObject* PyString_FromString(const char *v) PyObject* PyString_FromStringAndSize(const char *v, int len) PyObject* PyString_FromFormat(const char *format, ...) PyObject* PyString_FromFormatV(const char *format, va_list vargs) int PyString_Size(PyObject *string) int PyString_GET_SIZE(PyObject *string) char* PyString_AsString(PyObject *string) char* PyString_AS_STRING(PyObject *string) void PyString_Concat(PyObject **string, PyObject *newpart)

30. List Objects This is the complete list of list functions: int PyList_Check(PyObject *p) PyObject* PyList_New(int len) int PyList_Size(PyObject *list) int PyList_GET_SIZE(PyObject *list) PyObject* PyList_GetItem(PyObject *list, int index) PyObject* PyList_GET_ITEM(PyObject *list, int i) int PyList_SetItem(PyObject *list, int index, PyObject *item) void PyList_SET_ITEM(PyObject *list, int i, PyObject *o) int PyList_Insert(PyObject *list, int index, PyObject *item) int PyList_Append(PyObject *list, PyObject *item) PyObject* PyList_GetSlice(PyObject *list, int low, int high) int PyList_SetSlice(PyObject *list, int low, int high, PyObject *itemlist) int PyList_Sort(PyObject *list)

31. Helpful Technologies • SWIG/SIP - generate C/C++ interface code for Python • Boost Python - an interface generator for wrapping C++ classes with a Python interface • CXX - C++ wrapper around the Python API • Pyrex - a language for writing Python extension modules • Pyfort - a wrapper generator for interfacing Fortran with Python • PyInline/Weave – embed C/C++/Perl code inside your Python script