Python extensions with C libraries made easy by Cython

Suppose you have a library written in C that you like to be called from Python. There are many ways to accomplish this, and I would like to show here a complete example of doing this using Cython, which is basically a compiler from (an extension of Python) to C. Most of this, and much more, can be found in a quick tutorial.
For the purpose of an example, suppose you have a C function that computes the average of an array of doubles (all source files here, as well as the Makefile that basically contains the shell commands we show below, can be downloaded here, or cloned using git from github):

/* cmean.c */
double mean(int n, double* a)
{
double s;
int i;
for (s=0., i=0; i<n; i++) s+=*(a++);
return s/n;
}

with the prototype

/* cmean.h */
double mean(int, double*);

and you want to call it from Python, i.e. (we need to give the function another name here, cmean(), for technical as well as for semantic reasons—noone would pass the length of an array and the array to a function in Python, as the array itself would certainly do) say

b=[1.3,5.777777,-12.0,77.]
print cmean(b)

would print the same as

print sum(b)/len(b)

As cmean() is not a Python built-in, we would expect to import it first. So the following

$ python test.py

where

# test.py
import dyn
from dyn import cmean
b=[1.3,5.777777,-12.0,77.]
print cmean(b)
print sum(b)/len(b)

will print the number 18.01944425 twice. OK, so here is cmean() definition in Cython (note the file extension .pyx)

# m.pyx
cdef extern from "cmean.h":
double mean(int, double*)
from stdlib cimport *
def cmean(a):
n = len(a)
cdef double *v
v = malloc(n*sizeof(double))
for i in range(n):
v[i] = float(a[i])
m = mean(n, v)
free(v)
return m

The main task of cmean() is to create a “plain” C array of doubles, pass it to our C function, get the result, clean after itself, and return the result. Unlike Python, Cython has types declarations. Unless one calls C-function from the Cython code, they are not mandatory though, although they can speed the things up tremendously in “real” computations. In the example here

cdef double *v

cannot be avoided — the code does not compile without it. In a usual C-like fasion, instead of 2 lines

cdef double *v
v = malloc(n*sizeof(double))

one could have written just

cdef double *v = malloc(n*sizeof(double))

The top two lines pass the prototype declaration of the C-function mean to be included into the C file to be generated, and the third line imports the functions from the C library stdlib (we need malloc and free from there).

Having this, we need to create the extension module dyn, to be imported by Python. This is perhaps the least intuitive part of the job. One way to accomplish it is to use Python distutils: i.e. we need to create a setup.py file to be called as

$ python setup.py build_ext --inplace

This should look as follows:

# setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules=[
Extension("dyn",
["m.pyx"],
library_dirs = ['.'],
libraries=["cmean"]) # Unix-like specific
]
setup(
name = "Demos",
cmdclass = {"build_ext": build_ext},
ext_modules = ext_modules
)

One not yet explained thing here is how the shared library containing the compiled C-function mean() is hooked up to dyn. You see in the code above the declarations library_dirs and libraries. They assume that the shared library is called libcmean.so and that it is in the same directory (i.e. ‘.’ is the path to it) as the rest of the code. One can create libcmean.so (before creating dyn, more precisely, a file called dyn.so) by running

$ gcc -fPIC -shared cmean.c -o libcmean.so

One more point to watch is that on some Unix systems the loader will be unable to locate the dynamic libraries libcmean.so and/or dyn.so we created (unless they are placed in certain stanadard directories, which is not something one wants to do with experimental code!). So test.py will need to be run, e.g., as follows:

$ LD_LIBRARY_PATH=. python test.py

For the sake of completeness, the aforementioned archive contains a C file that calls mean from libcmean.so, and the corresponding entries in the Makefile can do the necessary work to build and run it.

Last but not least, Sage automates parts of the interface building even better.

Advertisements

Tags: ,

8 Responses to “Python extensions with C libraries made easy by Cython”

  1. Dinesh Says:

    I followed your instructions, but while compiling the setup.py file , I get

    Cannot assign type ‘void *’ to ‘double *’ error.
    Can you help me with this?

    • Dima Says:

      What is the full error message? (Most probably an explicit cast is missing somewhere…)

      • Dinesh Says:

        Error compiling Cython file:
        ————————————————————

        double mean(int, double*)
        from stdlib cimport *
        def cmean(a):
        n = len(a)
        cdef double *v
        v = malloc(n*sizeof(double))
        ^
        ————————————————————

        m.pyx:7:14: Cannot assign type ‘void *’ to ‘double *’
        building ‘dyn’ extension
        /usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/harishraja/anaconda/include -arch x86_64 -I/Users/harishraja/anaconda/include/python3.5m -c m.c -o build/temp.macosx-10.6-x86_64-3.5/m.o

        This is the traceback.

        • Dima Says:

          It works for me with Python 2.7.12 (and cython-0.25.2) on Linux (and gcc is version 5.4.0).

          On newer versions of OSX gcc is actually not gcc, but clang, and I never tested with these.

          Try adding explict cast to malloc, i.e.

          v = (double *)malloc(n*sizeof(double))

  2. Olivia Guest (@o_guest) Says:

    I get the following error on my MBpro:

    MBpro:dyn oliviaguest$ LD_LIBRARY_PATH=. python test.py
    Traceback (most recent call last):
    File “test.py”, line 1, in
    import dyn
    ImportError: dlopen(/Users/oliviaguest/GitHub/pdist/dyn/dyn.so, 2): Library not loaded: libcmean.so
    Referenced from: /Users/oliviaguest/GitHub/pdist/dyn/dyn.so
    Reason: unsafe use of relative rpath libcmean.so in /Users/oliviaguest/GitHub/pdist/dyn/dyn.so with restricted binary

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: