Added post on integrating Cython with setuptools

1 files changed, 143 insertions, 0 deletions

diff --git a/content/development/integrate_cython_setuptools.rst b/content/development/integrate_cython_setuptools.rst
new file mode 100644
index 0000000..04fa94d
--- /dev/null
+++ b/content/development/integrate_cython_setuptools.rst
@@ -0,0 +1,143 @@
+Integrating Cython extension with setuptools and unit testing
+#############################################################
+
+:date: 2016-06-26 19:54 +0530
+:slug: cython_setuptools
+:tags: python, cython, setuptools, pbr
+:author: copyninja
+:summary: This post describes how to integrate a cython extension with
+	  setuptools.
+
+
+I was reviewing changes for `indic-trans
+<https://github.com/libindic/indic-trans>`_ as part of GSoC 2016. The
+module is an improvisation for our original transliteration module
+which was doing its job by substitution.
+
+This new module uses machine learning of some sort and utilizes
+*Cython*, *numpy* and *scipy*. Student had kept pre-compiled shared
+library in the git tree to make sure it builds and passes the
+test. But this was not correct way. I started looking at way to build
+these files and remove it from the code base.
+
+There is a `cython documentation
+<http://docs.cython.org/src/quickstart/build.html>`_ for distutils but
+none for setuptools. Probably it is similar to other Python extension
+integration into setuptools, but this was first time for me so after a
+bit of searching and trial and error below is what I did.
+
+We need to use `Extensions` class from setuptools and give it path to
+modules we want to build. In my case `beamsearch` and `viterbi` are 2
+modules. So I added following lines to setup.py
+
+.. code-block:: python
+
+   from setuptools.extension import Extension
+   from Cython.Build import cythonize
+
+   extensions = [
+    Extension(
+        "indictrans._decode.beamsearch",
+        [
+            "indictrans/_decode/beamsearch.pyx"
+        ],
+        include_dirs=[numpy.get_include()]
+    ),
+    Extension(
+        "indictrans._decode.viterbi",
+        [
+            "indictrans/_decode/viterbi.pyx"
+        ],
+        include_dirs=[numpy.get_include()]
+    )
+
+   ]
+
+First argument to `Extensions` is the module name and second argument
+is a list of files to be used in building the module. The additional
+`inculde_dirs` argument is not normally necessary unless you are
+working in virtualenv. In my system the build used to work without
+this but it was failing in Travis CI, so added it to fix the CI
+builds. OTOH it did work without this on Circle CI.
+
+Next is provide this `extensions` to `ext_modules` argument to `setup`
+as shown below
+
+.. code-block:: python
+
+   setup(
+    setup_requires=['pbr'],
+    pbr=True,
+    ext_modules=cythonize(extensions)
+   )
+
+And for the reference here is full setup.py after modifications.
+
+.. code-block:: python
+
+   #!/usr/bin/env python
+
+   from setuptools import setup
+   from setuptools.extension import Extension
+   from Cython.Build import cythonize
+
+   import numpy
+
+
+   extensions = [
+     Extension(
+        "indictrans._decode.beamsearch",
+        [
+            "indictrans/_decode/beamsearch.pyx"
+        ],
+        include_dirs=[numpy.get_include()]
+     ),
+     Extension(
+        "indictrans._decode.viterbi",
+        [
+            "indictrans/_decode/viterbi.pyx"
+        ],
+        include_dirs=[numpy.get_include()]
+     )
+   ]
+
+   setup(
+     setup_requires=['pbr'],
+     pbr=True,
+     ext_modules=cythonize(extensions)
+   )
+
+So now we can build the extensions (shared library) using following
+command.
+
+.. code-block:: shell
+
+   python setup.py build_ext
+
+Another challenge I faced was missing extension when running test. We
+use pbr in above project and testrepository with subunit for running
+tests. Looks like it does not build extensions by default so I
+modified the Makefile to build the extension in place before running
+test. The `travis` target of my Makefile is as follows.
+
+.. code-block:: make
+
+   travis:
+	[ ! -d .testrepository ] || \
+		find .testrepository -name "times.dbm*" -delete
+	python setup.py build_ext -i
+	python setup.py test --coverage \
+		--coverage-package-name=indictrans
+	flake8 --max-complexity 10 indictrans
+
+I had to build the extension in place using `-i` switch. This is
+because other wise the tests won't find the
+`indictrans._decode.beamsearch` and `indictrans._decode.viterbi
+modules`. What basically `-i` switch does is after building shared
+library symlinks it to the module directory, in ourcase
+`indictrans._decode`
+
+The test for existence of .testrepository folder is over come `this
+bug  <https://bugs.launchpad.net/testrepository/+bug/1229445>`_ in
+testrepository which results in test failure when running tests using
+`tox`.