Compiling Against Python.h

python cython random-code-snippets

How to make sure that cython can find python.h when using a self compiled python.

Robert M Flight true
2022-06-22

Sometimes, the installed version of Python on your linux system isn’t what you need, and you can’t easily install a newer version system-wide. Of course, you can download, extract, and configure and then make Python that lives in it’s own directory. But then you don’t have all the pointers to the Include directories that are useful for linking stuff against. And you don’t want to change symlinks at the system level, that might break things, a lot of things.

If you are using cython to speed up your code, then it needs to know where exactly to find those extra header files. Unfortunately, it needs both the C++ and the C versions, it seems.

Lets go through the steps of setting up a new python, and then linking it for compiling cythonized programs.

Download and Make

Download python, and navigate to where you saved it. I’ll assume we downloaded version 10 to ~/software

cd ~/software
tar -xzf Python-3.10.5.tgz
cd Python-3.10.5
./configure --enable-optimizations
make

Setup New Virtualenv With It

Make sure you have virtualenv installed, its necessary if you want to specify a specific version of python. Yes, you could probably get around this with conda or docker, but we are doing things the painful way.

virtualenv -p ~/software/Python-3.10.5/python ~/py10_venv
source ~/py10_venv/activate

Add It to Environment

export C_INCLUDE_PATH=/home/rmflight/software/Python-3.10.5:$C_INCLUDE_PATH

Install Something Cython

We are going to install a package that needs only a few python deps, our labs icikt package.

# check that you are using the right version in the venv
python3 --version
python3 -m pip install docopt
python3 -m pip install scipy
python3 -m pip install cython
cd ~/software
git clone git@github.com:MoseleyBioinformaticsLab/icikt.git
python3 -m pip install --global-option=build_ext --global-option="-I/home/rmflight/software/Python-3.10.5/Include" ~/software/icikt

Notice we’ve done two very important things overall:

  1. Added a path, C_INCLUDE_PATH
  2. Added two --global-option in the call to build our local package

For some reason we need both the environment variable path, and the global option to get this to work.

If you forget one or the other, you will see error messages concerning python.h

Corrections

If you see mistakes or want to suggest changes, please create an issue on the source repository.

Reuse

Text and figures are licensed under Creative Commons Attribution CC BY 4.0. Source code is available at https://github.com/rmflight/researchBlog_distill, unless otherwise noted. The figures that have been reused from other sources don't fall under this license and can be recognized by a note in their caption: "Figure from ...".

Citation

For attribution, please cite this work as

Flight (2022, June 22). Deciphering Life: One Bit at a Time: Compiling Against Python.h. Retrieved from https://rmflight.github.io/posts/2022-06-22-compiling-against-pythonh/

BibTeX citation

@misc{flight2022compiling,
  author = {Flight, Robert M},
  title = {Deciphering Life: One Bit at a Time: Compiling Against Python.h},
  url = {https://rmflight.github.io/posts/2022-06-22-compiling-against-pythonh/},
  year = {2022}
}