Compiling Against Python.h

How to make sure that cython can find python.h when using a self compiled python.

python
cython
random-code-snippets
Author

Robert M Flight

Published

June 22, 2022

Sometimes, the installed version of Python on your linux system isn’t what you need, and you can’t easily install a newer version system-wide. Of course, you can download, extract, and configure and then make Python that lives in it’s own directory. But then you don’t have all the pointers to the Include directories that are useful for linking stuff against. And you don’t want to change symlinks at the system level, that might break things, a lot of things.

If you are using cython to speed up your code, then it needs to know where exactly to find those extra header files. Unfortunately, it needs both the C++ and the C versions, it seems.

Lets go through the steps of setting up a new python, and then linking it for compiling cythonized programs.

Download and Make

Download python, and navigate to where you saved it. I’ll assume we downloaded version 10 to ~/software

cd ~/software
tar -xzf Python-3.10.5.tgz
cd Python-3.10.5
./configure --enable-optimizations
make

Setup New Virtualenv With It

Make sure you have virtualenv installed, its necessary if you want to specify a specific version of python. Yes, you could probably get around this with conda or docker, but we are doing things the painful way.

virtualenv -p ~/software/Python-3.10.5/python ~/py10_venv
source ~/py10_venv/activate

Add It to Environment

export C_INCLUDE_PATH=/home/rmflight/software/Python-3.10.5:$C_INCLUDE_PATH

Install Something Cython

We are going to install a package that needs only a few python deps, our labs icikt package.

# check that you are using the right version in the venv
python3 --version
python3 -m pip install docopt
python3 -m pip install scipy
python3 -m pip install cython
cd ~/software
git clone git@github.com:MoseleyBioinformaticsLab/icikt.git
python3 -m pip install --global-option=build_ext --global-option="-I/home/rmflight/software/Python-3.10.5/Include" ~/software/icikt

Notice we’ve done two very important things overall:

  1. Added a path, C_INCLUDE_PATH
  2. Added two --global-option in the call to build our local package

For some reason we need both the environment variable path, and the global option to get this to work.

If you forget one or the other, you will see error messages concerning python.h

Reuse

Citation

BibTeX citation:
@online{mflight2022,
  author = {Robert M Flight},
  title = {Compiling {Against} {Python.h}},
  date = {2022-06-22},
  url = {https://rmflight.github.io/posts/2022-06-22-compiling-against-pythonh},
  langid = {en}
}
For attribution, please cite this work as:
Robert M Flight. 2022. “Compiling Against Python.h.” June 22, 2022. https://rmflight.github.io/posts/2022-06-22-compiling-against-pythonh.