Need to run your written python code using another cluster that does not have the libraries? As it was my first time using libraries outside of what we have for CORAL this was a little bit challenging for me, surely if it was c++, I would not have the time to write this blog entry and would be still wrestling with the libraries.
This tutorial is basic and If you can work smoothly with Python, the following notes are not for you, it only benefits individuals fairly new to Python who try to run their codes on a cluster without Administrator permissions and got confused like me how to tackle this problem.
So for a Binary SDP solver project for Integer Programming course we had to run so many tests in a code using libraries such as Pulp and Coinor.Gimpy, where the first one is a linear programming modeling framework for Python which works smoothly with great solvers such as CPLEX and GUROBI, and the second one is a great Graph Library, which is needed to manage the Branch and Bound Tree structure.
First we would make a directory in the cluster to install packages locally there, say “/home/sey212/pkg”. Now we would try to install the packages on this folder. To install a package we normally use “Pip install “package” ” however, here we need more than that since we are not administrator of the system.
pip install –ignore-installed –install-option=”–prefix=/home/sey212/pkg” coinor.gimpy
This way we install the library to the folder. Here I’ve use “–ignore-installed” argument, since the node I have access to, has this library installed however when submitted through Condor, the file runs on other nodes not having this library.
pip install –ignore-installed –install-option=”–prefix=/home/sey212/pkg” pulp
Now that I’ve libraries I need, I just need to use Environmental Variables to tell Python interpreter to look in my local folder for this libraries. This can be done by:
export PYTHONPATH=$PYTHONPATH:/home/sey212/pkg/lib/python2.7/site-packages
Which adds the local folder to the PYTHONPATH. Now, since the my home folder is mounted on any node I login, I can run the code in any node I want.
For running the code using Condor, I’d suggest writing a Bash Script in which you extend the PYTHONPATH variable and then run the python code in it, and submit the Bash script through Condor. However this is the easiest way Ican think of to run such thing on nodes managed by Condor Scheduler. It is important to have “get_env=True” in you condor submission file, so that condor does not ignore your modified variable.