the-python-programming-language-logo

Python Package Management

The sordid history of Python library and package management is full of hidden twists, perilous turns, and dark corners hiding sinister beasts. Good news: you don’t have to worry about any of that.

Modern Python provides an application called pip, which allows you to easily install third-party Python libraries and applications. It incorporates many of the lessons learned from its predecessors, sidestepping problems that previously had to be dealt with manually. And it works very well with Python virtual environments.

The first step is to install pip. With Python 3, this is included for you automatically, and is installed in your virtual environment:

Example
$ source venv/bin/activate
(venv)$ python -V
Python 3.6.0
(venv)$ which pip
/Users/paul/myapp/venv/bin/pip

For Python 2, if pip is already installed on the system, your virtual environment will be created to include it. For macOS and Windows, recent versions of Python 2 automatically include pip; if not, you can quickly find out how to install it by searching online.

Once you have the pip executable, you can use pip install to install libraries just within the virtutal environment. For example, requests is a high-quality HTTP library. Install it like so:

Ex.
pip install requests

This is the install command. You will see some output, narrating the process of installing requests at a specific version. Once complete, you will be able to open a Python prompt and import requests.

The pip install command is also used to upgrade packages. For example, sometimes a fresh virtual environment may install a slightly stale version of pip. pip is just another package, so you can upgrade it with the -U or –upgrade option:

Ex.
pip install –upgrade pip

Installed packages are, by default, fetched from Pypi – the official online Python package repository. Any package or library listed at https://pypi.python.org/pypi can be installed with pip. You can uninstall them with pip uninstall.

Now, some of these packages’ files are substantial, or even compiled into object files. You definitely don’t want to put them in version control. How do you register the exact version dependencies your app has for third-party libraries? And how do you manage upgrades (and even downgrades) over time?

pip provides a good solution for this. The first part of it relies on the pip freeze command:

Ex.
(venv)$ pip freeze
requests==2.7.0

This prints the packages installed from Pypi, one per line, with the exact version. What you can do is place this in a file named requirements.txt:

(venv)$ pip freeze > requirements.txt
(venv)$ cat requirements.txt
requests==2.7.0

This file is what you will check into version control. You can recreate the application environment, right down to the minor versions of each dependent library, simply by passing requirements.txt to pip. Whether your coworker is fetching the raw source to start development, or if the devops team sets up a CI environment that runs the automated tests, the environment is consistent and well-defined, from development to staging to production.

You can pass any file path to python -m venv (and pyenv, and virtualenv). For organizational convenience, many choose to put it in the top-level folder of the repository holding the Python application. There are two schools of thought on what to name it.

One school picks a consistent name, which is used for every project. “venv” is very popular:

Ex.
python3 -m venv venv

The idea is that every Python project will have a folder in its top level called venv to contain the virtual environment. This has several advantages. For one, you can easily activate the virtual environment for any application, just by typing source venv/bin/activate. In fact, you can define a shell alias to help:

Ex.

# Type “venv” <enter> to get in the virtual environment.
alias venv=’source venv/bin/activate’

You can also configure your version control system to ignore any folder named “venv”, and thereby avoid ever accidentally committing your virtual environment. (You don’t want to do that. It’s a lot of files, and will annoy your fellow developers.)

The other naming scheme is to give a name that has something to do with the application. For example, for an application called “mywebapp”, you might create it like this:

Ex.
python3 -m venv mywebappenv

The advantage of this is that, when activated, the prompt is modifed to tell you which particular virtual environment your shell prompt is using. This can be helpful if you work with many different Python applications, as it’s much more informative than a prompt that just says”(venv)”.

The downside comes from the inconsistency of the folder name: keeping the folder out of version control is more error-prone, and activating the virtual environment requires the distraction of conscious thought each time. Both approaches are valid; it really comes down to which you and your teammates like better.