Back to basics with pip and venv

I wish you didn't need to know this, but you do

Summary

You can use pip to install things in Python, and virtual environments to increase the chances that it works. They are provided with python.org installers.

On Windows, use “py -3.X -m venv .venv” to create a virtual environment, and “.venv\Scripts\activate” to use it.

On Mac and Linux, use “python3.X -m venv .venv” to create a virtual environment, and “source .venv/bin/activate” to use it.

Once you have activated a virtual environment, you can install a thing by doing “python -m pip install thing”. “thing” will then be available in this virtual environment and only in this virtual environment.

I recommend to at least use a virtual environment for each project you have, and one for all your small scripts.

How to read this article

This article does not target experts. Even if you are not an expert, you may encounter things you know already, but this doesn't mean you can't learn a trick or two. Just skip the parts you don't care about.

I'll assume you have installed Python correctly, following the advices in the previous articles. This tutorial may not work if you have not followed those. Particularly, installing Python with homebrew, anaconda, the windows store or pyenv could all lead this tutorial to fail. This is why we stick to basics: fewer ways of failing.

This article is in the spirit of the post “Relieving your packaging pain” and therefore is not trying to solve all the problems in the world, but give one single simple procedure that can help the most people.

What is a virtual environment, and why to use one?

A virtual environment provides a way to say "all this stuff I install here is for this particular code, and all that stuff I install there is for that other code".

"Virtual environment" may sound like a complicated name, but it's for something that is actually simple.

It has nothing to do with a virtual machine or anything sophisticated: it's just a folder. A folder in which you are going to install stuff, that is separated from other stuff elsewhere.

A virtual environment is only useful if you plan to install third party tools. E.G: if you are going to use pip.

In fact, if you are going to install third party tools, you should never do so outside of a virtual environment.

Indeed, installing stuff outside of a virtual environment can cause you a lot of pain:

• It may fail while installing, because of PATH or permission problems.

• It may fail on use now, because of PATH or version problems.

• It may fail on use later on, because of conflict problems.

The virtual environment prevents that, making sure two unrelated codes don't share the same installed stuff.

Let's take an example.

You start a project to download pictures on the internet. To reach that goal, you install the excellent "requests" library.

Time passes, now you write a script to talk to ChatGPT, using their "openai" library.

You didn't know it, but "openai" also uses "requests", albeit a different version, and the two versions are incompatible.

Now something breaks, and you want to scream on the internet that "Python packaging sucks!".

However, if you do the same thing, but use one virtual environment for the "picture download" project, and one for the ChatGPT script, they will be completely separated from each other. They each have their own "requests", and they never see each other.

And you avoid the conflict problem.

Virtual environments prevent a lot of problems like this, and more.

If you used a different system like cargo or npm before, you may wonder why in Python we have to deal with virtual environment at all. We have an article to answer that.

virtualenv, poetry, pipenv or virtualenvwrapper?

None of them.

The first hindrance in learning about virtual environments is the number of options to manage them.

There are so many things that seem related to them.

In this article, we are going to use the tool named "venv", which is provided with Python.

You can safely ignore "virtualenv", "poetry", "pipenv" and "virtualenvwrapper"; all are third party tools than require prior installation.

Sometime, you will read articles that shorten the term "virtual environments" and call them "virtual envs", "virtualenvs", "venvs", "envs", etc., adding to the confusion.

There is nothing we can do about this except making clear in your head that there are two separate things:

• The concept of a virtual environment, meaning the idea of creating a folder to isolate stuff you install. Sometimes people say "use a virtualenv" or "use a venv". They mean "use a virtual environment", but it's longer to type.

• The tools to create virtual environments. Some are called "virtualenv", "poetry", "pipenv" or "virtualenvwrapper". We will use the one called "venv".

How to create a virtual environment

First, chose the Python you want to use.

A virtual environment is always linked to a single version of Python. When you create a one, you must decide for which version of Python you create it. Also, this version of Python must already be installed on your computer.

Remember we have an article about installing Python properly, which details how to know what versions are installed on your machine, and how you to run a specific version of Python. You can go read it now if needed.

The next step is to decide of the name of your virtual environment. I often name it ".venv" (with a dot), but you can name it "blabla", "my_env" or whatever you want. It's just the name of the folder it will create. We have a section later on about choosing the name.

Then, you must decide where to put your virtual environment. Usually, I put one at the top-level directory of each of my project.

Open a terminal (if you are on Windows and terminals, shells and prompts are not familiar topics, check out this article), and make sure you are in the directory you want the create the virtual environment.

Don't start a Python shell now. Those commands are to be run in your system shell. When you run those commands, you should NOT see a prompt with:

>>>

If this is the case, exit the Python shell first.

On windows

We are going to use the "py launcher", which is provided on Windows if you installed Python as we recommend.

Assuming you are already in a terminal, where you want the virtual environment to be created, can now create a new one with this command:

py -3.X -m venv NAME_OF_THE_ENV

With "X" being the minor version of Python you want, and "NAME_OF_THE_ENV" being the name you want to give to the virtual environment.

E.G., if I want to create a virtual environment named ".venv" with Python 3.10, I would use:

py -3.10 -m venv .venv

Remember you need the Python version to be installed on this machine for it to work. You cannot create a virtual environment for Python 3.10 if no Python 3.10 exist.

When the command is done, it will show nothing. But a new folder will appear in the directory you are.

On Mac or Linux

Assuming you are already in a terminal, where you want the virtual environment to be created, you can now create a new one with this command:

python3.X -m venv NAME_OF_THE_ENV

With "X" being the minor version of Python you want, and "NAME_OF_THE_ENV" being the name you want to give to the virtual environment.

E.G., if I want to create a virtual environment named ".venv" with Python 3.10, I would use:

python3.10 -m venv .venv

Remember you need the Python version to be installed on this machine for it to work. You cannot create a virtual environment for Python 3.10 if no Python 3.10 exist.

When the command is done, it will show nothing. But a new folder will appear in the directory you are.

If you named the virtual environment with a leading dot (such as ".venv"), remember Mac and Linux hide those folders by default. Make sure you display hidden folders, or you may be under the impression the command didn't work.

What's a good name for a virtual environment?

You can use any name for your virtual environment, but I like to keep it standardized, so that every time I open a project, I know immediately what folder contains it.

I usual name it ".venv", because:

• Some editors check for this name and automatically load it.

• It contains "venv", so it's explicit.

• It has a dot, so it's hidden on Linux.

• I avoid ".env", because it's popular for environment variables.

If I have several virtual environments in the same directory, I will use a suffix to distinguish them.

E.G., if I have a project that must work with two different versions of Python (3.9 and 3.10), I will have a ".venv39" and a ".venv310" virtual environment.

If I have a virtual environment that is not linked to a project, but for a specific purpose, I will name it according to that purpose.

E.G.:

• .venv_test: I have one like this in my personal directory to install new tools I just read about and want to play with. It's disposable, and often broken, so I delete it and recreate it regularly.

• .venv_scripts: I have one like this for all the small scripts. I don't want to create one virtual environment for each script, so I centralize everything. It's better than installing outside of a virtual environment, but is not a big constraint.

How to use a virtual environment

To use a virtual environment, the most convenient way is to activate it.

It's a fancy word to say we tell the terminal, "Now I want to work with the stuff from this folder, and only the stuff from this folder".

To activate it, you need to be in the terminal, in the same directory you create it.

Don't go inside the virtual environment folder. You should be right next to it.

Activate the virtual environment on Windows

Run the following command:

NAME_OF_THE_ENV\Scripts\activate

E.G., if the virtual environment is named ".venv":

.venv\Scripts\activate

You should now see the name of the virtual environment in your terminal prompt, telling you it's activated.

Activate the virtual environment on Mac or Linux

Run the following command:

source NAME_OF_THE_ENV/bin/activate

E.G., if the virtual environment is named ".venv":

source .venv/bin/activate

You should now see the name of the virtual environment in your terminal prompt, telling you it's activated.

What now?

Anytime you use the "python" command while your virtual environment is activated, it will be only the one from this env.

If you start a Python shell now, it will see only the things in the current directory, and the things installed in the virtual environment.

If you run a script, it will see only the things in the current directory, and the things installed in the virtual environment.

If you run a command, the command will be taken from the virtual environment.

And they will only use exactly the version of Python of the virtual environment.

So now if you install and run stuff, it's limited to this virtual environment and it's not going to conflict with any other project.

You can deactivate the virtual environment at any time by typing:

deactivate

And activate a different one if you wish.

You can also have several terminals, each with a different activated virtual environment. They won't see each other.

What is pip and why to use it?

While Python has a very rich standard library, at some point you will want to use something made by someone else.

The standard way to do this is to use "pip".

pip is a package manager, you give it a name, it looks if something with this name exists on https://pypi.org, download it, and install it.

E.G: If you want to display a histogram, coding it yourself would take a long time. But you can install "matplotlib" with pip, which already contains everything you need to do it.

pip is provided with Python, assuming again you followed our advice on how to install Python.

How to install something with pip

First, you should have activated a virtual environment.

Never use pip without making sure you have a virtual environment activated.

Also, you must have access to the internet, so make sure you are connected to it.

Then in a terminal, use this command:

python -m pip install NAME

With "NAME" being the thing you wish to install.

E.G, to install "pendulum", the excellent date and time handling library:

python -m pip install pendulum

This will download and install pendulum in the folder of the virtual environment. If you run Python with this virtual environment activated, you will be able to import pendulum. However, outside of this particular virtual environment, pendulum is not available.

The reverse of this command is:

python -m pip uninstall NAME

You can use the same commands on windows, Linux and Mac because you are in a virtual environment. No need for choosing a particular version of Python, since you are using the one from the virtual environment, so no need for "python3.X" or "py -3.X" when you are inside a virtual environment. That's one of the benefits.

At this stage, you may wonder about the “python -m” thingy. Why not call “pip” directly, isn’t the virtual environment making sure it’s the right one? It should, but sometimes it fails to do so. “python -m pip” has more chances to succeed than just “pip”, even in virtual environment. Yes, it’s annoying.

I said “virtual environment” a lot, didn't I?

In fact, I have a VSCode snippet just to type "virtual environment" quickly for this article.

Recreating a virtual environment

You cannot move a virtual environment, it will stop working. Also, if you change the Python used in the virtual environment, such as when uninstalling it, it will stop working.

For all these reasons, it can be handy to be able to recreate exactly the same virtual environment.

To do this, run in a terminal (with a virtual environment activated):

python -m pip freeze > requirements.txt

This will create a new file named "requirements.txt" listing all installed packages in your virtual environment.

If you ever need so, you can create a new virtual environment and reinstall all the packages in it by doing this:

python -m pip install -r requirements.txt

You can send the "requirements.txt" file to a colleague so they can install the same virtual environment as you, or use "requirements.txt" to relocate your virtual environment somewhere else by deleting and recreating it.

"requirements.txt" files are not perfect. They don't separate dependencies for production and development. They always include the whole dependency graph, with pinned version, and for a specific platform.

But you can go very far with just "venv", "pip" and a "requirements.txt" file.

What not to do:

• Don't use pip outside of a virtual environment. I repeat myself, but this point is important. Many tutorials will invite you to do so. E.G: to install pipx or poetry first. It will add to the list of potential problems you are exposing yourself to. Read "Why not tell people to "simply" use pyenv, poetry or anaconda" for a detailed explanation.

• Hence, don't use pipx or poetry.

• And neither should you therefore use sudo, since it will install things outside of the virtual environment.

• Nor should you use "--user", since it's made to install things outside of the virtual environment.

• Don't mix pip, venv and Anaconda. Avoid Anaconda if you can. If you have to use Anaconda, ignore this entire tutorial, and don't use pip and venv. Limit yourself to Anaconda tools.

• Don't manually go inside a virtual environment or create files inside the virtual environment. It's not necessary.

• Don't move a virtual environment. Create a "requirements.txt" file, delete the virtual environment and create a new one.

• Don't rename a directory containing a virtual environment. Or if you do, prepare yourself to create a "requirements.txt" file, delete the virtual environment and create a new one.

Tips and tricks

• Create one big virtual environment for all small scripts. If you make a lot of them, you may be tempted to install everything at the system level for convenience. After all, it's a bore to create and activate a virtual environment each time you want to write a five liner. A good balance is one single virtual environment you use for all things quick and dirty. It's not perfect isolation, but it's better than the alternative.

• Be liberal with virtual environments. Use and abuse them, create, delete… They are cheap.

• Create several virtual environments per versions of python if your project needs to support several versions. You may need several requirements.txt files as well, one for each env.

• You don't need to activate a virtual environment to use it! This is just a convenience feature. If you use the commands situated in the Scripts (on windows) or bin (on Unix) folders in the virtual environment directly, they are specially crafted to only see the virtual environment even without activation. In fact, activating just make those commands the default ones by manipulating a bunch of environment variables.

• If you get SSL errors (common if you are in a hotel or a company network) use the "--trusted-host pypi.org --trusted-host files.pythonhosted.org" options with pip to work around the problem. E.G: "python -m pip install pendulum --trusted-host pypi.org --trusted-host files.pythonhosted.org".

• If you are behind a corporate proxy that requires authentication (common if you are in a company network), you can use the "--proxy" option with pip to give the proxy address and your credentials. E.G.: "python -m pip install pendulum --proxy http://your_username:yourpassword@proxy_address" It also works with the https_proxy environment variables, but most people don't know how to use them in my experience. I will likely write a tutorial on that.

• If you want to download a package without installing it, you can use "python -m pip download NAME". It will download the package and all its dependencies in the current directory (the files, called, wheels, have a .whl extension). You can then install them offline by doing "python -m pip install" on the wheels.

Configuring your code editor to use your virtual environment

It's worth an article on its own, but the gist of it is to find the particular settings on your editor where you can give the full path to the python inside the virtual environment (the one in Scripts or bin).

Remember, virtual environment activation is just a convenience feature, but one can perfectly not activate a virtual environment and just use the python that is in the Scripts or bin folder.

So if you tell your editor "for this project, use the this particular python interpreter" by giving it the path to the python in the virtual environment, it will usually work out of the box.

However, the way to do that is different for each editor.