All your Python project in one file with shiv

Now guess which typo I always do with this tool

Jan 02, 2025

Summary

(This article assumes you are comfy with pip + venv. If you are not, we have articles for you)

With shiv, you can get an executable zip of all your Python code and its dependencies (called a zipapp) that will run on any server with Python that uses the same OS.

No need to create a venv and pip install on the target machine.

To make a zipapp on your local machine, create a venv, and pip install all your project dependencies, as well as shiv itself.

Then you can do:

cp <SCRIPT_ENTRY_POINT> <PATH_TO_PACKAGES>
shiv --site-packages <PATH_TO_PACKAGES> -e <MODULE_ENTRY_POINT> -o <ZIPAPP_NAME>

To get your executable archive.

E.G, if I have a hello_word.py with:

import requests

def main():
    print(len(requests.get("https://www.bitecode.dev").content))

if __name__ == "__main__":
    main()

And I do:

cp hello_word.py .venv/lib/python3.11/site-packages/
shiv --site-packages .venv/lib/python3.11/site-packages/ -e hello_word.main -o hello_word.pyz

I'll get a new file called hello_word.pyz containing hello_word.py and requests.

On the target machine, you can run python hello_word.pyz, and this will extract the whole content of the archive in a ~/.shiv sub-directory then start your program transparently.

The process, however, comes with a few gotchas, particularly regarding non-Python files or WSGI apps.

I want it all, and I want it now

Once "It works on my machine™", what do I do with my Python program?

If it's a lib, share it on Pypi. If it's a program for an end user, create an executable.

And if it's a web service, deploy it on a server.

Now there is no universal story for the latter, but you need to know how to replicate your setup on the distant machine, particularly the dependencies in your virtualenv.

Most projects have some way to push the code on the machine and then trigger the installation of the new venv.

For example, a revolutionary bitecode-home-page-size micro-service project could be a hello_word.py file that looks like this:

import requests
print(len(requests.get("https://www.bitecode.dev").content))

But it needs requests, how do you get that on the server?

One way is of course to pip freeze then pip install -r, or whatever is the equivalent of your packaging tool of choice.

Another way is to gulp the uv hype and do:

# /// script
# requires-python = ">=3.11"
# dependencies = [
#     "requests",
# ]
# ///
import requests
print(len(requests.get("https://www.bitecode.dev").content))

Then on the server, install uv so you can uv run.

But what if the machine doesn't have access to Pypi? Or what if you want to create a single artifact, push that on the server, and make it just work "as-is", like a *.war file in Java?

There is a standard format for that in Python, called the zipapp, which is essentially an executable zip file containing all your code and its dependencies.

Several tools, such as the stdlib zipapp module or pex, can build one. But the first program is limited in features, while the second one doesn't work on Windows.

Today, we are going to look at shiv, which is what I currently use for this kind of task.

What is shiv good and bad at?

Shiv does only one thing: take your code and turn it into a zipapp.

But it comes with two tricks:

The first run, the zipapp will decompress into a cache directory. This means the first run is slow, but the next ones are faster. It also allows you to use code that is not zip-safe, like a Django project that wants to read static files in its own source using __file__.
You can give it a script to execute before your code runs but after the decompression, if you need some setup to take place.

Unfortunately, shiv's doc is not great, and there are several ways to use it, none being particularly obvious.

It also requires you to understand well the process because you will likely want to tweak it, so it's not transparent magic that you can run and forget.

More than that, if your dependencies contain compiled extensions, the zipapp will only work on the same type of OS + architecture than the one you created the zipapp on. You can make a zipapp on windows and run it on linux with a bit of a hack, but I wouldn't advise you to do it (use WSL instead).

Finally, zipapps are not standalone executables like you could produce with pyinstaller or nuitka. You still need Python installed on the target machine.

Still, it's a nice tool once it's set up correctly, as it lets you to have one big file you just drop on the server.

Hello Shiv

Let's create a venv, pip install shiv requests in it, and then let's go back to our original hello_word.py script:

import requests

if __name__ == "__main__":
    print(len(requests.get("https://www.bitecode.dev").content))

There are several ways to use shiv, with a console script, or with a Python module. I find using the Python module easier and more robust. For this, we can turn our script into a well-mannered entry point:

import requests

def main():
    print(len(requests.get("https://www.bitecode.dev").content))

if __name__ == "__main__":
    main()

This way, our script can be imported without side effects, and we can point to the function main() using the import path hello_word.main to say “this is how our program starts”.

You can put any code here: an argparse declaration, the code of Django's manage.py, your entire program, or just a few imports and one call. It doesn't matter, the important thing is that it's the main entry point.

Then we create the zipapp with:

cp <MODULE> <PATH_TO_PACKAGES> # don't forget this or your code won't be in the zip
shiv --site-packages <PATH_TO_PACKAGES> -e <MODULE_ENTRY_POINT> -o <ZIPAPP_NAME>

E.G:

cp hello_word.py .venv/lib/python3.11/site-packages/
shiv --site-packages .venv/lib/python3.11/site-packages/ -e hello_word.main -o hello_word.pyz

In my case, I directly passed .venv/lib/python3.11/site-packages/, which is the directory where all the packages are installed by pip in my venv. It's different depending on the tooling and the platform. Note that you don't even have to use a venv, shiv only wants a directory of stuff to copy, you can pass any directory that contains code.

In fact, I usually create a separate dir and copy both my entry point and my venv packages in there, it's cleaner.

hello_word.main tells shiv it can run the entry point by doing import hello_word; main().

The goal of all this, the output, will be a zip file named hello_word.pyz.

You can then run the zipapp with Python:

❯ python hello_word.pyz
62236

Our revolutionary script has successfully run. You can even see that shiv uncompressed it transparently:

❯ ls ~/.shiv
hello_word.pyz_2ecc0aa8cd46b0ed00ffc897ca8e009664adb36582f63c85274c2f0f2fceb64d

How to add static files

All files in the --site-packages directory will be copied in the zip, Python or not, so you can put them there before building the zipapp. Now, if you don't want to pollute your venv with this, create a temp dir with all the things you need: packages, static files, whatever.

It's a good practice anyway.

However, how do you read them? After all, they will be on a path like:

~/.shiv/hello_word.pyz_2ecc0aa8cd46b0ed00ffc897ca8e009664adb36582f63c85274c2f0f2fceb64d/goeste.jpg

If they are inside a dependency package, it's not a problem. However, it's my experience that most users use shiv with projects that are not packages, such as web projects, for which they have tons of static files they want to use.

For this, I recommend simply copying the files you need using a preamble script, which I will explain below.

Setting up your project after deployment

When you execute your zipapp, the code is actually decompressed and then run from a directory inside ~/.shiv (or anything you configured).

But this can be a problem if you have things that are path-dependent, like:

Sqlite DBs.
Static assets served from apache/nginx/caddy.
CSV/JSON/YAML config files your code wants to read assuming it could do pathlib.Path(__file__).open().
Commands to migrate your projects in prod, like manage.py collectstatic/migrate/makemessages in Django.

Since shiv can't find a generic solution to all those problems, it instead provides you with hooks to deal with this.

The simplest one is --build-id TEXT that you can pass when you build your zipapp. You can set the ID to whatever you want, BUT IT MUST BE UNIQUE PER BUILD.

Good candidates are timestamps, git commit hashes, or even a release version:

❯ shiv --site-packages .venv/lib/python3.11/site-packages/ -e hello_word.main -o hello_word.pyz --build-id 3.1.2
❯ python hello_word.pyz
62236
❯ ls ~/.shiv/
hello_word.pyz_2ecc0aa8cd46b0ed00ffc897ca8e009664adb36582f63c85274c2f0f2fceb64d
hello_word.pyz_3.1.2

This way the directory becomes predictable, and you can configure the rest of your stack based on this.

If you prefer for automation, dynamism, or simply need something more custom, you can use a preamble script, which is a separate Python script shiv will call after decompression is confirmed, but before your project code runs.

It's a special script because shiv magically injects 3 variables in there:

archive: the path to the zip of zipapp.
site_packages: the path to the decompressed directory.
env: a dict of metadata about the zipapp.

Here is an example of preamble.py script that prints those information:

import pprint

print('preamble starts!')

print(archive)

print(site_packages)

pprint.pprint(vars(env))

print('preamble ends!')

Note that I don't need to import those variables, they are automatically present in the preamble script.

Then I build my zipapp with it:

❯ shiv --site-packages .venv/lib/python3.11/site-packages/ -e hello_word.main -o hello_word.pyz --preamble preamble.py
❯ python hello_word.pyz
preamble starts!
./hello_word.pyz
/home/user/.shiv/hello_word.pyz_cee74e724d5354fa7c04798ae7424b041b944d91d5e3f1f614d49bdb87947263
{'_compile_pyc': False,
 '_entry_point': 'hello_word.main',
 '_extend_pythonpath': False,
 '_prepend_pythonpath': None,
 '_root': None,
 '_script': None,
 'always_write_cache': False,
 'build_id': '3.1.2',
 'built_at': '2025-01-01 17:41:57',
 'hashes': {},
 'no_modify': False,
 'preamble': 'preamble.py',
 'reproducible': False,
 'shiv_version': '1.0.8'}
preamble ends!
62236

You can put whatever you want in this script, like moving or copying files around, running db migrations, calling systemd commands, etc.

I tend to put things like cleaning up old decompressed directories, moving static files into a directory that nginx can serve, or calls to chmod.

The preamble will run every time, so make sure you make it idempotent.

Tips

uv run works with zipapp.
You can change the place where shiv will decompress the files using --root. Shiv will make this information available through the env var SHIV_ROOT to the zipapp code.
You don't really need the preamble, it's more of a convenience thing. You can replicate it roughly by doing this in your entry point:

import os
import sys
import zipfile
from pathlib import Path

from _bootstrap.environment import Environment

SHIV_ROOT = Path(os.environ.get("SHIV_ROOT", "~/.shiv")).expanduser().absolute()
ZIPAPP_NAME = sys.argv[0]

with zipfile.ZipFile(ZIPAPP_NAME) as archive:
    env = Environment.from_json(archive.read("environment.json").decode())
    extraction_dir = SHIV_ROOT / f"{ZIPAPP_NAME}_{env.build_id}"

Passing --python TEXT can set a shebang python version to execute the zipapp like a script on Unix.
You can put ANYTHING in the entry point. You can call gunicorn from there. You can call pydoit from there. You can import your whole project and run different parts of it depending of the args you pass to the zipapp.
If you use a WSGI or ASGI Web framework like Django, Flask, or FastAPI, you will actually have to do this!

E.G, a minimalist hello_django.py production entry point using waitress:

import os
import sys

import django
from django.core.management import execute_from_command_line
from waitress import serve


def main():
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings")
    django.setup()

    if len(sys.argv) > 1 and sys.argv[1] == "runserver":
        from project.wsgi import application

        # you probably want to put this in env vars (and also DEBUG in settings)
        serve(application, host="127.0.0.1", port=8000)
    else:
        execute_from_command_line(sys.argv)


if __name__ == "__main__":
    main()

Now if I do:

> source .venv/bin/activate
> pip install django waitress # or -r requirements.txt
> cp -r project .venv/lib/python3.11/site-packages/ # add the django project to the zipapp
> cp -r hello_django.py .venv/lib/python3.11/site-packages/
> shiv --site-packages .venv/lib/python3.11/site-packages/ -e hello_django.main -o hello_django.pyz

I can run the waitress server in production this way:

> python hello_django.pyz runserver

Or do things like run migrations this way:

> python hello_django.pyz migrate

This is definitely not straightforward and doesn't even deal with other issues like the static files

Therefore, shiv, while allowing us to achieve our goal, is still an expert tool at this stage where you need to understand the underlying layer to make sure it works well.

Bite code!

Discussion about this post