What's up Python? 3.13 is out, t-strings look awesome, dep groups come in handy...
October, 2024
Summary
Python 3.13 is out.
The Template Strings update looks awesome.
New proposal for external wheel hosting to host huge libs elsewhere.
We have an extra extra: dependency groups in pyproject.toml.
Mypy 1.12 now supports Generic and Python 3.13.
Python 3.13 fresh out of the oven
I know you know, and you probably already read our article on it at that point.
But what monthly recap of Python activity would it be if I didn't at least mention a new major release?
I'm not going to dwell on it, so in short:
It's a superb release with awesome new features like a refreshed REPL, better error messages, improved static typing and tons of goodies.
It's likely to be a dangerous release because it comes with the experimental JIT compiler and alternative no-GIL build, but interesting times ahead.
Yes you should play with it, test it and have fun. No, you should not upgrade until next year.
The Template Strings proposal got a serious update
Strangely, f-string were a very controversial project when they were suggested, although they are today almost universally considered a fantastic feature.
But there is more controversial in that department: template strings. They started as i-string to make i18n easier, then debated to death to turn into tagged strings where any arbitrary prefix for any string was allowed (!), to finally land on what they are today.
Their current form is way more reasonable (Guido was part of the PEP, so maybe he helped with that), and I believe they now have all it takes to follow the steps of f-string and become a beloved feature.
So what are they?
If an f-string like this:
name = "world"
f"Hello, {name}"Is syntactic sugar for:
name = "world"
"Hello, {name}".format(name=name)Template strings, well, let's call them t-strings because we all know how it's going to end anyway, like this:
name = "world"
t"Hello, {name}"Is sugar for this:
from templatelib import Template, Interpolation # python 3.14?
name = "world"
Template(args=["Hello", Interpolation(value="world", exprs="name")])Now you might wonder, why the hell would you want that?
Well, there are plenty of situations where you need formatted strings, but with processing separated from declaration:
For deferred evaluation like for translation or logging.
For escaping like to avoid SQL/shell/JS injection.
Right now, every API that needs to perform lazy string processing provides its own way of doing it.
gettext does this for pluralization:
message = cat.ngettext(
'There is %(num)d file in this directory',
'There are %(num)d files in this directory',
n
) % {'num': n}sqlite3 requires this for escaping user input:
cursor.execute("insert into user(username, password) values(?, ?)", (username, password))logging suggests this to avoid calling functions at ignored log levels:
if logger.isEnabledFor(logging.DEBUG):
logger.debug('Message with %s, %s', expensive_func1(),
expensive_func2())You cannot use f-strings for those, as they are evaluated immediately, but it would be super nice to have one standard and clean way to do all this. With t-strings, we could do:
message = cat.ngettext(
t'There is {n} file in this directory',
t'There are {n} files in this directory',
)cursor.execute(t"insert into user(username, password) values({username}, {password})")logger.debug(t'Message with {expensive_func1()}, {expensive_func2()}')No need to think about it too much, and the chance of introducing a problem due to a mistake or indolence is much lower. It's clean, practical, elegant, and encourages best practices.
I like it very much.
Now it does introduce a third way of doing lazy processing (after generators and coroutines), not to mention lazy imports have also been in discussion for like, forever.
So at this stage, I would have preferred to have a lazy keyword to carry a general mechanism for the whole language. But that ship has sailed, and nobody wants to touch such bonker change with a ten-foot pole anyway.
External Wheel Hosting
The very first day of this month, PEP 759 came to be, proposing a mechanism by which projects hosted on pypi.org can safely host wheel artifacts on external sites other than PyPI.
This is NOT about third party's Pypi instances or private index. It's about having a custom download URL on pypi so that projects can host libs that are bigger than Pypi's default upload size limit (100Mib per lib, 10Gib per project). And also save some bandwith for Pypi which amounts to terabytes of data... every day.
If you wonder "wait, pytorch is 800 Mib and is on Pypi, how can it be?” It's because it's the default limit, but there is a manual process to make your case and request a higher limit.
Right now, to bypass the limit, projects that are not blessed with an exception use old-school source distributions and then ask the build backend to perform the download elsewhere during installation. It works, but there is no standardized way to do it, so it's kinda of a free-for-all, with all the UX and security implications.
The new PEP is basically a better version of this shim and introduces a new format, the .rim file (for Remote Installable Metadata), which is basically a .whl file with:
All Python code removed.
An
EXTERNAL-HOSTING.jsonfile telling you where to find the real code.All the other files being the same, the format is the same, and the download URL is a regular warehouse-shaped one using the so-called Simple API.
More importantly, Pypi's API will expose the metadata and download URL from .rim files in the same way it does for wheels. So if you don't query the files themselves directly (most clients don't), you won't even know there is a .rim file, and will just proceed as usual from the data returned by the API, transparently.
Some clients with special download strategies (like uv) might need a little adaptation, but given the format is 99% the same as wheels, it should be minimal.
As for end users, it will be automatic. So you don't have to do anything. And of course, hashes are used to check the integrity of the download.
The PEP is not accepted yet, but the discussion thread has been quite positive so far.
Dependency Groups in pyproject.toml
PEP 735 is one year old and has just been accepted this month. Good things take time.
It standardizes how to declare optional dependencies in pyproject.toml for things you don't want the end users to see. Like:
dev tools: linter, debugger, shells, formatters, etc.
deps for private projects: web projects, internal code bases, and so on, for which you want to install deps without having a build system in place.
It's an alternative to extras, which are already available to declare public optional dependencies such as alternative backends or accelerators.
Typically, with "extra", an end user install it with pip install package[extra], like pip install fastapi[orjson] if you want to have fast JSON support.
For dependency groups, however, there is no such syntax, and end users don't even know they exist (it's not exposed in pypi's available metadata). But somebody with access to the source code will be able to install them with pip install --dependency-groups=name_of_the_group. E.G: pip install --dependency-groups=dev to install, say, black and mypy.
While you can create as many arbitrary groups as you like, for anything you wish, the main use case will be obviously to let you separate development dependencies from production ones, like or poetry does with [tool.poetry.group.dev.dependencies] or pip-tools with [project.optional-dependencies].
You probably don't want ruff or ipython to be installed in production, but it's nice to be able to declare it as the dev stack officially and automate things. I suspect the lack of such an option was one of the reasons people eventually migrated out of pip, tired of manually maintaining a dev-requirements.txt. Probably more so than any correctness of deployment argument.
Having finally an official format for this means all tools can align, and pip can move on. uv famously implemented it almost immedialty, replacing its old "tool.uv.dev-dependencies" system.
Mypy 1.12 supports the new generics syntax
Not all Mypy releases are equal, but I'm very glad for 1.12, because it finally supports the new generics syntax introduced in Python 3.12, at the very moment I'm migrating to it:
def f[T](x: T) -> T: ...I get it, this is not the most familiar syntax in Python right now. Python is beloved for being a pseudo-code language, so professional-oriented syntax additions are popular only among a small subset of the user base. But to those who need it, it's really great.
Let me explain. Sometimes you want to declare a type, but you don't know what it is in advance, so you want a sort of "variable" type.
E.G: You know your function selects one stuff from a list and returns it. You know the type of the stuff it returns is the same as the types of the elements in the list. But you don't know what is the type of the elements of the list in advance.
Declaring that is kinda tedious with the old syntax:
from typing import TypeVar
TypeOfSelectedStuff = TypeVar('TypeOfSelectedStuff')
def choose_stuff(items: list[TypeOfSelectedStuff]) -> TypeOfSelectedStuff:
...The new syntax lets you do:
def choose_stuff[T](items: list[T]) -> T: # T is TypeOfSelectedStuff
...Now people (and type checkers) read this and know that whatever is shoved into items, what goes out of this function is of the same type.
The new Mypy also brings partial support for Python 3.13, with the excellent typing.ReadOnly and typing.TypeIs but unfortunately not @deprecated yet.
I think ReadOnly and deprecated are self-explanatory, but you might be wondering what this TypeIs thingy is about. It's basically a new version of TypeGuard, but one that works (because TypeGuard was pretty much useless except in very specific cases).
I know, I know, if you never used any of this, you might still have zero idea of what I'm rambling about. Let's dive in.
TypeIs is an advanced type annotation, so it's only useful if:
You want your code typed.
You really, really want your code typed, God damn it.
You have reached the limit of typing in Python, and you want to keep going because those limits are for the weaks.
Basically, it lets you say "look Mypy, look me in the eye, if this function returns True, this object is of type X, you hear me?".
Example:
from typing import TypeIs
class Animal: ...
class Dog(Animal):
def pet(self): ...
def bark(self): ...
class Cat(Animal):
def pet(self) -> ...
def mew(self) -> ...
# this function guarantee the subtype of Animal
def is_pet(animal: Animal) -> TypeIs[Dog | Cat]:
return hasattr(animal, "bark") or hasattr(animal, "mew")
def feed_animal(animal: Animal) -> str:
if is_pet(animal):
return animal.pet() # mypy doesn't complain here
feed(animal)You might rightly say "don't architecture your code like that", but I only did so to have a simple example. There are very valid reasons to deduct a type according to some property of an object, especially in a duck typing language, but they are usually not easy to explain in a short paragraph.
Will you need that? Most people don't.
But I needed it badly last year, and I was sad.