Python 3.12: what didn't make the headlines

It's less interesting, but it's a niche I can fill

Oct 03, 2023

Performances are underwhelming
Pathlib has been improved
We get a better debugging experience
Slice objects are now hashable
An unexpected math.sumprod() appears
New cool command-line interfaces
Heavy deprecation round

It's release time!

Python 3.12 has been released, and there are thousands of articles out there to repeat what the released notes already say. Some are pretty good, though, so if you haven't read RealPython's intro on the new typing features yet, add it to your list.

And yes, I'm a fan of the better error messages (they are for me the best feature of the release), I applaud the formalization of f-string parsing, and hurrah for itertools.batched, but we have been discussing that for ages on Twitter.

What is everyone not talking about?

The performance let down

When you get an amazing product like Python for free, it's not a good look to send bad vibes to the people that have been sweating hard to bring this gift to the world.

So it's understandable people don't really want to address the elephant in the room: the performances of this new version don't live up to the hype.

By this, I mean the fact that MS is paying a team with Guido to speed up the language, and that we were all quite giddy when they announced that the target was a 50% gain at each release for the next 5 ones. We saw that coming, but still.

The release notes talk about a 5% gain in general, an underwhelming number, but I wanted to check by myself. So I grabbed 3.11 and 3.12, installed the headers, grabbed pyperformance and ran it on my laptop with:

pyperformance run --python=/usr/bin/python3.11 -o py311.json
pyperformance run --python=/usr/bin/python3.12 -o py312.json
pyperformance compare py311.json py312.json

You know what they say about benchmarks... Plus, despite the fact I killed everything I could and let the machine alone for the solid hour it ran, getting some serious numbers is very difficult.

That being said, here is the report:

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters


	py311.json
	==========

	Performance version: 1.0.9
	Report on Linux-5.15.0-84-generic-x86_64-with-glibc2.31
	Number of logical CPUs: 8
	Start date: 2023-10-03 15:46:40.037527
	End date: 2023-10-03 16:30:51.189955

	py312.json
	==========

	Performance version: 1.0.9
	Report on Linux-5.15.0-84-generic-x86_64-with-glibc2.31
	Number of logical CPUs: 8
	Start date: 2023-10-03 16:51:38.432027
	End date: 2023-10-03 17:38:09.827299

	### 2to3 ###
	Mean +- std dev: 310 ms +- 13 ms -> 285 ms +- 13 ms: 1.09x faster
	Significant (t=10.34)

	### async_generators ###
	Mean +- std dev: 304 ms +- 53 ms -> 371 ms +- 6 ms: 1.22x slower
	Significant (t=-9.81)

	### async_tree_cpu_io_mixed ###
	Mean +- std dev: 624 ms +- 47 ms -> 676 ms +- 20 ms: 1.08x slower
	Significant (t=-7.82)

	### async_tree_cpu_io_mixed_tg ###
	Mean +- std dev: 589 ms +- 26 ms -> 679 ms +- 36 ms: 1.15x slower
	Significant (t=-15.78)

	### async_tree_io ###
	Mean +- std dev: 934 ms +- 9 ms -> 992 ms +- 29 ms: 1.06x slower
	Significant (t=-15.00)

	### async_tree_io_tg ###
	Mean +- std dev: 905 ms +- 5 ms -> 1014 ms +- 19 ms: 1.12x slower
	Significant (t=-41.81)

	### async_tree_memoization ###
	Mean +- std dev: 493 ms +- 15 ms -> 521 ms +- 33 ms: 1.06x slower
	Significant (t=-6.03)

	### async_tree_memoization_tg ###
	Mean +- std dev: 465 ms +- 12 ms -> 525 ms +- 21 ms: 1.13x slower
	Significant (t=-19.37)

	### async_tree_none ###
	Mean +- std dev: 433 ms +- 75 ms -> 412 ms +- 8 ms: 1.05x faster
	Significant (t=2.18)

	### async_tree_none_tg ###
	Mean +- std dev: 364 ms +- 4 ms -> 390 ms +- 7 ms: 1.07x slower
	Significant (t=-25.80)

	### asyncio_tcp ###
	Mean +- std dev: 611 ms +- 5 ms -> 348 ms +- 3 ms: 1.76x faster
	Significant (t=353.16)

	### asyncio_tcp_ssl ###
	Mean +- std dev: 2.11 sec +- 0.01 sec -> 1.17 sec +- 0.02 sec: 1.80x faster
	Significant (t=303.48)

	### bench_mp_pool ###
	Mean +- std dev: 5.74 ms +- 1.20 ms -> 14.17 ms +- 6.45 ms: 2.47x slower
	Significant (t=-9.94)

	### bench_thread_pool ###
	Mean +- std dev: 704 us +- 76 us -> 772 us +- 77 us: 1.10x slower
	Significant (t=-4.83)

	### chameleon ###
	Mean +- std dev: 5.14 ms +- 0.08 ms -> 6.23 ms +- 0.13 ms: 1.21x slower
	Significant (t=-55.85)

	### chaos ###
	Mean +- std dev: 50.4 ms +- 0.7 ms -> 55.9 ms +- 0.8 ms: 1.11x slower
	Significant (t=-40.36)

	### comprehensions ###
	Mean +- std dev: 17.6 us +- 0.3 us -> 19.0 us +- 0.2 us: 1.08x slower
	Significant (t=-29.67)

	### coroutines ###
	Mean +- std dev: 18.2 ms +- 0.3 ms -> 19.7 ms +- 0.3 ms: 1.08x slower
	Significant (t=-27.58)

	### coverage ###
	Mean +- std dev: 317 ms +- 5 ms -> 634 ms +- 14 ms: 2.00x slower
	Significant (t=-168.43)

	### create_gc_cycles ###
	Mean +- std dev: 905 us +- 6 us -> 972 us +- 7 us: 1.07x slower
	Significant (t=-52.69)

	### crypto_pyaes ###
	Mean +- std dev: 58.6 ms +- 1.0 ms -> 65.6 ms +- 1.5 ms: 1.12x slower
	Significant (t=-29.99)

	### dask ###
	Mean +- std dev: 375 ms +- 14 ms -> 461 ms +- 27 ms: 1.23x slower
	Significant (t=-22.00)

	### deepcopy ###
	Mean +- std dev: 271 us +- 3 us -> 326 us +- 9 us: 1.20x slower
	Significant (t=-44.68)

	### deepcopy_memo ###
	Mean +- std dev: 27.6 us +- 0.5 us -> 31.7 us +- 0.4 us: 1.15x slower
	Significant (t=-48.08)

	### deepcopy_reduce ###
	Mean +- std dev: 2.46 us +- 0.03 us -> 3.10 us +- 0.05 us: 1.26x slower
	Significant (t=-88.00)

	### deltablue ###
	Mean +- std dev: 2.82 ms +- 0.03 ms -> 2.71 ms +- 0.05 ms: 1.04x faster
	Significant (t=14.81)

	### django_template ###
	Mean +- std dev: 26.1 ms +- 0.4 ms -> 33.7 ms +- 0.8 ms: 1.29x slower
	Significant (t=-64.15)

	### docutils ###
	Mean +- std dev: 1.93 sec +- 0.02 sec -> 2.24 sec +- 0.05 sec: 1.16x slower
	Significant (t=-42.17)

	### dulwich_log ###
	Mean +- std dev: 48.7 ms +- 0.7 ms -> 54.4 ms +- 1.1 ms: 1.12x slower
	Significant (t=-33.24)

	### fannkuch ###
	Mean +- std dev: 287 ms +- 5 ms -> 315 ms +- 4 ms: 1.10x slower
	Significant (t=-36.51)

	### float ###
	Mean +- std dev: 55.9 ms +- 1.0 ms -> 68.9 ms +- 1.3 ms: 1.23x slower
	Significant (t=-63.74)

	### gc_traversal ###
	Mean +- std dev: 2.60 ms +- 0.02 ms -> 2.83 ms +- 0.03 ms: 1.09x slower
	Significant (t=-52.25)

	### generators ###
	Mean +- std dev: 53.2 ms +- 0.7 ms -> 26.3 ms +- 0.4 ms: 2.02x faster
	Significant (t=250.74)

	### genshi_text ###
	Mean +- std dev: 17.1 ms +- 0.3 ms -> 20.8 ms +- 0.5 ms: 1.22x slower
	Significant (t=-50.48)

	### genshi_xml ###
	Mean +- std dev: 40.7 ms +- 0.7 ms -> 45.6 ms +- 0.6 ms: 1.12x slower
	Significant (t=-40.56)

	### go ###
	Mean +- std dev: 107 ms +- 2 ms -> 112 ms +- 2 ms: 1.04x slower
	Significant (t=-16.10)

	### hexiom ###
	Mean +- std dev: 4.79 ms +- 0.06 ms -> 4.80 ms +- 0.08 ms: 1.00x slower
	Not significant

	### html5lib ###
	Mean +- std dev: 50.7 ms +- 2.0 ms -> 53.2 ms +- 2.5 ms: 1.05x slower
	Significant (t=-6.08)

	### json_dumps ###
	Mean +- std dev: 9.77 ms +- 0.12 ms -> 9.29 ms +- 0.23 ms: 1.05x faster
	Significant (t=14.27)

	### json_loads ###
	Mean +- std dev: 16.6 us +- 0.3 us -> 22.1 us +- 3.0 us: 1.34x slower
	Significant (t=-14.48)

	### logging_format ###
	Mean +- std dev: 5.17 us +- 0.10 us -> 6.25 us +- 0.14 us: 1.21x slower
	Significant (t=-48.23)

	### logging_silent ###
	Mean +- std dev: 76.1 ns +- 1.5 ns -> 89.3 ns +- 2.3 ns: 1.17x slower
	Significant (t=-36.73)

	### logging_simple ###
	Mean +- std dev: 4.76 us +- 0.06 us -> 5.65 us +- 0.13 us: 1.19x slower
	Significant (t=-47.77)

	### mako ###
	Mean +- std dev: 7.24 ms +- 0.15 ms -> 9.37 ms +- 0.24 ms: 1.29x slower
	Significant (t=-58.60)

	### mdp ###
	Mean +- std dev: 1.97 sec +- 0.02 sec -> 2.44 sec +- 0.06 sec: 1.24x slower
	Significant (t=-56.93)

	### meteor_contest ###
	Mean +- std dev: 81.6 ms +- 0.8 ms -> 89.9 ms +- 2.1 ms: 1.10x slower
	Significant (t=-28.63)

	### nbody ###
	Mean +- std dev: 68.0 ms +- 1.0 ms -> 71.6 ms +- 1.7 ms: 1.05x slower
	Significant (t=-14.31)

	### nqueens ###
	Mean +- std dev: 70.7 ms +- 0.9 ms -> 82.1 ms +- 3.6 ms: 1.16x slower
	Significant (t=-23.67)

	### pathlib ###
	Mean +- std dev: 13.7 ms +- 0.2 ms -> 15.7 ms +- 0.3 ms: 1.14x slower
	Significant (t=-50.10)

	### pickle ###
	Mean +- std dev: 6.71 us +- 0.11 us -> 8.28 us +- 0.09 us: 1.23x slower
	Significant (t=-85.78)

	### pickle_dict ###
	Mean +- std dev: 21.6 us +- 0.4 us -> 21.6 us +- 0.3 us: 1.00x faster
	Not significant

	### pickle_list ###
	Mean +- std dev: 2.78 us +- 0.04 us -> 3.23 us +- 0.07 us: 1.16x slower
	Significant (t=-41.39)

	### pickle_pure_python ###
	Mean +- std dev: 236 us +- 12 us -> 269 us +- 3 us: 1.14x slower
	Significant (t=-19.91)

	### pidigits ###
	Mean +- std dev: 175 ms +- 3 ms -> 179 ms +- 3 ms: 1.02x slower
	Significant (t=-7.66)

	### pprint_pformat ###
	Mean +- std dev: 1.18 sec +- 0.01 sec -> 1.52 sec +- 0.04 sec: 1.29x slower
	Significant (t=-58.58)

	### pprint_safe_repr ###
	Mean +- std dev: 573 ms +- 7 ms -> 753 ms +- 14 ms: 1.31x slower
	Significant (t=-91.17)

	### pyflate ###
	Mean +- std dev: 325 ms +- 4 ms -> 358 ms +- 5 ms: 1.10x slower
	Significant (t=-41.65)

	### python_startup ###
	Mean +- std dev: 6.72 ms +- 0.26 ms -> 8.15 ms +- 0.26 ms: 1.21x slower
	Significant (t=-55.41)

	### python_startup_no_site ###
	Mean +- std dev: 4.86 ms +- 0.15 ms -> 6.44 ms +- 0.31 ms: 1.32x slower
	Significant (t=-65.44)

	### raytrace ###
	Mean +- std dev: 225 ms +- 3 ms -> 274 ms +- 3 ms: 1.22x slower
	Significant (t=-84.20)

	### regex_compile ###
	Mean +- std dev: 108 ms +- 2 ms -> 123 ms +- 2 ms: 1.14x slower
	Significant (t=-40.31)

	### regex_dna ###
	Mean +- std dev: 141 ms +- 1 ms -> 144 ms +- 1 ms: 1.03x slower
	Significant (t=-15.26)

	### regex_effbot ###
	Mean +- std dev: 2.17 ms +- 0.03 ms -> 2.35 ms +- 0.04 ms: 1.08x slower
	Significant (t=-30.02)

	### regex_v8 ###
	Mean +- std dev: 15.8 ms +- 0.1 ms -> 17.6 ms +- 0.2 ms: 1.11x slower
	Significant (t=-56.43)

	### richards ###
	Mean +- std dev: 37.2 ms +- 0.9 ms -> 37.6 ms +- 1.4 ms: 1.01x slower
	Not significant

	### richards_super ###
	Mean +- std dev: 45.1 ms +- 1.1 ms -> 42.1 ms +- 1.2 ms: 1.07x faster
	Significant (t=14.52)

	### scimark_fft ###
	Mean +- std dev: 209 ms +- 2 ms -> 246 ms +- 3 ms: 1.17x slower
	Significant (t=-69.77)

	### scimark_lu ###
	Mean +- std dev: 72.9 ms +- 1.1 ms -> 92.9 ms +- 2.1 ms: 1.27x slower
	Significant (t=-67.02)

	### scimark_monte_carlo ###
	Mean +- std dev: 50.1 ms +- 0.9 ms -> 58.3 ms +- 0.8 ms: 1.16x slower
	Significant (t=-51.06)

	### scimark_sor ###
	Mean +- std dev: 86.9 ms +- 1.9 ms -> 102.6 ms +- 1.4 ms: 1.18x slower
	Significant (t=-51.19)

	### scimark_sparse_mat_mult ###
	Mean +- std dev: 3.21 ms +- 0.08 ms -> 3.36 ms +- 0.11 ms: 1.05x slower
	Significant (t=-8.89)

	### spectral_norm ###
	Mean +- std dev: 70.4 ms +- 1.2 ms -> 87.2 ms +- 1.4 ms: 1.24x slower
	Significant (t=-69.46)

	### sqlglot_normalize ###
	Mean +- std dev: 221 ms +- 2 ms -> 103 ms +- 1 ms: 2.15x faster
	Significant (t=350.67)

	### sqlglot_optimize ###
	Mean +- std dev: 41.2 ms +- 0.5 ms -> 50.2 ms +- 0.5 ms: 1.22x slower
	Significant (t=-100.42)

	### sqlglot_parse ###
	Mean +- std dev: 1.04 ms +- 0.02 ms -> 1.12 ms +- 0.02 ms: 1.07x slower
	Significant (t=-25.70)

	### sqlglot_transpile ###
	Mean +- std dev: 1.28 ms +- 0.02 ms -> 1.40 ms +- 0.02 ms: 1.10x slower
	Significant (t=-30.61)

	### sqlite_synth ###
	Mean +- std dev: 1.96 us +- 0.04 us -> 2.36 us +- 0.04 us: 1.21x slower
	Significant (t=-55.03)

	### sympy_expand ###
	Mean +- std dev: 388 ms +- 4 ms -> 443 ms +- 19 ms: 1.14x slower
	Significant (t=-21.91)

	### sympy_integrate ###
	Mean +- std dev: 16.1 ms +- 0.2 ms -> 17.3 ms +- 0.3 ms: 1.07x slower
	Significant (t=-26.28)

	### sympy_str ###
	Mean +- std dev: 227 ms +- 2 ms -> 252 ms +- 3 ms: 1.11x slower
	Significant (t=-52.55)

	### sympy_sum ###
	Mean +- std dev: 121 ms +- 1 ms -> 128 ms +- 2 ms: 1.05x slower
	Significant (t=-19.10)

	### telco ###
	Mean +- std dev: 5.22 ms +- 0.16 ms -> 6.47 ms +- 0.12 ms: 1.24x slower
	Significant (t=-49.13)

	### tomli_loads ###
	Mean +- std dev: 1.64 sec +- 0.03 sec -> 1.98 sec +- 0.06 sec: 1.21x slower
	Significant (t=-41.96)

	### tornado_http ###
	Mean +- std dev: 86.7 ms +- 2.1 ms -> 92.5 ms +- 2.3 ms: 1.07x slower
	Significant (t=-14.57)

	### typing_runtime_protocols ###
	Mean +- std dev: 369 us +- 5 us -> 138 us +- 3 us: 2.67x faster
	Significant (t=288.51)

	### unpack_sequence ###
	Mean +- std dev: 32.8 ns +- 0.7 ns -> 32.0 ns +- 0.5 ns: 1.03x faster
	Significant (t=7.23)

	### unpickle ###
	Mean +- std dev: 9.40 us +- 0.15 us -> 12.33 us +- 0.42 us: 1.31x slower
	Significant (t=-51.26)

	### unpickle_list ###
	Mean +- std dev: 2.88 us +- 0.07 us -> 3.71 us +- 0.11 us: 1.29x slower
	Significant (t=-48.07)

	### unpickle_pure_python ###
	Mean +- std dev: 176 us +- 2 us -> 201 us +- 3 us: 1.14x slower
	Significant (t=-52.27)

	### xml_etree_generate ###
	Mean +- std dev: 58.5 ms +- 1.0 ms -> 86.5 ms +- 2.2 ms: 1.48x slower
	Significant (t=-88.39)

	### xml_etree_iterparse ###
	Mean +- std dev: 80.6 ms +- 0.8 ms -> 80.3 ms +- 1.2 ms: 1.00x faster
	Not significant

	### xml_etree_parse ###
	Mean +- std dev: 119 ms +- 2 ms -> 117 ms +- 2 ms: 1.01x faster
	Not significant

	### xml_etree_process ###
	Mean +- std dev: 40.7 ms +- 0.5 ms -> 58.7 ms +- 1.3 ms: 1.44x slower
	Significant (t=-98.24)

	Skipped 2 benchmarks only in py311.json: sqlalchemy_declarative, sqlalchemy_imperative

view raw pyperformance benchmarks hosted with ❤ by GitHub

You'll notice that 14 tests run faster, but 79 run slower.

You can't draw conclusions from that, of course, yet I can see this being not the best-selling point for the release.

The pathlib improvements

It's almost a footnote in the release, but it's actually really great news.

First, Path can now be subclassed. There is a lot of pain around pathlib because of some weird magic it used to do that made projects like path unable to be fully compatible.

Personally, I had several times in my career where I wanted a FS path wrapper, and just add to go through a lot of troubles just because this basic Python feature was not working as expected, which you don't know at first, of course.

Cherry on top, pathlib.Path.walk(), a Path() aware version of os.walk() is now available, which will make a few of my scripts look tidier.

Finally, pathlib.Path.glob() can now be case-insensitive. That's groovy.

A better debugging experience

Again, I love that the trend of better error messages continues. If I ever meet Pablo Galindo, I will pay him a Michelin restaurant. But that's not the only level up we get.

Pretty much nobody cared about comprehension inlining, but you cannot imagine how many times I move in pdb only to be stuck in this weird stack frame artificially created for comprehensions to work.

The removal of this is not just giving us better perfs and ergonomics, it's also removing a big "WTF" that all beginners will experience using the Python debugger with nobody in sight to explain to them what's going on. Not to mention tracers everywhere will rejoice at not seeing those marked mistakenly as function calls.

Another very, VERY nice change is the addition of magic variables in PDB: $_retval (for the value returned in the block) and $_exception (for the exception raised in the block).

Imagine you have this code:

def bar():
    breakpoint()
    return random.randint(0, 100)

How do you read the return value in pdb? You can't, and have to get it from outside the function call, or create an intermediary variable. Not anymore:

-> return random.randint(0, 100)
(Pdb) $_retval
3

If you add the improved monitoring machinery which should make observability cheaper and more accurate, plus support for linux perf profiler, it makes me very happy. Maybe one day we'll get time travel debugging.

slice objects are now hashable

You may not know it but:

dict keys can be something else that strings (tuples in there are quite handy);
you can create slice objects that represent the [x:y:z] operations on lists and pass that to __getitem__.

So it seems logical to have a mapping (or set) of possible slicing, but it was not, up to this release.

I didn't know I wanted this before they told me I couldn't :)

math.sumprod()

In my last project, we had sum(itertools.starmap(operator.mul, zip(p, q))) in a lot of places, then we created a function for it, then we realized we had bugs because of length issues and added frenetically strict=True everywhere.

Seems like 3.12 is solving that once and for all.

New command line interface

The fact you can start a web server with python -m http.server is almost a meme at this point, and I love those little utilities.

3.12 adds an sqlite shell:

python3.12 -m sqlite3
sqlite3 shell, running on SQLite version 3.31.1
Connected to a transient in-memory database

Each command will be run using execute() on the cursor.
Type ".help" for more information; type ".quit" or CTRL-D to quit.
sqlite>

And my favorite, an uid generator:

python3.12 -m uuid
a3b10cab-064d-456e-ac9a-58a125171218

Deprecations, deprecations, deprecations

This version marks a ton of stuff to be removed, like telnetlib, 2to3 or datetime.utcnow(). 3.13 better be not released a Friday.

It also removes distutils, right now, and setuptools is kicked out of ensurepip.

You will have to test Python 3.12 a bit more vigorously than usual before migrating, which, of course, you should only do in one year. Right?

Bite code!

Discussion about this post