What new Python features are the most useful for you?

42 points by artimis 3 years ago | 42 comments

In recent years Python ecosystem evolved significantly. From using requirements.txt to dependency management tools like Poetry, from simple typing to Protocols [1] and Variadic Generics [2]. We've got walrus operator, match case statement, and finally even faster Python. More and more low-level libraries are being written in Rust [3].

What new features, tools, libraries (already implemented or being worked on) are the most useful for your work with Python? What are you the most excited about?

[1] https://peps.python.org/pep-0544/ [2] https://peps.python.org/pep-0646/ [3] https://github.com/pydantic/pydantic-core

simeonf 3 years ago |

Just getting back on the latest open source Python ecosystem after a long stint of mostly writing Python in a corporate mono-repo context with bespoke tooling.

So far I like:

  - black autoformats my code swiftly
  - ruff is fast and replaces multiple python linters
  - pip-tools lets me manage multiple dependency specifications (local, prod, test) more sanely than manually syncing `pip freeze` with multiple requirements files. 
  - f-strings are nice improvement over .format calls, % string interpolation, string.Templates, etc.
  - I appreciate pattern matching and case classes in Scala. I expect to like them  using dataclasses and pattern matching Python

kingkongjaffa 3 years ago |

I use Type hints and using mypy strictly.

Huge fan of dataclasses as well.

Most of the python I write is functions / functional + data classes with type hinted everything and it seems to work nicely and make writing tests for functions very easy.

Using black, isort, mypy, pylint, pytest, makes every project simpler.

pizza 3 years ago |

- einops (probably by far the most useful Python dsl for me as of late, esp. if you deal a lot with tensors)

- numpy __array__ interface standardization (easy compatibility across a lot of numerical libraries, e.g. torch, awkward-array, zarr, jax.numpy, dask, xarray)

- the various jit compilers (e.g. numba, jax.jit, torch.compile)

- hw-accelerated functional programming style (e.g. functorch.vmap)

- pybind11 for easy af bindings to c++

- named dimensions for tensors

- pandas-compatible ecosystem (eg seaborn)

- pytrees

- pyenv

- mamba

What I really need, though, is a realtime successor to matplotlib with a much simpler interface/dsl. Haven't looked into plotly much but probably will try it soonish

randomluck040 3 years ago | |

Matplotlib is really killing me. I never have learned it properly and switch between methods to address the same things in Matplotlib. Every time I have to plot something I have to Google how to do it or I have a template ready already. In my opinion it’s the most annoying library I have to use.

z3c0 3 years ago | | |

In situations where matplotlib is the lingua franca, like notebooks, I've opted for seaborn. You'll still have to deal with matplotlib's complex nonsense, but substantially less of it.

xkcd-sucks 3 years ago | | |

Altair/Vega ( https://altair-viz.github.io/ ) is really nice to use, although headless rendering outside of a notebook environment is a bit of a pain

vram22 3 years ago | |

I had checked out plotly some years ago:

plot.ly now has an API:

https://jugad2.blogspot.com/2013/07/plotly-now-has-api.html

rsfern 3 years ago | |

Maybe not practical for your workflow, but making interactive plots in Pluto.jl is so much nicer than hacking around in matplotlib

If it was easier to use virtual environments with PyCall I think I’d abandon matplotlib entirely.

snowpid 3 years ago | |

plotnine is an API based on Grammar on Graphics. I use it as an replacement for matplotlib.

va1210 3 years ago |

I find myself using the walrus operator a lot nowadays, and find I often miss it when writing JS/TS (previously I mostly found myself craving arrow functions and the far superior type system from TS when writing Python).

mianos 3 years ago |

In the 15 years I have been developing in python (and C++ at the same time, daily), I can't remember a single bug I ever had on account of an incorrect type.

I do like the := to save a few lines here and there.

3.10 does seem to run faster but maybe that is my imagination.

I wish more people would focus on writing good python.

uglycoyote 3 years ago | |

I live on the opposite planet. basically every bug that I have ever had in python over my 15 years of using it has had to do with a type issue. So the increased use of types and python and ability to express type concepts in python is the thing that I'm most excited about.

it is not necessarily about type mismatch bugs per se, but more like without type information in the code I will misunderstand what kind of structure is passed in to a function, how to decompose it, what members the data has, what functions does it have, how do you spell them correctly, what elements to expect in dictionaries and what kinds of values do they have, and how can I make massive changes to code without knowing all those answers and expect it all to still work.

type annotations are solving a lot of those issues for me nicely, and the number of things that it doesn't solve well is decreasing with each version and the new features they are adding

nicwolff 3 years ago | |

3.11 runs a bunch faster.

uptownfunk 3 years ago | |

what do you attribute to never finding a bug due to incorrect type? and what is good python vs bad python?

mianos 3 years ago | | |

> what do you attribute to never finding a bug due to incorrect type?

If the function/method expects an int, string or complex type and I accidentally call it with the wrong type, resulting in an error, such as "can't add 1 to a string", or "can't access access that method because a string was passed".

>and what is good python vs bad python?

This is kinda of subjective but, for a start, don't just start at the type and do stuff until the bottom like a shell script. This is by far how I see most python scripts. All the other normal things. Don't repeat yourself (in moderation, a few times is OK). Keep it simple.

thenipper 3 years ago |

- Pydantic: basically it might as well be part of the standard library for me - I'm excited by polars as a potential pandas replacement - Speaking of pandas i really like pandera and how it integrates with both pandas and pydantic

jstx1 3 years ago | |

Why use Pydantic instead of dataclasses?

rsfern 3 years ago | | |

Pydantic can do validation. There’s also the related Typer, which builds command line interfaces from the type annotations on your entry point

musicale 3 years ago |

The most useful feature to me would be for Python to have long-lived standard editions of the language the way other programming languages do.

For example, clang and gcc can still compile C89 programs.

asicsp 3 years ago |

Good to have Possessive quantifiers and Atomic groups added to the `re` module (wrote about it here https://learnbyexample.github.io/python-regex-possessive-qua...)

Hope they will add support for Unicode sets `\p{}`, subexpression calls, (*SKIP)(*F), etc. For now, the third-party `regex` module is handy for such features.

mharig 3 years ago |

Most used: f-Strings Second: pattern matching and dataclasses

If I ever could wrap my brain around asyncio, it would be an appreciated third.

I never used the walrus operator, IIRC, and I find type hints not so useful. Somewhere in the future I will try to use them in APIs, they may have their benefit there.

Edit: I forgot enums. Not so new, but a must.

bradwood 3 years ago |

The `click` library would be a nice replacement to `argparse`

vladsanchez 3 years ago | |

"Click" and "DocOpt" are the 'argparse' replacements. They're the most semantic and intuitive way of developing command-line apps.

Hasnep 3 years ago | | |

I've found Typer [1] pretty intuitive to integrate with existing code, it uses type hints to generate the CLI arguments.

[1] https://typer.tiangolo.com/

agons 3 years ago | | |

I find `typer` way more ergonomic than either argparse or click (which it uses under the hood).

If you're developing something more complex, you might find the feature set too small, but you can get surprisingly far with what's there.

The benefits from its type safety and lack of boilerplate alone are enough to keep me using it in favour of anything else.

vladsanchez 3 years ago | | |

Thanks for sharing Typer, I was completely oblivious!

> Typer uses Click internally. That's the only dependency.

Pretty clever way to walk on giants shoulders!

BerislavLopac 3 years ago | |

Which is fine when building a complex CLI tool which requires other external libraries; but otherwise the compexity of argparse is simply not significant enough to justify the cost of adding external dependencies.

bjourne 3 years ago |

The improved destructuring syntax is amazing:

    first, *middle, last = range(10)

Set and dict comprehensions are also very useful but aren't that new anymore. Also, the Rich library is amazingly useful for all kinds of console applications.

_8j50 3 years ago |

f strings. I am averse to complexity and they actually do reduce complexity. Like I actively avoid object oriented stuff, asyncio,etc... simplicity, predictability and maintainability rule for me. Most other python newer features I have seen add more code or complexity. I suppose managing packages, build environments and constantly migrating code is something some people enjoy so no problem with it until it becomes my problem by way of dependency hell. My favorite python feature is not a python feature but it should be: pyenv -- because backwards compatibility is not a cool new feature for language devs.

kbrannigan 3 years ago |

The lack of boilerplate. Low ceremony. Just write the code, save the file and run it.

That in itself is a great feature.

jstx1 3 years ago | |

But all the other files, and the dependencies, and where is it going to run, and does it have permissions to see all the data and account resources it needs, and the testing, and...

the__alchemist 3 years ago |

Type hints, Data Classes, and Enums

- black autoformats my code swiftly - ruff is fast and replaces multiple python linters - pip-tools lets me manage multiple dependency specifications (local, prod, test) more sanely than manually syncing `pip freeze` with multiple requirements files. - f-strings are nice improvement over .format calls, % string interpolation, string.Templates, etc. - I appreciate pattern matching and case classes in Scala. I expect to like them using dataclasses and pattern matching Python