This oft-repeated quote is always relevant on Python metaclass posts:

“[Metaclasses] are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).”

Tim Peters, Inventor of the timsort algorithm and prolific Python contributor

https://www.oreilly.com/library/view/fluent-python/978149194...

https://en.m.wikipedia.org/wiki/Tim_Peters_(software_enginee...

I would then also concur with the other comment that if you “know” you need metaclasses, 99% of the time actually you only need __subclass_init__.

A lot of online literature about Python meta programming misses out __subclass_init__ as it was only added to Python 3.6 in 2015 via PEP 487.

https://peps.python.org/pep-0487/

This is a cool overview, and I certainly learned new things about the Python language from it. Thanks for posting!

We do lots of Python metaprogramming at Mito [1], but generally avoid all of this fancy Python fluff to get it done. Specifically, we avoid metaclasses, invisible decorators, etc. Instead, we take a much simpler approach of having Python code that literally populates as template .py file, and then writes it to the correct location in our codebase.

As a concrete example: we’re a spreadsheet, so we let our users transform data in a bunch of different ways - adding a column, writing a formula, deleting some rows. Anytime I want to add a new transform (say, encoding a column), I tell the metaprogramming package “python -m metaprogramming step —name “Encoding A Column”. It will ask me some questions about the parameters and types of those parameters, and then write most of the 4-6 boilerplate Python and Typescript files I need automatically! You can see it here [2].

This is still metaprogramming (it’s certainly code that writes code). But the code you end up with at the end of the day is very simple Python code that is extremely easy to understand / maintain long-term.

I’ll pass on the fancy stuff for now. Thanks though!

[1] https://trymito.io

[2] https://github.com/mito-ds/monorepo/blob/dev/mitosheet/dev/c...

Thanks for sharing! Instead of using meta classes for subclass registration though, please just use __subclass_init__ instead. It’ll often be simpler unless you have a complex use case.

https://docs.python.org/3/reference/datamodel.html#object.__...

It is shocking how much vodoo magic there is in a programming language marketed as "easy to learn and use".

And with the recent addition of new syntax, I would hardly call Python easy.

Python is an amazing language. It's basics are very easy to learn and I'd like to think it doesn't feel like a incomprehensible sorcery to the beginner.

But even for that, it's not a toy language as clearly evident by this example.

The article's reference to "classobj" is wrong for Python 3 (that was the type of old-style classes in Python 2). In Python 3:

    >>> class SomeClass:
    ...     pass
    ...
    >>> type(SomeClass)
    <class 'type'>

The reason why you have to reinvent lisp when you go deep enough is that if you are not, it is getting too hard and too confusing.

Small typo at the start of the code examples.

We define the instance as: someobject = SomeClass()

but then we refer to it as someobj and some_object

This is awesome! Now, never ever do it.

Python is powerful and flexible enough that you don't need metaprogramming.

(I mean this quite literally: I sincerely doubt that there any code in Python using metaclasses etc., that wouldn't be more clear and maintainable if rewritten in "plain old" Python without them.)

(With the caveat that I'm not including "art" projects, I'm talking about working production code.)

(In case it's not clear, this is one of those "Prove Me Wrong" scenarios... If you think you have a counter-example to my claim, please don't keep it to yourself, "Shout it out so the whole theatre can hear you!")

It's interesting how differently two similar dynlangs, Python and Ruby, decided to offer meta-programming. Ruby decided to offer a very simple way of evaluating code at construction, but python has you define custom metaclasses. One sees it as a routine way of extending the language, and the other treats it as a 'black-art'.

Metaprogramming done wrong, no matter the mechanism, is very painful. So, I'm not sure which way is better.

It’s interesting that meta programming, one of the main selling points of lisps, is actually pretty common and possible in other languages as well. Lisp’s “code = data” achieves meta programming in a straightforward way, but this link shows that it’s not necessary. In fact, Python’s way might even be better because the language sorta “gets outta the way” because it’s really darn simple.

Stuff like this is how I created a very quick-and-dirty SaltStack module for AD Federation Services:

https://github.com/irtnog/salt-states/blob/production/_modul...

The SaltStack module starts out empty except for a function that runs at module load time. The initialization function queries PowerShell for all AD FS-related cmdlets and creates Python wrapper functions for them. It even copies the cmdlet's help to the function's docstring.

As someone who just dabbles in python for small scripts, this is fascinating to me.

You can use some metaprogramming to create very clean interface points in python! I always wondered how django did so much with very clean readable implementations for end users.

very interesting, bookmarking for later. thanks.

Python is easy, not because it prevent people from doing complicated things, but because with only a few simple things, you are productive.

Most python programmer use 10% of the language, the same 10% that is described in most tutorial, because that's enough, and they don't even know the rest exist.

This makes for a very smooth, but long learning curve which allow you to enjoy python in the early years, yet keep getting a kick out of it after 15 years.

I hear a lot of people raising the same kind of concern you do in comments, but in the field, I never do with people actually using the language.

> It is shocking how much vodoo magic there is in a programming language marketed as "easy to learn and use".

But that's exactly what that expression means?

A car with an automatic transmission is "easy to learn and use", but an automatic transmission is way more complicated to construct than a manual one.

Ran into this face-first, just yesterday. I'm converting a rather old, rather big API (~400 endpoints) from Python 2.7 into 3, and apparently in Python 2, _hasattr(something, someattribute)_ just returns False if attribute access throws an exception! Specifically, if you have

  class A():
    @property
    def a(self): 
      raise Exception('go away')

  a = A()
 

then hasattr(a, 'a') will return False in Python 2.7 (and throw as expected in Python3)

A true WTF moment, the tests and the API have been slightly broken for years, without anyone noticing.

What an odd thing to say. This article is not for beginner Python programmers, and you don’t need to know anything in this article to use Python. I know people who refuse to use decorators because they seem difficult, and yet they are able to make very useful applications.

I’ve been writing Python professionally for over 20 years, and I’ve needed to use the things here once. It was highly encapsulated inside one module of an ORM-type project, where other code wanted to use it without having to know anything about its implementation details.

You can do these things in the extremely rare cases that you must, but other than that you shouldn’t.

For a language that is actually "easy to learn and use", look at Lua[1].

It's so easy to learn & use that children manage to be productive with it (the Roblox community).

It's still powerful though. With few enough primitives to remain simple & understandable, it has just enough primitives to build anything. People have written web frameworks, window managers and video games in Lua.

It's got less pitfalls, "voodoo magic" and advanced weirdness than any other language I know. You can learn 90% of the language from its Wikipedia page. Every time I encounter some of Python's internal weirdness, I wish Lua had been the scripting language of the 2000s instead. Unfortunately Python has a vast collection of libraries available that keep us using it instead.

[1]https://www.lua.org/pil/contents.html

That's not uncommon, when you make something easy* to learn and use you obfuscate certain things that are more "advanced" to streamline the onboarding experience, and then when you want to break out of the streamlining you gotta learn to circumvent those "stops" somewhat, which often end up being pretty non ergonomic. It compounds as well because the ones developing the language will have a bias towards what it currently offers and the "unorthodox" side keeps getting set aside.

*and by easy there I mean streamlined and more intuitive on most expected "common" tasks than the alternatives

Most Python programmers never need to worry about the "voodoo magic". But for writing tools and libraries that can nicely encapsulate a desired behavior and make it simple for other programmers, Python's "voodoo magic" is great.

You consider python full of vodoo magic? Go learn guile for a week and let me know what you think then :-)