CPython Internals Explained(github.com) |
CPython Internals Explained(github.com) |
It took me back to the old days of source diving and accumulated knowledge that you carried around in your head.
I found that agentic coding tools were quite good at answering my architectural questions; even when their answers were only half correct, they usually pointed me in the right direction. (I didn’t use AI to write code and I wonder if agentic tools would struggle with certain aspects of the codebase like, for instance, the Cambrian explosion of utility macros used throughout.)
I blended the python type system with a large low-level type system (STEP AIM low level types) and a smaller set of higher-level types (STEP ARM, similar to a database view). I already was familiar with STEP, so I needed to really grok what Python was doing under the covers because I needed to virtualize the STEP ARM and AIM access while making it look like "normal" Python.
Py_DECREF(m)From the later call PyModule_AddObject, it's clear this code has come from the PyInit_ module initialisation function. This code is running on import of the C extension to initialise the "FruitEnum" module attribute. https://docs.python.org/3/c-api/extension-modules.html#c.PyI...
Python Developer's Guide > "CPython's internals": https://devguide.python.org/internals/index.html
Python/cpython//InternalDocs/README.md > "CPython Internals Documentation": https://github.com/python/cpython/blob/main/InternalDocs/REA...
Ideally what's in InternalDocs/ would be built into the docs.python.org docs .
Is it just that markdown support in sphinx is not understood to exist?
Sphinx has native markdown support. Sphinx does not have native MyST Markdown support. To support MyST Markdown in a sphinx-doc project, you must e.g. `pip install myst_parser` and add "myst_parser" to the extensions list in conf.py.
MyST Markdown supports docutils and sphinx RestructuredText roles and directives: https://myst-parser.readthedocs.io/en/latest/syntax/roles-an...
Directive in ReStructuredText .rst:
.. directivename:: arguments
:key1: val1
:key2: val2
This is
directive content
```{directivename} arguments
:key1: val1
:key2: val2
This is
directive content
```
:role-name:`role content`
{role-name}`role content`
.. _label-name:
(label-name)=
:ref:`Link title <label-name>`
{ref}`Link title <label-name>`https://realpython.com/products/cpython-internals-book/ But it's for 3.9 and doesn't cover the massive changes regarding delayed annotations and the GIL updates
The Ruby under a Microscope guy is updating it.
I'm wondering how much of that is true and what is just a hallucination."
Btw: JavaScript seems to have similar complexity issues.
Edit: Python has no JIT
I guess they “CPython’ed” back when people thought Jython would take off , and it never did because Java sucks.
Then, there quite a few python-likes out there.
I wish they would stay precise.
It’s a bit like arguing people should start saying “homo sapiens” when referencing “people” for added precision. It may be useful to anthropologists but the rest of us really don’t need that. Similarly, CPython is really only a sensible level of precision in a discussion directly about alternative Python implementations.
(although in this case the original post is about implementation internals so I’d give it a pass)
CPython is Python. The others are attempts.
JavaScript and Node.js are different too.
No harm in being explicit right? Tis part of the zen of Python after all.
In general, I never, ever see anyone saying "I will write a CPython script". Everybody says "Python" in my experience... do you see it differently?
EDIT: I don't think that your opinion deserves to be downvoted, though...
If you expose internals in documentation, then people depend on internals.
And when you break it, because it isn't meant to be tracked by any kind of API, there are wonderful groups who will sue you (usually under "devaluation").
Of course in the real life there are de facto implementations and language features give way to better/worse tradeoffs.
With that out of the way, Python is basically the de facto glue language. It is very often used to provide a scripting API over lower level C libraries. To be ergonomic in this function, CPython (the major implementation) exposed some internal details of its execution model, which C libraries can reach into. This makes it very hard to make more aggressive optimizations, as one example a C library can just increase/decrease the reference count of an object. Another design decision (that got some discussion recently) is the GIL (global interpreter lock) that makes python much less competitive than something like Java. (JS also does a single thread of execution, though there are ways around it).
JS has a different use case, so access to the C world doesn't impose such restrictions on it.
> Edit: Python has no JIT
JIT-compiler for Python https://pypy.org/
Python enhancement proposal for JIT in CPython https://peps.python.org/pep-0744/
And there are several JIT-compilers for various subsets of Python, usually with focus on numerical code and often with GPU support, for example
Numba https://numba.pydata.org/numba-doc/dev/user/jit.html
Taichi Lang https://github.com/taichi-dev/taichi
See https://docs.python.org/3/whatsnew/3.13.html#an-experimental...
In 3.14 and up you can enable JIT by setting the env var PYTHON_JIT=1
Of course, you can't really do all of that on a free plan.
That's far from ideal, but if you are motivated and care about these technical details (which you probably do), you can get pretty good results.
=====
Putting all of this aside, you can sometimes find YouTube videos on obscure channels that talk about these things. Chances are that someone who cares to make a YouTube video about these hardcore topics know what they are talking.
No clue if that's the reason for the downvotes, but maybe next time don't mention ChatGPT and just formulate this as "From what I read, [...]".
Much of the initial work on the JIT came from Microsoft's Faster CPython team: https://lwn.net/Articles/1029307/
The module init function is where you would normally create the module object (PyModule_Create) and decref it if an error occurs. The blog example is utility code that you would call within the module init function to add an enum.
Someone should really create a blog post compiler to catch these sorts of things :-)
Python it’s not hard to write a module in pure C that manipulates other Python objects. This means the representation of Python objects has to be stable enough for the C code. V8 does not allow that.
https://nodejs.org/api/n-api.html
I’ve only skimmed this, but it sure sounds like it lets you write C code that operates on JS objects. In fact, it explicitly says “APIs exposed by Node-API are generally used to create and manipulate JavaScript values.”
But, they are technically correct. The language is defined as by CPython: it is the standard!!! None of the others fully meet that standard, which includes quirks! It's knows trade offs with them! They are, literally, attempts to adhere to that standard.
CPython is Python. Every time your buddy says “just download python” you are using CPython . There’s no reason to be pedantic.
Are things defined by the dictionary or by everyday experiences?
Python docs procedures: (0) Devguide, (1) PEPs w/ front matter in RST, (2) RST in /Doc with Sphinx, (3) MD and TXT in /InternalDocs without a toctree
The .. warning: or even admonition directives could be used for indicating that docs under /internals are not public API and can change with or without a PEP; though that should also or at least be indicated in the source unless that's a given expectation that not marked public APIs are not to be considered stable
Saying "here be dragons" is not enough to discourage people whose job it is to be creative.
It is advantageous to format, interlink, and create a table of contents for documentation. I doubt anyone would advocate for removing the Makefile and conf.py from /Doc.
That a document describes something internal does not mean that it should be excluded from the docs.
Is there a consistent standard for whether something is internal or external to the CPython project? Aren't there already internal things documented in /Doc? Should they be removed from the docs build then? Or moved to an internal folder with a DRY additional toctree?