Compile Python applications into stand-alone executables(pyinstaller.org) |
Compile Python applications into stand-alone executables(pyinstaller.org) |
Unfortunately high-quality bundling into executables just isn’t a focus of Python (nor of any other high-level language). Personally, I’ve gone back to C++ for building command-line apps - as a developer I’d much rather be writing Python, but that’s no good if I can’t actually deliver software to users.
You seem to be ignoring Tcl, which solved this problem about 20 years ago with Starkits (https://wiki.tcl-lang.org/page/Starkit) and Starpacks (https://wiki.tcl-lang.org/page/Starpack).
Wouldn't Dart's [1] and Go's executable support qualify as such (high levels languages and good executable support)?
For example, Sass is also using now Dart [2].
At work I've built a simple cross-platform packaging system which avoids issues like this and also separates library components from the main application (the former is rarely changed, the latter is frequently changed). Despite the entire application being around ~25 MB, it starts in around 30-50 ms from a network share on Linux and 1-2 seconds on Windows. On Linux it's just a bash script eventually calling a system-provided python 3 interpreter, on Windows there's a C# .exe which does roughly the same but the package also includes a Python interpreter (which is just the official "Python for embedding into applications" ZIP) which the C# process loads via P/Invoke. The entire thing is less than 1000 lines of Python/bash/C# and works with every package under the sun and apart from the C# stubs, which basically never change, it's supremely easy to "cross compile" because compilation in this system is combining the wheels of all dependencies into a single zip, which also makes builds completely reproducible.
Is that built into python or does the developer call that cleanup? I ask because I recently switched from youtube-dl to yt-dlp on a windows machine and it left multiple temporary directories full of .py files to be cleaned up by other tools. I did cntrl+c out of it at least once. Perhaps I interrupted the cleanup routine? Is the cleanup routine expected to trap kill signals or is that on the developer?
I considered using Go for command-line apps - but in my case they often have to interact with existing C++ code. That’s easy enough to do from Python, but I was advised that interop with other languages is an area in which Go is particularly weak.
And we are mostly developing internal tools, so the executable size isn't a huge concern. But we still want an executable that can be copied from computer to computer and can be opened via a simple double-click, because not everybody in our company is a software engineer, a lot of people are just used to copying the executable to their computer and double-clicking on it to launch it. We don't use the --onefile feature since it makes the launches slow because of the extraction step. So we have a folder with an executable and the dependencies in it.
PyInstaller was a very good fit for our case. Yes the size of the bundle is quite big but it does the job. And allowed us to keep our whole code base in Python.
Despite the title, pyinstaller doesn't really compile anything, it just bundles your bytecode and the interpreter into a binary. That's often useful, but it's not the same thing. You can get similar-ish results to Nuitka by combining pyinstaller with Cython but it's quite a bit of work.
Using Go solved so many long-tail bugs for us and just simplified the whole process of shipping code to user machines.
Here's the old but working build script that built both PyInstaller and Nuitka [3].
[1] https://github.com/wakatime/legacy-python-cli/tree/standalon...
[2] https://github.com/wakatime/wakatime-cli
[3] https://github.com/wakatime/legacy-python-cli/blob/standalon...
It’s the main reason I’m looking at switching to Go for these kinds of apps. Python should have a working solution as part of the standard library.
Where Python fails is dependency management and packaging, much like this xkcd portrays: https://xkcd.com/1987/
It's a shame, too, because the language is perhaps one of the easiest to read, write and just generally develop in, and has a really rich ecosystem which can be leveraged to great success.
I've felt for a long time that language specs should be more clearly separated from their runtimes. Why couldn't we have statically compiled Python? Why did native Java executables need something like GraalVM be painstakingly introduced over many years, and even then fail to work properly whenever dynamic loading is involved (e.g. Spring framework)? The answer probably lies in the insane complexity all of that involves and making these decoupled isn't feasible with our current tooling, unless we want to spend a decade developing a new language/runtime like that.
What do you exactly expect if Spring/Java heads insist on late binding and religously exercise "dependency injection"? You can configure native-image reflection based on a closed-world assumption wrt what classes are known at compile-time, but TBH it seems futile if devs use shit-tonnes of annotation, dynamisms, and reflection magic. Or, as someone else here said, "in idiomatic Java/Spring code, behavior is expressed through anything and everything, except actual Java code."
Python packaging has its problems. But other languages are confusing if you have 5 or more overlapping versions from 4 sources too.
> Why couldn't we have statically compiled Python?
We have RPython and Cython.
It's written in rust and can embed resources and dependencies in the executable.
Just repackaged my own open source project for multiple platforms using pyoxidizier.
I wish it would merge into the python ecosystem itself.
However I eventually abandoned hope that I was on a fruitful path, and changed course for web server land, embarking on the python web framework selection dance.
The updates to 3 have a bit more impact now that I'm doing most of my work in python. Some of the features dropping seem too cool to not update.
I’m from a IT dept of a oil company not much of savvy apps mostly dull.
As manager have to do lots of data wrangling for this and that form of reports.
scripts to do jobs on machine.
Python is only fullstack i know. Pyinstaller is mine goto tool to build executable and make it portable across environment.
>In the end I threw all of that out , rewrote the UI and deployed it as a web application on a server. Now the users are happy.
This sounds familiar.
[1]: https://github.com/hankhank10/false-positive-malware-reporti...
I have a GitHub CI job which is able to automate the process for each release. https://github.com/jellyfin/jellyfin-mpv-shim/blob/master/.g...
One issue with PyInstaller is that, by default, it includes all dynamic libraries that the Python interpreter has on your machine. It makes sense, it needs an interpreter and collects what is needed for it to run.
Unfortunately this might include libreadline.so, which is licensed under GPL, making your resulting executable unable to be distributed under a proprietary license.
There are ways to solve this issue, but one has to search and read documentation (and code, in my case -- when I was researching it, the docs were not clear).
I tried pyinstaller a few years ago, but it does not support some 3rd party libraries like TensorFlow that I frequently use. That said, I like that they clearly list supported 3rd party libraries so it is a quick check if pyinstaller will work for a specific project.
Still, I am a huge fan of the project and it makes it possible to make webview desktop “apps” (like or hate them) with Python.
Your app will start quicker, because it won't be a case of the "single executable" doing the old unzip-and-run thing every time.
The CPython runtime. The point of Nuitka is to not use the interpreter :P
Getting it to work with dependencies can be a bit trickier, some of them need some Nuitka-specific hacks. The Nuitka creator/maintainer actually maintains a shocking number of these himself, in the form of Nuitka "plugins". It can be hard to figure out the right command line syntax to enable these, which is obviously a small inconvenience compared to creating them in the first place but can still be annoying. So far I've used Qt and numpy plugins and they work well. Trio doesn't work at the moment but support is likely to be coming soon. As one of the top comments for pyinstaller here says, you'll probably be better off if you're using Nuitka from day 1 rather than suddenly using it for a big exclusive project.
As with pyinstaller, it has the option to bundle everything into a single file. This is convenient but has the same disadvantage of being a bit wasteful, especially on Windows where it extracts files to the temp directory on every run (as another comment says about pyinstaller). On Linux I think it uses a ramdeive.
A very nice feature of Nuitka is a alternative mode where it doesn't put everything into one file. All the pure Python is compiled to a binary but it leaves all the binary dependencies and resources separate, which you can then deploy together in a directory. This is a bit less convenient for users but a bit more efficient and has the very nice property of being LGPL compliant if you're using PyQt/PySide. It's also useful for debugging the single-file mode, since that is really a wrapper around this mode.
- Does not allow splitting the application up into stable/unstable parts to reduce update sizes.
- Does not directly allow multiple entry points.
- Requires Python to be already installed, or requires bundling Python loosely, which is very slow when used from a network share.
- Uses zipimport under the hood, which is very slow on network shares, and very very very slow on Windows network shares.
Its ample usage of reflection is just evil, polluting your stack traces with needless proxy classes and abstraction upon abstraction upon abstraction. You can debug code pretty easily, whereas doing that with annotations or XML that gets parsed and executed by code that you know nothing about leaves you in hopeless situations more often than you'd like. Instead of solving business problems, you end up solving whatever it is that Spring wants you to do.
Thus, most of your post is spot on. Rather often, it is the frameworks that are keeping us in a pretty unhappy place. In contrast, languages like Go feel a bit more pure in that regard, even generics were only added recently. Not giving the framework developers tools to express endless complexity is probably a good idea.
So to illustrate: if you were to work on a pure-python project with pure-python dependencies, existing tooling (such as pyinstaller, or nuitka, or others) can provide single binary executables, just as easily as you can in go.
Even if something officially supports multiple languages, the obscure ones will be usually get inferiorly implemented bindings just because they're not used as much and will consequently not be as tested and as patched as the main ones. It's unfortunate but that's life I guess.
To use a Python-specific framework you're better off using Python. Well duh.
Just like Go which embed it’s runtime, .NET Core binary compiled with AOT embed the CLR but, unlike Go, it has options to trim out the pieces that aren’t necessary, which I really like.
Lol what?
What goes into a 70 MB implementation of 'hello world'?
Java ‘hello world’ compiled into a static binary is just a couple of MB.
For a modern CLI app it's basically that easy. For an older or GUI WPF app it's not. It used to be fashionable in .NET to use lots of runtime reflection, dynamic runtime configuration based on config files, dynamic loading, and runtime code generation. I suppose the language was less featureful back then and people looked longingly at Enterprise Java for some reason. This type of dynamism doesn't work with AOT compilation because the compiler needs to be able to statically determine what code is required. The solution is to remove the dynamism, replace it with compile-time code generation, or annotate the reflection to tell the compiler which types will actually be used. As of .NET 6 most of the standard library has been annotated or otherwise made compatible with trimming[0]. WPF apps will probably never be able to use trimming because that's a huge amount of code and MS has like one intern working on it.
I believe the situation with Java is basically the same? You can't just build a Java 6 Spring app into a small AOT bundle, right?
[0]: https://themesof.net/roadmap?product=.NET&release=6.0&q=trim