My Python Development Environment, 2020 Edition(jacobian.org) |
My Python Development Environment, 2020 Edition(jacobian.org) |
1. Install Anaconda to your home user directory .
2. create environment using (conda create --name myenv python=3.6) .
3. Switch to the environment using (conda activate myenv) .
4. Use (conda install mypackage), (pip install mypackage) in that priority order .
5. Export environment using (conda env export > conda_env.yaml) .
6. Environment can be created on an other system using (conda env create -f conda_env.yaml) .
Anaconda: https://www.anaconda.com/distribution/#download-section .
Dockerized Anaconda: https://docs.anaconda.com/anaconda/user-guide/tasks/docker/ .
One pain point though: getting it to work with Sublime Text 3 requires you to set the `CONDA_DLL_SEARCH_MODIFICATION_ENABLE` environment variable to `1` on Windows.
Not a flaw of Anaconda: it just pays attention to how to with multiple Python installations on Windows.
This difference is important due to dependencies- if you have two different CLI tools you want to install but they have conflicting dependencies then pip is going to put at least one of them into an unusable state, while pipx will allow them to both coexist on the same system.
pipx, poetry, pipenv and co are still nice wrappers to have, I suppose. It just feel less useful now that most of my projects are dockerized.
Doesn't seem fair. You're not abandoning requests, are you?
Edit: From https://www.python.org/psf/github/,
> ... we have created a GitHub organization, @psf, to support and protect projects that have outgrown ownership by their original author.
2. Conda is not used as much as you might think... it's really only used within the data science community.
2. Conda is quite widely used outside of data science. It’s for example part of Anaconda enterprise offerings used by huge banks, government agencies, universities, etc., on large projects often with no use cases related to data science. Conda itself has no logical connection with data science, it’s just a package & environment manager.
In each of my last 4 jobs, 2 at large Fortune 500 ecommerce companies, conda has been the environment manager used for all internal Python development. Still use pip a lot within conda envs, but conda is the one broader constant.
That way 'pipx install foo' can download and install wheels rather than downloading source distributions and building/installing them...
> large Fortune 500 e-commerce companies
Sorry, but argument from authority doesn't mean anything to me.
In all seriousness though, you literally did not provide any logical reasons to think conda is better.
Yes you do.
I don't say this out of negativity for the sake of negativity. Earlier today, I was trying to resurrect an old Python project that was using pipenv.
"pipenv install" gave me an error about accepting 1 argument, but 3 were provided.
Then I switched to Poetry. Poetry kept on detecting my Python2 installation, but not my Python3 installation. It seemed like I had to use pyenv, which I didn't want to use, since that's another tool to use and setup on different machines.
I gave up and started rewriting the project (web scraper) in Node.js with Puppeteer.
What people casually think of as "Python" is really a huge dynamic ecosystem of packages. Imagine that there are 60k packages, each with 1k lines of code... that's a 60 million line codebase, and it can't be checked for breaking changes. Short of continually testing your code against the latest versions of packages, you're going to hit some bumps if you haul an old code out of the vault and try to fire it up on a new system.
I don't know how Javascript developers handle this.
I handle it by running Python inside an isolated environment -- WinPython does this for me -- and occasionally having to fix something if a new version of a package causes a breaking change.
The drawback of my method is deployment -- there is no small nugget of code that I can confidently share with someone. They have to install their own environment for running my stuff, or take their chances, which usually ends badly.
Most package managers have lockfiles that allow for some degree of determinism. Of course if a library introduces breaking changes you're going to have to rewrite, but only when explicitly upgrading the dependency.
That said, I 100% agree. I don't have the answer, except that I wish that there was one official answer, for developers and deployment that was easy to explain to beginners.
For what it's worth I've been using pipenv for over a year and it works good enough. I think npm's approach is better, but not perfect. I've heard good things about yarn. I know CPAN is the grandfather of all of them. I've barely used gems but they seem like magic, and Go get your repo URI out of my code please and thank you. :-) All fun aside, what languages have it right? and is there maybe a way to come up with a unified language dependency manger?
Technically, OP didn’t have to use pipenv for anything except knowing which versions of each dependency to install (Pipfile.lock) with good old pip. Those other tools are mere conveniences. Giving up on a language for that...that’s drastic.
For interpreter/compiler version management you can use asdf [0]. It works for all popular programming languages. You can use it to replace tools such as pyenv, nvm, gvm, etc.
Everything is pinned to exact source revisions. You can be relatively sure to be able to git clone and nix-shell and be off to the races.
You can even go the extra mile and provide working editor integration in your nix-shell (especially easy with emacs). So you can enable anyone to git clone, nix-shell, and open a working editor.
The biggest downside is becoming proficient in Nix isn't easy or even straightforward.
While you're here, how do you do this with nix-pkgs?
I looked into using nixos for nodejs deployments recently, and was amazed to find that the versions of node in the nix-pkgs repo are just pinned to X.Y.0 releases, with no discernable way to update to a bugfix release after .0 , so... I don't see how this could possibly be used for production deployments?
https://nixos.org/nixos/packages.html?channel=nixpkgs-unstab...
What I really want is to be able to tell nix to give me nodejs at version 10.16.3 and have it do the right thing every time.
I'm happy to be wrong on any of this.
Occasionally I still run into weird non-deterministic issues with builds, though: https://github.com/NixOS/nixpkgs/issues/71178
You'd think "pinning" a python version + channel would avoid this.
In Python it's a mess. Some packages specify their deps in setup.py; some in a requirements file, which may or may not be read in by their setup.py. It's not rare to have to have multiple 'pip install' commands to get a working environment, especially when installing fairly boutique extensions to frameworks like Django.
There just isn't a holistic, opinionated approach to specifying dependencies, especially at a project level. Which leaves unexpected upgrades of dependencies (occasionally leading to regressions) as a reality for Python devs.
One is pipenv [0], which works similar to yarn & co. It uses Pipfile/Pipfile.lock to define and lock dependencies. Pipenv has a major flaw: It can't be used to publish packages on pypi.org (you still need twine & setup.py for that). It's also known for being slow and somewhat buggy. Despite all that pipenv is an "official" tool maintained by the "Python Packaging Authority".
The other one is poetry [1], which works exactly like yarn & co. It uses "pyproject.toml" to specify dependencies and "poetry.lock" to lock them. Poetry does most of the things "right", but it's still an underdog compared to pipenv.
Both tools have not yet fully matured, thus there are a lot of complaints.
Now try that with JavaScript or Rust. If you don't fix versions, come back three months later and compatibility is usually fucked up beyond all recognition.
Some languages embraced better dependency locking because they absolutely couldn't not solve the problem.
pipenv generates a Pipfile.lock - if it can, I've used it primarily for Airflow, and some packages within Airflow have incompatible version ranges for the same dependency, which means it can't generate the lock file.
npm sucks and all, but at least it just works and doesn't get in my way as much.
used many package managers: pip, gem, go, npm, composer, etc... npm is the only one i have recent memories of having to kill & the only one that makes the fan go off (well okay most c/c++ do that too...)
quite frankly surprised by what i am seeing about python here. i have never been into the new stuff, pipenv, poetry, pipx, etc... maybe that's where the bad experience is coming from? i even don't know when and why it got so complex...
Like any dictatorship, that can be fine if those in charge are benevolent and competent. For programming languages, the first is almost always true, but the second is far from guaranteed. Skill at programming language development has no bearing on skill at developing a build, packaging, and distribution system.
Go is a prime example of this. The language is pleasant to use with few ugly surprises, but their build system has been awful for a decade, only now reaching a semi-decent state with modules (which are still pretty damn ugly).
With python, on the other hand, there's competition in this space, and as a result the tools are pretty nice, albeit fragmented.
But then there's rust, which has a nice language AND a nice build system. You take a big risk when building both the language and build system; sometimes it works, sometimes it doesn't. And you risk fragmentation if you don't. It's a tough choice.
python3 -m venv /tmp/foo
/tmp/foo/bin/pip -U pip wheel
/tmp/foo/bin/pip -r requirements.txt
I understand some might not like it, but really, it's simple and it works.
Tkinter and mpi4py are the most recent ones I've had this problem with. I expect someone will tell me "it's trivial to install these in a venv, just do X", but X is not obvious to me.
Having said that: you generally want this packages integrated with your system (which provides a self-consistent Tcl as well as MPI-environment)
It’s super useful for maintaining static versions of python, like 2.7, 3.6 and 3.7 when you have many projects that have different python requirements.
If you're working on closed-source code or have tight control of your environment, it's enough to develop and run on a single version, rendering pyenv and whatnot unnecessary.
This post reads most to me as fad. It is an opinion, you would most likely bail out.
Very few issues arise from our choice of build tooling. Not enough to consider switching at the moment. I suspect I'll try pyenv next time I have a new dev maching to set up but only as it seems fairly painless to switch from virtualenv.
It just shows that Python is used for a lot of purposes and there is no single tool that handles all usecases.
> I gave up and started rewriting the project (web scraper) in Node.js with Puppeteer.
And when you'll get to the point where you need to work with different node projects, you will also need several tools to manage node versions and different environments, so that doesn't help at all.
* "I need to develop against multiple Python versions - various Python 3 versions (3.6, 3.7, 3.8, mostly), PyPy, and occasionally Python 2.7 (less and less often, thankfully)."
* "I work on many projects simultaneously, each with different sets of dependencies, so some sort of virtual environment or isolation is critical."
So sure, there is more than one way to manage a Ruby runtime version, but are there any competitors to Ruby's Bundler? I feel like it's the undisputed champion of Ruby dependency management, working with Rubygems, another unopposed incumbent in its own space, and it never even occurred to me that my language should have more than one of either such tool. Can someone help me understand what drove the Python ecosystem to need more than one dependency manager, ELI5?
I am pretty cloistered as a Ruby developer, but my limited professional experience with other language runtimes tells me that the Ruby "developer environment" experience is really second to none, (change my mind.) Is there any tool which is nearly comparable to Ruby's Pry for Python or other interpreted languages? (I doubt that there is!)
Managing gemsets and multiple versions of Ruby is old-hat.
poetry == npm
pyenv == nvm
pipx == npx
No big difference, IMO.
Package Management - npm? bower? yarn? Which should I use this week?
Interpreter Versions - Revisiting a project I last touched 2 years ago on node 8 has a load of locked dependencies that only work on that node version. OK, let's bring in nvm so I can get a dev environment going.
Executable Packages - oh no I've got two different projects with different and incompatible versions of apollo, what do I do? Oh right, sure npx for executable isolation so we don't pollute the global namespace.
Every ecosystem has these problems, and if they don't it's probably because they're still relatively esoteric.
Since Python did not prescribe a way to handle it the community has invented multiple competing ways to solve the problem, most of which have shortcomings in one way or another.
To further add to the confusion, most Linux and Unix-based operating systems (Linux, MacOS, etc.) have their own system Python which can easily get fouled up if one is not careful.
This is one place where Java's use of a virtual machine REALLY shines. You can build an uberjar of your application and throw it at a JVM and (barring JNI or non-standard database drivers) it just works. There is also usually no "system Java", so there is nothing to break along those lines.
Exactly. It's not Python's only problem, but far and away the most painful snags I've hit with packages is when they use C code, and thereby drag in the whole system. "I'll just `pip install` this- Oh, I need to install foo-dev? Okay, `apt-install foo-dev`... oh, that's in Ubuntu but not Debian? Well this is gonna be fun..." Now I trend a bit more exotic in my systems (NixOS, Termux, *BSD, ...) but if my Python were just Python it would just work so long as I have a working Python install; in practice that's rarely enough.
Oh, but there is :( `jenv shell 1.8` is muscle memory for me now.
Given that: are you sure problems arise around pure python-packages (which generally have a well enough forward compatibility), or is the problem with all the cool "machine-code"-embedded packages (which are a lot!)? And yeah, these might indeed break randomly, when installed on different systems of different time-periods. But that's a problem all binary packages have!
Nowadays, using setuptools to create packages is really easy too, there's a great tutorial on the main Python site. It's not as easy as node.js, sure, but there's tools like Cookiecutter to remove the boilerplate from new packages.
requirements.txt files aren't very elegant, but they work well enough.
And with docker, all this is is even easier. The python + docker story is really nice.
Honestly I just love these basic tools and how they let me do my job without worrying about are they the latest and greatest. My python setup has been stable for years and I am so productive with it.
Being from JAVA Shop for long time, If I have to switch between different version of JAVA, all I do is change JAVA_HOME to point to correct version, go to base project directory and "mvn clean install" does the job. :).
The one thing I thought was neat here was pipx. I do have a few CLIs set up in my default conda env and haven't run into any dependency problems yet, but have occasionally tried to use them while another env is activated. Having a separate env automatically created for the entry points is a nice value add.
Need to have multiple versions of python installed and easily accessible? Use pyenv.
Need to run tests across multiple versions of python? Use tox.
Need to freeze environments for deployment purposes? Use pip-tools.
Need to freeze the entire operating system? Use docker or vagrant.
Don't use tools you don't need. That would be silly.
However, you can run into issues if you are using different versions of Python, or Python on different operating systems.
I mean compared to Java and C# I have a MUCH MORE EASIER time to set up my development environment. Installing Python, if I am on a Windows box I mean, is enough to satisfy a lot of the requirements. I then clone the repo of the project and
source venv/bin/activate
pip install -r requirements.txt
is enough to get me to start coding.
Because it's more complex if you have projects on multiple Python versions and if you want to lock your Python packages to specific versions. (Pip can bite you when different packages have different requirements for the same lib).
https://github.com/sdispater/poetry/issues/571
the OP himself has a fix for this in his own dotfiles repo:
https://github.com/jacobian/dotfiles/commit/e7889c5954daacfe...
(Though, how'd you find that? Mildly creepy that you know more about my dotfiles than I do!)
How so? I've been using Docker for development for years now and haven't experienced this EXCEPT with some slowness I experienced with Docker Compose upon upgrading to MacOS Catalina (which turned out to be bug with PyInstaller, not Docker or Docker Compose). This is on a Mac, btw; I hear that Docker on Linux is blazing fast.
I personally would absolutely leverage Docker for the setup being described here: multiple versions with lots of environmental differences between each other. That's what Docker was made for!
My experience has been that it's significantly more effort to meet my requirements with Docker, and that I spend a _lot_ of time waiting on Docker builds, or trying to debug finicky issues in my Dockerfile/docker-compose setup.
I'm sure all of these things have fixes -- of course they do! But I find the toolset challenging, and the documentation difficult to follow. I'd love to learn what I'm missing, but I also need to balance that against Getting Shit Done.
What's the situation where your application needs to go first in the Dockerfile, and then you need to put a bunch of stuff that doesn't depend on your application?
I find it useful to keep a WinPython installation on a flash drive in my pocket. I can plug it into somebody's computer and run my own stuff, without worrying that I'm going to bollix up their system.
I never manged to hose the OS Python on Linux, by sticking to a very simple rule: DON'T BE ROOT. Don't work as root, don't run `sudo`.
On Linux, I use the system python + virtualenv. Good enough.
When I need a different python version, I use docker (or podman, which is an awesome docker replacement in context of development environments) + virtualenv in the container. (Why virtualenv in the container? Because I use them outside the container as well, and IMHO it can't hurt to be consistent).
python -m venv venv
source venv/bin/activate
pip install -U pip
pip install whatever
# <do you stuff here>
deactivate
[1] https://docs.python.org/3/whatsnew/3.4.html#whatsnew-pep-453
And, given that I've been trying NixOS lately and had loads of trouble and failing to get Conda to work, I will definitely give this setup a try.
(I haven't quite embraced the nix-shell everything solution. It still has trouble with some things. My current workaround is a Dockerfile and a requirements.txt file, which does work...)
1. Highlight to run 2. Remoting into a kernel
Both features are somewhat related. I want to be able to fire up a Python Kernel on a remote server. I want to be able to connect to it easily (not having to ssh tunnel over 6 different ports). I want connect my IDE to it and easily send commands and view data objects remotely. Spyder does all this but its not great. You have to run a custom kernel to be able to view variables locally.
Finally, I want to be able to connect to a Nameko or Flask instance as I would any remote kernel and hot-swap code out as needed.
I gotta give poetry a try, though.
In principle, it's a good idea; in practice, I'm not satisfied. On Windows, it's an easy solution, especially for packages that depend on non-python dependencies (e.g. hdf5).
Can anyone comment on the Docker learning & troubleshooting story for python?
You'll get confident and from there learning `docker-compose` is a breeze.
It feels pretty comfy to effectively be on an island and far away from the hustle and bustle of the industrial Python tooling.
There is a lot of tools and confusion between versions especially because of the breaking changes between v2 and v3 (try to explain someone why he has both and when he types python it will take v2.7 as default).
I love the elegance and simplicity of the language and many tools written with it but this is a point I'd really much appreciate to be improved.
Because of that I sometimes just rewrite something before fixing it with 2.7. It's perhaps a bit more work sometimes but not as frustrating as trying to get something running that is deprecated in parts.
In Smalltalk you change parts of the environment. In Python, Java (and Ruby?), you change the entire environment, as described in TFA.
./gradlew run
Haskell with new-style cabal works like a charm with `cabal init` for package setup. And then ghci from there will give you...
- expression evaluation ofc
- type-at-point via type hole annotations
- Laziness inspector via :print
- Breakpoint debugger (someone just posted a nice Reddit text post about it today)
- Package information of terms via :info (and :load $MODULE to get any top-level term in $MODULE into repl scope)
- Docs via :doc (this is pretty new)
- Module export lists via :browse
off the top of my head
$ curl -sSL https://get.haskellstack.org/ | sh
$ stack install turtle
$ cat >> ./hello.hs <<EOL
#!/usr/bin/env stack
-- stack --resolver lts-14.14 script
{-# LANGUAGE OverloadedStrings #-}
import Turtle
main = echo "Hello, world!"
EOL
$ chmod u+x ./hello.hs
$ ./hello.hs
Hello, world!
On the contrary, I seldom had these kind of issues with projects coded in C#, C or C++: most of the time the few steps to compile the project succeeded and produced a usable binary.
As far as virtual enviroments go, I actually kind of like them. They were containers before containers became a thing, and I’ve had much fewer issues with them than say NPM, but they are more burdensome than adding full binaries to your C# project.
Where compiled languages shine, and maybe I’m misunderstanding you, is when you need to distribute and executional to end-usere. C# is much better at that than Python, but we haven’t actually done that at my shop, in around 10 years.
That said, once you get it down, you're not burdened by it much/at all. You can start and begin work on a new project in seconds, which feels important for a language that prides itself on ease of use.
But yeah, not a great look for newcomers.
From there it's just "workon projectname" and just "deactivate" when I'm done (or "workon otherprojectname")
It has been stable and working for ages now. I just don't see any strong incentive to change.
Isn't this just a variant of what the original comment is critiquing, though? In order to sanely use Python and external dependencies you need some highly stateful, magical tool that does 'environments' for you. The conceptual load of this is quite high. Adding docker only makes it higher - now you're not merely 'virtualizing' a single language runtime but an entire OS as well - just to deal with the fact your language runtime has trouble handling dependencies.
(and for all npm-friends: it's exactly the same "conceptual load" which arises from having node 8 and 12 around. except that these will be just incompatible by default, so noone bothers)
You can use `pip-tools` to get something like a Gemfile/package.json, but there are a few restrictions that are suboptimal.
So Pipenv/Poetry are the current best ways to get something like the package management story that other languages like Ruby and JS have had for a long time (and that Go's been polishing recently too).
I did a quick comparison here[0], and I'm planning to do an update with the latest version of Poetry.
[0] https://johnfraney.ca/posts/2019/03/06/pipenv-poetry-benchma...
But what really pushed me away is that installing or upgrading any single package upgraded all dependencies, with no way to disable this behavior. (I believe you can now.) A package manager should help me manage change (and thereby risk), not prevent me from doing so.
Poetry is probably the best of the all-in-one solutions. It does its job well but I've found the documentation lacking.
In the end, I've settled on pyenv-virtualenv to manage my environments and pip-tools to manage dependencies. It's simple and meets my needs.
Now that poetry also manages virtual environments it’s far and away the better choice.
Caveat - Heroku doesn’t understand pyproject files yet, so no native poetry integration. Heroku are working on this.
The main reasoning was so that I could easily build and publish packages to a private repository and then easily import packages from both pypi and the private repository.
Happy to answer more questions.
We have a custom pypi server and need all requests to go through it, however haven't figured a way to make poetry always use our index server for all modules instead of pypi.org
I've flagged trying to manage python versions on my machine and just develop inside docker containers now (Thanks to VSCode). Using tightly versioned python base images
That said, all I want is for a unified standard to emerge, this is getting a little ridiculous...
Coming from the scientific computing realm, I've only ever done "real" code in Python until a couple of weeks ago when I was forced to do some work in Typescript. Once I got over the initial worries about the best method to install npm to avoid Python-esque problems and gave nvm a try, I was very pleasantly surprised by the package management process and was able to just get on with the work.
I've tried various Python env management solutions in the past (mostly leaning towards conda), but had recently settled on just using separate LXC/LXD containers for each project.
Exactly! I'm not aware of any non-compiled language where (all) these issues are solved much better. I can be very productive with the tools I mentioned above and I'm glad that they work almost identical for both my main drivers (Python and JS/TS).
Pip works, virtualenv works. They might not be great tools, but they do the job.
I don't want to worry to much about my environment, that's why I'm skeptical about new tools. Because they might break when you least expect
Virtualenv solves 90% of the problem, and it is not like you need to revisit your setup on a daily basis.
I would be more willing to spend time on my test setup than anything else, because that affects the development experience.
It's not as nice as just doing a regular pip -r, but it works and isn't that much effort.
$ nix run "(import <nixpkgs> {}).python37.withPackages([ pandas statsmodels ])"
This would drop you into a shell with Python 3.7, pandas, and statsmodels installed.
For more complex use cases, you should specify your dependencies in a shell.nix file or write your own nix package.
[[tool.poetry.source]]
name = "my-repo-name"
url = "https://myrepo.url/"If Python would have a similar concept rather than depending on a global module location we would be able to replicate the same developer ergonomics as we have for the JVM.
These days, most productions environments are effectively isolated containers. If your production environment is a container, you probably should develop in a container as well. In that case you don't need much tooling for isolating an application's dependencies from other applications.
The tooling that you need is to build a python application, which means (1) get the dependencies (2) copy over some source code (or invoke the C-compiler if build a C<->Python extension) (3) run tests. Python's builtin setuptools does that fine. It didn't strike me as amazingly simple, but it's not amazingly complex either. pip is essentially a convenience wrapper for setuptools, i.e. pip is to setuptools as apt is to dpkg.
Basically, I believe that because of docker isolation is a irrelevant criterion by which to judge a language/ecosystem.
I'm a developer that supports multiple production applications, and I frequently need to make my development environment as similar to production as possible in order to reproduce a production issue for debugging purposes. I depend on that isolation to be able to do this. My work environment is such that I'm not generally permitted to use containers in production (yet). So it stands to reason through your argument that perhaps I shouldn't use them in development either. It sounds like if I were using Python as well instead of Ruby, I'd be having a much harder time.
Honestly, if I could use containers in dev and prod I would, I truly do believe the grass is greener ;) but I would not sacrifice this marvelous isolation tech, in fact I'd prefer to take RVM with me into docker-land so that I can A-B test Ruby versions within the same container image, and be guaranteed that all my cluster's worker nodes will not have to take extraordinary measures and carry both images in order to ensure the application can boot without a download delay, whenever we have to revert the canary (or whatever other minor potentially reversible lifecycle event would normally trigger a node to need to download a new, expensive base set of image layers all over again.)
This works really well: https://github.com/ms-ati/docker-rvm
However in SemVer a major version number change is how breaking changes are documented, so seeing a v1 to v2 change coupled with having to do some work to fix breakage is just expected, something that may well be necessary for a project to make progress.
My very brief exposure to TF seems to suggest that dev environments surrounding TF are way harder to set up than my “list of bare packages in a requirements.txt file” scenario which is sufficient for a lot of more traditional endeavors.
Also if you want to link directly to a git repo, you can only install it in editable mode, you can't just install it with the rest of your packages (means you get a `src` directory where you ran pip, which makes Dockerizing slightly annoying, and probably impacts performance slightly).
Maybe this has also been fixed since I last upgraded.
I imagine if you're deploying docker, you probably should be developing in docker (e.g. using PyCharm's remote interpreter/docker interpreter integration).
But in any case, it just becomes one more version of Python to consider. If you're already dealing with multiple versions, what difference does it make?
If you have root privileges and you run "sudo pip" commands you might accidentally break the specific Python dependencies that the system scripts rely on. See https://developers.redhat.com/blog/2018/11/14/python-in-rhel...
There's no issue with using the system Python, but any Python packages should be installed via yum or similar Red Hat / Fedora tools and not pip.
Note that the newer versions of RHEL have created developer-specific tool packages to separate the system packages from developer packages. This allows the developer packages to get upgraded quickly so developers have nee, shiny tools without breaking the compatibility that the base system needs to keep running.
For individual people perhaps. For companies with older projects, newer projects, greenfield stuff, etc, no.
For example I've had 3.7 on my macOS system for a while, installed from the official installer, not through Homebrew. I just installed 3.8, which pointed my python3 to 3.8. But my 3.7 was still there; I created an alias python3.7 to point to that. So I can run or install anything I want against 3.7 or 3.8.
Why do we still need pyenv? I'm not asking that antagonistically, I really don't understand at this point and I'm wondering if I'm missing something.
I like being able to specify the global and local versions for my projects and the system as a whole. I also use it as a virtualenv manager. It works well with pipenv (which I still use in anger) and vscode.
I think this is a reasonable choice, considering that the main purpose of nixpkgs is to provide packages for the NixOS distribution. It's impossible to actively maintain every single version of every package.
Imagine you want to check out the hypothetical language Jagoust. This goes something like so - you install Jagoust and maybe some Jagoust tools. You fire up your interactive environment and set it up so it knows where Jagoust and its tools are. Using your enviornment you do things like tell Jagoust where your project is or which particular version of Jagoust to use, etc.. And that's pretty much it and while the details are different, the process is quite familiar to you since it just leverages your standard interactive environment in ways mechanically similar to Rugoja, another language you know.
Later you decide you want to try the language Snek. The first thing you notice in the Snek tutorial is that your standard interactive environment is somehow not good enough - you need a meta-environment for it and your project. Perplexed, you google some more docs and ask some Snek people and find out that meta-environment used to be how things were done but these days, you probably want a quasi- or para-environment. But why do you need an environment for your environment? You do not know. Such are the mysteries of the Snek.
Specifically, pyenv will dynamically and programmatically link the $PATH for the Python executables (python, pip, etc.) to the desired version as defined either by an environment variable or by the contents of a .python-version file in CWD or an ancestor thereof. The .python-version file can be checked into version control.
Noone forces people to use virtualenvs with "snek"/Python. They can always install python somewhere and "pip install" packages into the installation hierarchy, switching around different hierarchies with bash-scripts (and now have a look what all the other interactive environments do...).
Closest I found was to pin the nix-channel entirely, but not different components to different versions.
let pinnedPkgs = import (builtins.fetchTarball {...}) {};
node = pinnedPkgs.nodejs-10_x;
I've mostly stuck to nix-shell -p foo bar baz to setup my environments. It's holding me back and I should pull the trigger on a shell.nix file, just gotta learn the language.