Boring Python: Code quality(b-list.org) |
Boring Python: Code quality(b-list.org) |
https://github.com/charliermarsh/ruff
It’s literally 100 times faster, with comparable coverage to Flake8 plus dozens of plugins, automatic fixes, and very active development.
relevant section from my pyproject.toml
[tool.ruff]
line-length = 88
# pyflakes, pycodestyle, isort
select = ["F", "E", "W", "I001"]What do you mean by this? Are you indenting Python with tabs?
Though their `v0.0.X` versioning is very funny to me (https://0ver.org/).
replaced both flake8 and isort across all my projects
I would nitpick this. You build images not containers and since files are not copied by default there is more nuance here that the .dockerignore file makes builds faster by not including them in the build context.
That does ultimately prevent COPY directives from using them but it is these sorts of brief, slightly inaccurate summaries that mislead folks as they build understanding.
> slightly inaccurate Not entirely, I'm not sure the author even wanted to stress on this in the article. People won't learn docker from a python article about the same.
I absolutely let Black change code and see the value in Black that it does that so the devs do not have to spend time on manually formatting code.
Black shouldn't break anything (and hasn't broken anything for me in the years I used it) but in the unlikely case it does it, there's still pytests/unittests after that that should catch problems...
Also, mypy has gotten really good in recent years and I can vouch that on projects that have typing I catch bugs much much sooner. Previously I would only catch bugs when unit testing, now they are much more commonly type errors.
The other thing typing does is allow for refactoring code. If anything, high code quality relates to the ability to refactor code confidently and typing helps this. Therefore I would put it at the top of the list above all the tooling presented (exception I agree with ci/cd)
This is an odd complaint. typing.Sequence[T] has been there since the first iteration of typing (3.5), for exactly that use case, along with many related collection types.
https://docs.python.org/3/library/typing.html
mypy isn’t perfect, but it’s sure better than making things up without any checks; you’re going to want it for all but the smallest projects.
Dynamically typed code is 1/3rd the size of statically typed code, that means that one developer who is using dynamic typing is equivalent to 3 developers using statically typed code via MyPy.
Since the code is 1/3rd of the size it contains 1/3rd of the bugs.
This is confirmed by all the studies that have been done on the topic.
If you use a static type checking with Python, you have increased your development time by 3 and your bug count by 3.
Static typing's advantage is that the code runs a lot faster but that's only true if the language itself is statically typed. So with Python you have just screwed up.
This is absolutely not true.
> Since the code is 1/3rd of the size it contains 1/3rd of the bugs.
That is made up and contrary to all empirical evidence I've ever collected.
I'd be curious if you have a source, but I doubt it.
You should use it where it makes sense, and not where it doesn’t. I haven’t used any of Ruby’s type checkers, but Python makes this easy enough; make what has a reason to be dynamic dynamic, and have static safety rails everywhere else.
(This is true with many “statically typed” languages that have dynamic escape hatches, too, not just traditionally “scripting” languages.)
Still it's a good low bar for testing. It's easy and rises code quality. I have very good results with coverage driving colleagues to write tests. And on code review we can discuss how to make tests more useful and robust and how to decrease number of mocks, etc.
> This is the first in hopefully a series of posts I intend to write about how to build/manage/deploy/etc. Python applications in as boring a way as possible.
It's a riff on Boring Technology, see https://boringtechnology.club/
Terrible advice not to use type hints and this reason makes no sense. There's already pretty good support for Sequence and Iterable and so on, and if you run into a place where you really can't write down the types (e.g. kwargs, which a lot of Python programmers abuse), then you can use Any.
Blows my mind how allergic Python programmers are to static typing despite the huge and obvious benefits.
It's true that Python's static typing does suck balls compared to most languages, but they're still a gazillion times better than nothing, and most of the reason they suck so much is that so many Python developers don't use them!
Black formats things differently depending on the version. So a project with 2 developers, one running arch and one running ubuntu, will get formatted back and forth.
isort's completely random… For example the latest version I tried decided to alphabetically sort all the imports, regardless if they are part of standard library or 3rd party. This is a big change of behaviour from what it was doing before.
All those big changes introduce commits that make git bisect generally slower. Which might be awful if you also have some C code to recompile at every step of bisecting.
https://github.com/cjolowicz/cookiecutter-hypermodern-python
I would go so far as to say that the hypermodern template, nomenclature aside, is strictly better than the recommendations that the OP put forward both here and in the previous essay on dependency management. Poetry and ruff, for instance, are both very good tools — and I can understand _not_ recommending them for one reason or another but to not even mention them strikes me as worrisome.
My concern is a) It needs to be reliable (don't wanna spend a ton of time chasing bugs later on) b) How can I write the actual code better? I see what pro devs write and they use smarter language features or better organization of the code itself that makes it faster and reliable, I wish I could learn that explicitly somewhere.
I mean, just the 2.7->3.0 jump was big for me because since I don't code regularly that meant googling errors a lot basically. Even now, I dread new python versions because some dependency would start using those features and that means I have to use venv to get that small script to work and then figure out how to troubleshoot bugs in that other lib's code with the new feature so I can do a PR for them.
I love python but this is exactly why I prioritize languages that don't churn out new drastic features quickly. Those are just not suitable for people whose day job is not coding and migrating to new versions, supporting code bases, messing with build systems, unit tests, qa,ci,etc... coding is a tool for me, not the centerpiece of all I do. But python is still great despite all that.
About typings: I agree the eco-system is not mature enough, especially for some frameworks such as Django, but the effort is still valuable and in many cases the static analysis provided by mypy is more useful than not using it at all. So I would suggest to try do your best to make it work.
When python converges on consistent typing across its extended numpy and pandas ecosystem, I believe we will be able to move towards a fully JIT'd language.
Unless they actually go ahead with the deferred evaluation of types (PEP 563), make all types strings at runtime and make it impossible to know which type they actually are. :)
But they will probably not: https://discuss.python.org/t/type-annotations-pep-649-and-pe...
But it could be a breaking change in the language. As it is, I can run this "a: str = 3" and it will work.
On Ubuntu and Windows I use Poetry [0], and it works, although it has (had?) some quirks during the installation on Windows. I liked its portability and lockfile format though.
A few years ago I used conda [1], which was nice because it came batteries included especially for Deep Learning stuff. I switched because it felt way to heavy for porting scripts and small applications to constrained devices like a Raspberry Pi.
And then there are also Docker Images, which I use if I want to give an application to somebody that "just works".
What's your method of choice?
Even while it won’t break anything you want CI to be your safety net, flagging a local setup as being wrong is more valuable than magically autocorrecting it.
CI/CD has no business changing your code; it builds stuff using it, exactly as if commit such-and-such.
I liked black, though I was never satisfied with the fact that there was no way to normalize quotes to be single quotes: '. Shift keys are hard on your hands, so avoiding " makes a lot of sense to me. But there's the -S option that simply doesn't normalize quotes so it has never been a real issue.
However, this new project has a lot of typer functions with fairly long parameter lists (which correspond to command line arguments so they can't be broken up).
black reformats these into these weird blocks of uneven code that are very hard to read, particularly if you have comments.
Everyone is a fan of black; no one liked the result. :-/
I have a key in my editor to blacken individual files, but we don't have it as part of our CI. Perhaps next project again.
100% this. I also let Black auto-format code in the CI and commit these formats.
A lot of developers, intentionally or not, don't have commit hooks properly setup. If Black doesn't change the code in CI they need to spend another cycle manually fixing the issues that Black could have just fixed for them.
You're saying that there's a risk that Black could break your code when formatting? Well, so could developers and I'd trust a machine to be less error-prone.
There is nothing more frustrating than coming back from a coffee break only to find out that you have to rerun your CI check because of a trivial formatting issue.
Let black format code before it is checked in. Code should not be reformatted for CI or production, and bad formatting should either ALWAYS throw errors (no known defects allowed) or NEVER throw errors (if it passes tests & runs ship it). Consistency is the key.
There's zero harm in using list in private interfaces: I know I'm the only one passing the value, I know it is always a list.
As an argument type, Iterable is compatible with list, so it's benefits are minimal (with rare exceptions).
Lists are easier to inspect in a debugging session.
Iterable can be useful as return type, because it limits the interface.
Iterable is useful if you are actually making use of generators because of memory implications, but in this case you already know to use it, because your interfaces are incompatible with lists.
I can count on fingers of my hands when using Iterable instead of list actually made a difference.
Iterable is not compatible with list, but list is compatible with iterable. As the more general type, Iterable is better as an argument type unless you have a reason to force consumers to use lists. Even in private interfaces, I tend to prefer it, because I often end up wanting to pass something constructed on the fly, and creating an extra list for that rather than using a genexp just seems wasteful.
`list` might be but `List` isn't. Are you not defining the type of the contents of the list?
No. What allows you confident refactoring code are automated tests. I honestly can't understand why people are so obsessed about types, especially in languages like Python or Javascript.
By depending on interfaces/abstractions instead of specific cases you can refactor the interface and not break clients. It's very difficult to do this unless you have types.
This is something that Go is really good at and encourages but can be done with python/js on top of their type systems.
Types in Python feel like an added layer of confidence that my code is structured the way I expect it to be. PyCharm frequently catches incorrect argument types and other mistakes I've made while coding that would likely result in more time spent debugging. If you don't use any tools that leverage types you won't see any benefit.
It's a very powerful sanity check that lets me write correct code faster, avoiding stupid bugs that the unit tests will also, eventually, find.
And, to me, reading the code is much much nicer. Types provide additional context to what's going on, at first glance, so I don't have to try to guess what something is, based on its name:
results: list[SomeAPIResult] = some_api.get_results()
Typing facilitates automated testing; e.g., hypothesis can infer test strategies for type-annotated code.
[0]: https://github.com/agronholm/typeguard/
[1]: https://typeguard.readthedocs.io/en/latest/userguide.html#us...
These statements contradict themselves? List is too specific, and Sequence[item] is preferred. Sometimes you are dealing with a tuple, or a generator, and so it makes more sense to annotate that it is a generic iterable versus a concrete list.
> For example, you basically never care whether something is exactly of type list, you care about things like whether you can iterate over it or index into it. Yet the Python type-annotation ecosystem was strongly oriented around nominal typing (i.e., caring that something is exactly a list) from the beginning.
I'm saying that this quote is a straw man and that contrary to what is claimed in the quote, instead, the ecosystem would go with/recommend Iterable[Item] or Sequence[Item] and not List[Item] if applicable.
I think we both agree, not sure which part of my comment you think is contradictory.
As an argument type, Iterable is permissive (generic).
As a return type, Iterable is restrictive (specific).
Depending on the language and the particular project, my sweet spot for test coverage is between 30-70%, testing the tricky bits.
I've seen 100% code coverage with tests for all the getters and setters. These tests were not only 100% useless, they actively hindered any changes to the system.
You can have bad unittests which make the system worse and you would be better of without them. You can also have useless unittests with 100% coverage, which is pretty much the same as bad tests because more code means more bugs and more work. Unittests are also code after all.
The only thing you can say about a very low coverage is that you probably don't have good tests. That's not a very useful metric, since you likely already know that.
The metric 'coverage' is almost useless. Code coverage starts to be useful once you let go of it as a goal and ignore the total percentage number. I found it is very useful though if you can generate detailed reports on each line of code or better yet, each branch in the code, indicating whether that line or branch is tested. Eyeball all the lines which don't have tests and ask yourself: would it be useful to add a test exercising this codepath? How do I make sure it works and what cases can I think of that could go wrong? This doesn't automatically lead to good tests, but it helps you spot where you should focus your testing efforts.
Code coverage is a good tool to help think of test cases, as a metric for the total codebase it is nearly useless.
It's a red flag to blame high coverage for fragile tests. Use narrow public component interfaces to reach code parts and you simultaneously gain robust tests which can be used during refactoring and you can be guided by coverage to generate test cases. Bob Martin has a great article: https://blog.cleancoder.com/uncle-bob/2017/10/03/TestContrav...
Absolutely not. This leads to testing being invasive and driving the design of your software, usually at the cost of something else (like readability). Testing is a tool, you can't let it turn into a goal.
> Testing is a tool, you can't let it turn into a goal.
Yep, and I use testing as a tool to be sure we ship quality code. It's 2x important for our case, we don't have control on hosts where our product is run and 100% coverage was a salvation. We even start to ship new versions without any manual QA.
Then add black as part of your environment with an specific version...
Reformatting the whole code every version isn't so good. It's also very slow.
The further you get away from the project folder the more likely each developer is to have a different environment.
Why? It is expected for the thing to run on different python versions and different setups… what's the point of forcing developers to a uniformity that will not exist?
It's actually better to NOT have this uniformity, so issues can get fixed before the end users complain about them.
Any team of developers who aren't using the exact same environment are going to run into conflicts.
At the very least, there must be a CI job that runs quality gates in a single environment in a PR and refuses to merge until the code is correct. The simplest way is to just fail the build if the job results in modified code, which leaves it to the dev to "get things right". Or you could have the job do the rewriting for simplicity. Just assuming the devs did things the right way before shipping their code is literally problems waiting to happen.
To avoid CI being a bottleneck, the devs should be developing using the same environment as the CI qualify gates (or just running them locally before pushing) with the same environment. The two simple ways to do this are a Docker image or a VM. People who hate that ("kids today and their Docker! get off my lawn!!") could theoretically use pyenv or poetry to install exact versions of all the Python stuff, but different system deps would still lead to problems.
You've never done any open source development I guess?
Do you think all the kernel developers run the same distribution, the same IDE, the same compiler version? LOL.
Same applies for most open source projects.
Bisection search is log2(n) so doubling the number of commits should only add one more bisection step, yes?
> Which might be awful if you also have some C code to recompile at every step of bisecting.
That reminds me, I've got to try out ccache (https://ccache.dev/ ) for my project. My full compile is one minute, but the three files that take longest to compile rarely change.
And testing 1 extra step could only add a 1 hour build more, yes?
This is not isort! isort has never done that. And it has a formatting guarantee across the major versions that it actively tests against projects online that use it on every single commit to the repository: https://pycqa.github.io/isort/docs/major_releases/release_po...
You should never develop using the system Python interpreter. I recommend pyenv [0] to manage the installed interpreters, with a virtual environment for the actual dependencies.
Yes yes… never ever make the software run in a realistic scenario! You might end up finding some bugs and that would be bad! (I'm being sarcastic)
use pre-commit https://pre-commit.com/ so that everyone is on the same version for commits.
Not using a formatter at all is clearly worse than either option.
why?
Do you hate terse diffs in git?
What do you mean by "drastic" features "quickly"? Python releases new version once a year these days, and upgrading our Django-based source code with 150 dependencies from 3.4 to 3.11 literally meant switching out the python version in our CI configuration and README.rst every once in a while, no code changes were necessary for any of those jumps...
Our developer README also contains a guide how to set-up and use pyenv and it's virtualenv plugin which makes installing new python versions and managing virtualenvs easy, just pyenv install, pyenv virtualenv, pyenv local, and your shell automatically uses the correct virtualenv whenever you're anywhere inside your project folder...
jumping to python3 was big, but you had plenty of time to prepare for that and plenty of good utilities to make the jump easier (2to3, six, ...). python2.7 itself was released 18 months after python3.0, and by the time python2.7's support ended, python3.8 was already out...
Second, yes, all you have to do is switch out the python version to upgrade but let's say you start using f-strings that means all of your users (doesn't apply to django since it is server software) have to upgrade to the right python version including all the deps. But what if your project is a library? That means all other libraries need to use the same or greater python version but what if your distro doesn't yet support the very latesr python version? It's such a nightmare.
New versions should come out no more often than every 3-4 years imho and even then every effort should be made to have those features backward compatible like have a tool that will degrade scripts to be usable on a previous language version.
If a dependency breaks compatibility with earlier Python versions because the author wants to use a fancy new feature is not really the fault of Python, is it? Library authors should target the earliest supported Python version they can.
Being backwards compatible (at which Python has been doing a good job since the 2->3 fiasco) is one thing, but trying to be forwards compatible is something else.
Are you suggesting that Python developers should only ship bug fixes so that Python 3.0 can still run code written for Python 3.11?
In 3.8 someone decided that they didn't like the way people were excepting the Exception for cancelled asycnio tasks. So they changed the cancelled task exception to inherit from base exception instead of exception. This meant a bunch of well used libraries immediately had a load of subtle bugs that in normal operation just didn't happen. I can't remember the exact details but I think when the bug did happen the task queue would just continue to grow until we ran out of memory.
This change wasn't a bug fix, more an optimization or an attempt to get people to code a certain way.
I'm all in favour of bug fixes, but Devs shouldn't have to worry about minor upgrades breaking everything.
I have a library… most downloaded version is 3 years old. The newer versions are massively faster but nobody uses them.
Agreed. I like docker images for smallish portable scripts. At home I can develop on my Mac and port it to a Raspberry PI or another x86 Windows/Linux box.
Planning on running a docker swarm with a few Pi’s to see how it works.
It's far from perfect, but it helps if you don't know any better. And most people don't know any better.
I'm currently spinning up a new project (React Native, Typescript) and I'm spending a lot of effort in locking down the project - eslint, unit tests & coverage, CI, strict typescript rules, etc - because this did not happen with the previous iteration of this project, leading to tens of thousands of LOC worth of unit- and end-to-end integration tests to become worthless and unusable. Sure, that was a lack of developer discipline as well, but why rely on other people when you can do it through technology as well? You can't control everyone.
https://docs.python.org/3/library/typing.html#typing.List
> Deprecated since version 3.9: builtins.list now supports subscripting ([]). See PEP 585 and Generic Alias Type.
If your goal is 100% coverage then it will turn testing into ritual and only give you the illusion of quality. Instead of testing inputs and edge cases, you will focus on testing lines of code.
There's a good illustration of uselessness of 100% coverage in one of Raymond Hettinger talks: https://www.youtube.com/watch?v=ARKbfWk4Xyw
> I use testing as a tool to be sure we ship quality code
I suspect we have different definitions of quality, and your might include testing, so I doubt I will be able to convince you.
When a measure becomes a target, it ceases to be a good measure.
It takes immense discipline to actually let go of a metric to keep it valuable.
Set black up in the pre-commit with a specific version. When you make a commit it will black the files being committed using the specific version of black. As it's a subset, it's fast. As it's a specific version, it's not going back and forth.
I hope this solves your issues.
The authors of black just don't understand that it'd be ok to introduce new rules to format new syntax, but it isn't ok to just change how previous things work.
And if they are not, then maintainers can pull, run black over the diff, and commit.
CI prevents poorly formatted code from entering main.
The actual changes between black versions of late have been minor at best. You’re making a mountain out of a molehill.
Having a tool that dictates formatting is a lot less oppressive to new developers than 100 comments nitpicking style choices.
2.7 was supported for 10 years and it's support ended 2 years ago. There's been ample time to upgrade the code or look for an alternative. If I "absolutely needed" to use a piece of code that I didn't write, is for an unsupported platform and is itself unsupported, I'd absolutely find the time for it. As a developer if I use a library that hasn't been touched for 3 years it's a red flag and I start to look for alternative libraries or forking the code.
> That means all other libraries need to use the same or greater python version but what if your distro doesn't yet support the very latesr python version? It's such a nightmare.
if your distro doesn't support the latest python version you're probably on a very old distro. For example python3.11 installs fine on all supported versions of Ubuntu (18.04+) and Debian (10+) and both Windows (8.1+) and macOS (10.15+). And python3.9 installs fine even on centos7 (released in 2014) and still supports the vast majority of python libraries.
If you're on an OS nearing or past its end of support, you can't reasonably expect all the latest software to work on it. And it's usually fine to just use an older version of python / libraries until you're ready to update.
> New versions should come out no more often than every 3-4 years imho
If new versions came out every 3-4 years, that would mean they would have more drastic changes, because the smaller changes would accumulate over that duration. The longer the "new features" are out, the longer users have to upgrade their system and the longer developers can take getting used to them.
But in the end, it doesn't really matter how often a new version comes out but rather how long the old versions should be supported, right? And I think it's up to the library authors to decide how long to support older versions, not the authors of the programming language.
Like I said, my day job isn't being a dev which means time for that is rare.
Languages are not user software, they should never be deprecated. I get deprecating standard libraries bur under no context is it ok to deprecate a whole language. Yoh can freeze development and only perform security updates to the interpreter but there is no need to deprecate a language. It is a betrayal of the trusr users put into python when they invested time on it and this is exactly what I mean by avoiding rapidly changing languages. They don't care one bit that people are using their language, they treat like any other software that gets supported and discarded. C89 is still supported! People write new stuff with it. You know why? Because there is nothing to support, just parsing and compiling of a langauge. No new features need to be developed and bugs should be accepted instead of fixed. The interpreter should be available for download and use on any platform for as long as even one guy is using the language.
> you're probably on a very old distro.
Maybe, I usually go for debian but I have run into this issue and it becomes a dependency nightmare on anything that needs the old python version (in the package manager dependency resolver not in python).
> If new versions came out every 3-4 years, that would mean they would have more drastic changes
That's fine, because there would be less versions. All changes are drastic changes from the perspective of someone that is having to google random python errors to figure out what broke and how to fix it. At least it won't be a constant nightmare fixing problems made by the langauge itself in addition to the 3rd party code and your own code. The frequency of how many bugs you have causes by those 3 cause categories should in that increasing frequency. I should not have lnaguage version bugs more often than bugs in my own code.
> But in the end, it doesn't really matter how often a new version comes out but rather how long the old versions should be supported, right? And I think it's up to the library authors to decide how long to support older versions
It does matter because most devs that code for a living like to tinker with new flashy versions so each version of a library they release is that much more prone to requiring newer python versions. The less frequent python releases, the more they will use the current python features before introducing version breaks.
I really think it is a developer culture problem at the end of the day where because you are writing foss code, you don't care about the experience of those who depend on your code.
I'm not used to Python code (which is all that black touches) as being notably slow to build, nor am I used to incremental build systems for Python byte compilation.
And I expect in a project with 2 developers which is big enough for things to be slow, then most of the files will be unchanged semantically speaking, only swapping back and forth between two syntactically re-blackened representations, so wouldn't an caching build system be able to cache both forms?
(NB: I said "caching build system" because an incremental build system which expects time linear order, wouldn't be that helpful in bisection, which jumps back-and-forth through the commits.)
The funny thing is that if you run the versions backwards you will NOT obtain identical files with what you started with.
I think the sane part of the software engineering world has realised that auto-formatting is just the right way to do it, and the people that disagree just haven't figured out that they're wrong yet.
Maybe you meant "why is Black specifically better than no autoformatting, given that it isn't perfectly stable across versions?" in which case the answer is:
a) In practice it is very stable. Minor changes are easily worth the benefits.
b) They have a stability guarantee of one calendar year which seems reasonable: https://black.readthedocs.io/en/stable/the_black_code_style/...
c) You can pin the version!!
This is unnecessarily confrontational. Please read my other comments where I consider the extra effort that automatic formatting causes for code reviews.
> In practice it is very stable.
It has never happened to me to upgrade black and have it not change opinion about black formatted code.
> Minor changes are easily worth the benefits.
It doesn't matter how minor they are. A 1 bit difference is still going to fail my CI.
> You can pin the version!!
I usually do, but working with old releases that must be maintained, mean that I can't cherry pick bug fixes from one branch to the other, because black fails my CI.
Why would it cause extra effort? Not having automatic formatting causes extra effort because you have to tell people to fix their formatting!
> It has never happened to me to upgrade black and have it not change opinion about black formatted code.
I'm sure small things change but large differences? No way. Even the differences between YAPF and Black aren't that big in most cases.
> It doesn't matter how minor they are. A 1 bit difference is still going to fail my CI.
Right but you have a pre-push hook to format the code using the same version of Black as is used in CI. Then CI won't ever fail.
> I can't cherry pick bug fixes from one branch to the other, because black fails my CI.
Cherry pick, then run Black. Sounds like you have a very awkward workflow to be honest.
1) you don't have one black commit for every non-black commit, do you? Because the general best practice is to do like kuu suggested and have a specific black version as part of the development environment, with a pre-commit hook to ensure no random formatting gets introduced.
2) assuming 500 commits in your bisection, that's, what, about 9 compilations you'll need to do, so it will take you 9 hours to run. So even with a black commit after every human commit, that yes, 1 hour more, but it's also only 11% longer.
Even with only 10 non-black commits and 10 black commits, your average bisect time will only increase from 3.6 hours to 4.6 hours, or 30% longer.
I'm curious to know what project you are on with 1 hour build times and the regular need for hours-long bisection search, but where there isn't a common Python dev environment with a specific black version. Are you using ccache and/or distributed builds? If not, why isn't there a clear economic justification for improving your build environment? I mean, I assume developers need to build and test before commit, which means each commit is already taking an hour. Isn't that a waste of expensive developer time?
And, I assume it's not the black formatting changes which result in hour-long builds. If they do, could you explain how?
Infact, you could just try it out for yourself.
But here is your internet source for this blatantly obvious fact: https://games.greggman.com/game/dynamic-typing-static-typing...
Sometimes I still sneak in minor changes after I opened a PR. Sometimes I open PR early because CI has integration/e2e tests that are hard to run locally. Sometimes I want feedback on certain parts early on and PR is the easiest way to show something.
There can be workflows in which pushing through CI works fine, but as a general advice it's not great because there are many edge cases.
That going too far unless you define code to be a subset of the files checked into the repository and simply define any file that's touched in an automated manner to be not code
There are a lot of useful automations that can be part of the CI/CD pipeline, such as increasing a version number, generating a changelog, creating new deployment configuration etc
They don't have to be part of it and it's possible to work around it/don't commit... But that comes with it's own challenges and issues
It's mostly about the flow of data and control: source files, beside some known auto-generated files / single lines, are the source, and whatever is generated is downstream from them, not altering them. It's like a React app: data flows through props in only one direction, you don't patch DOM in event handlers, or something.
There are a lot of people which include their helm package as part of their project repository and even more that generate the changelog from standardized git messages like conventional commits and still wish to persist them in a changelog.md inside of the repository to make completely banal examples spelled out to the letter. It works well and lets you scale these things pretty well inside of a corporation with a lot of teams.
Its completely fine to keep all that out of your automated pipeline, triggering only after these things have occured... but that's not the only viable choice a developer can make and you're very much talking from of an extremely limited perspective if thats your point of view.
As for build times, it was an extreme example. But even an extra step taking 5 extra minutes is very annoying to me…
That's not been my experience. To the contrary, having a requirements.txt means your contributors are more likely to have a working environment, as when your package depends on package X but the contributor has a 5-year-old buggy version of X, doesn't realize it, and it causes your program to do the wrong thing.
In any case, your argument only makes sense if no one on the project uses black or other code formatter. Even if you alone use it, odds are good that most of your collaborator's commits will need to be reformatted.
> .. an extra step taking 5 extra minutes ...
How do black reformatting changes cause an extra 5 minutes? What Python code base with only a couple of contributors and no need for a requirements.txt takes 5+ minutes to byte-compile and package the Python code, and why?
Adding 5 minutes to you build means your bisections are taking at least an hour, so it seems like focusing on black changes is the wrong place to look.
python -m venv venv
pip install -r requirements.txt
> But here is your internet source for this blatantly obvious fact: https://games.greggman.com/game/dynamic-typing-static-typing...
Ah no I meant a proper peer reviewed source. The claim that untyped code has fewer bugs is completely bonkers, so I was quite sure that no such source existed.
Why do you think microsoft, google and facebook are all in the business of typechecking python? If typechecking would actually introduce bugs, it'd be better not doing it right?
Using github for statistics is flawed. There are millions of 10 line js libraries. Yes it's easy to not make type mistakes in 10 lines. I suppose that type errors increase more than linearly with size.
Not really. It is, however, quite expensive to measure, because dynamic typing really shines at the evolution of software, that is being able to respond fast to changing requirements. Legos vs play-doh: https://weblog.jamisbuck.org/2008/11/9/legos-play-doh-and-pr...
> Why do you think microsoft, google and facebook are all in the business of typechecking
A billion flies can't be wrong? Companies with unlimited amount of money are not the right place to search for good practices. Both Facebook and Google became flush with cash way before modern type obsession. Sure, once you are a multi-billion dollar company slowing down can be a good thing. But you need to get there first.
> If typechecking would actually introduce bugs, it'd be better not doing it right?
If sugar caused us to die sooner, we'd be better to eating too much if it, right? And yet, here we are.
There are plenty of academic sources that will tell you that the number of bugs in a program is directly proportional to the number of lines in the program and static typing has no effect on this.
https://stackoverflow.com/questions/2898571/basis-for-claim-...
Additionally, statically typed code involves large amounts of boilerplate code in the form of abstract base classes, interfaces, generics, templating, etc. It's a very verbose code style.
It's your turn, find an academic source to backup your claim that static typing reduces the number of bugs. Cause it just isn't true.
Microsoft, google and Facebook have a lot of programmers coming from languages with static typing and want to make Python more familiar.
It's a far distance away from anything resembling good practice.
Actual Python houses typically don't use static typing.
Correct if you misapply a tool to the wrong situation you get poor or negative results.
The right tools are unit testing, integration testing, uat and automated whole system testing.
> Iterable is better as an argument type unless you have a reason to force consumers to use lists
See, I feel the exact opposite: I use Iterable only if I have a reason to force consumers to use Iterable.
When you're marking argument as Iterable, how confident do you feel that you will never query collection size or access it by index?
I understand the desire to limit the interface and YAGNI, but since lists are more familiar and ubiquitous, using Iterable feels more complicated and unnecessarily pedantic.
A broad argument type doesn’t force consumers not to use a narrower type. (It forces the implementer of the function to not rely on additional features of the narrower type, but if I am writing the function, I can be certain whether or not that is acceptable.)
Meanwhile, using a narrower type than needed for an argument does impose additional, unnecessary constraints on the consumer.
> When you're marking argument as Iterable, how confident do you feel that you will never query collection size or access it by index?
Absolute certainty, since I know what the function does and what I need to do it.
> I understand the desire to limit the interface and YAGNI, but since lists are more familiar and ubiquitous, using Iterable feels more complicated and unnecessarily pedantic.
Since all lists are Iterables but not all Iterables are lists, Iterables are necessarily more ubiquitous than lists.
Yeah, that's what I meant by being pedantic :)
Here's a question: you receive a JSON payload that contains a list. You will then pass this list to two functions, one of them only iterates, another one uses list interface (let's say checks length among other things). Should you mark the argument as a list, or as an Iterable in the first function?
Solely from the code perspective, it's definitely an Iterable. But in my mental model it still remains a list. I don't like it when code deviates from my mental model. Forcibly treating it as an Iterable only makes it more complicated, while not giving anything in return.
Sure, you could say that callee should not have expectations of the caller, but what if those functions are already coupled? They are in the same module, and argument names clearly denote a collection. The fact that in certain scenarios it is "technically Iterable" serves nothing but pedantic value.
It's probably just a bad example, but in case it isn't:
Sounds like you ended up at the same place. You went from guessing what is some_api.get_results(), based on it's name, to guessing what is SomeAPIResult, also based on it's name.
If some_api is your library, then you could have just added type hints to get_results() and let type inference do it's job.
If it's a third party library, then using your custom SomeAPIResult means that code is becoming alien to other engineers that worked with that library in the past. It might be worth it, but it's definitely controversial. You probably should've done it with stubs anyway.
I disagree. It’s not a guess, it is precisely what it is, where the variable name is free to betray me. A sane IDE/linter will tell me if my local assumption is incorrect, where a variable called result_SomeAPIResult relies on an assumed, possibly ancient, state of reality.
list[SomeAPIResult] in your example is redundant. You can get all the benefits of types without it.
If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
In Python, the easiest way to achieve this is using Poetry, which creates a lock file so that all developers are using a consistent set of versions. In other languages, this is generally the default configuration of the standard package manager.
With all due respect, I don't think you're correct.
Source: I maintain distro packages.
Yes, it would work very well if said tool didn't change its mind every 6 months, generating huge commits at every bump
> Most people contributing to open source would be familiar with at least some of these methods and if they are not it’s a good opportunity for learning.
You seem unfamiliar with the fact that other people aren't necessarily clones of yourself and might not behave like you.
> CI prevents poorly formatted code from entering main.
If you run black on CI… which of course I don't since every time they make a new version I'd have the CI start failing.
And no pinning is not a "solution"… it's at best a workaround for badly written software.
> The actual changes between black versions of late have been minor at best. You’re making a mountain out of a molehill.
If you have 10 lines of code, I guess your diff can't be more than 10 lines. If you have more than 10 lines…
from application.module.modules import (
Model1, Model2, Model3, Model4, Model5, Model6
)
Two or more leading spaces on each line.[1]
As I said… because a 1 word change easily ends up changing multiple lines and from the diff it's not clear it's just a 1 word change. So… extra effort.
> Right but you have a pre-push hook to format the code using the same version of Black as is used in CI. Then CI won't ever fail.
No I don't. My release branch and old-release branch use different versions. Such a thing would need to understand which base branch I'm working on, or recreate the venv continuously.
> Cherry pick, then run Black. Sounds like you have a very awkward workflow to be honest.
Seems to me you don't support older versions, in which case everything is easy.
I now use version 5.6.4, from 4.3.4. In the end we passed a flag to keep the old behaviour, but in my mind behaviours shouldn't just change.
But it’s totally reasonable to pin the private requirements that you develop it against (listed in requirements.txt, poetry.lock, or similar), updating them every so often during the course of development, so that contributors can use a consistent set of tools.
> For your dependency/versioning issue, use a virtualenv per-project and pin your dependency versions in requirements.txt
requirements.txt is not uploaded to PyPI and has no effect on your package’s dependencies when a user installs it (leaf package or no). It’s only used for developing the package itself, typically in a unique virtual environment.
Instead, only use indentation.
It is interesting to note that Black-formatted code uses indentation only and never uses alignment. It would be perfectly compatible with tabs, unlike Google-formatted code which relies heavily on column alignment.
There can be changes that are different than function signature changes.
Where dependency pinning is the norm, there is a culture of breaking API compatibility. And you might not have a compiler error to inform you that the API has changed. Sometimes all you have is a commit message.
Relying on type inference isn’t some rule. Your can find many projects that use it selectively, being explicit where it makes sense. The point of writing code is to make it readable and maintainable. The explicit type isn’t redundant, it’s explicit in presentation, and can be functional, like my example.
I mean, just look at this example. You know the type without having to dig in, do you not? You don’t have to look at the function definition. You know immediately. That’s the point of being explicit, where it makes sense. No guessing, where it makes sense. This is why we have all these type hints now, in a dynamic language: because guessing sucks.
A list is an iterable with special additional features, so this is no conflict at all.
> I don't like it when code deviates from my mental model
But how is there a deviation; being an Iterable is part of being a list, not a deviation from it.
> Forcibly treating it as an Iterable only makes it more complicated, while not giving anything in return.
How is there anything “forcible”. Broader typing doesn’t “forcibly” impose anything. And it does give something, more freedom to callers.
> Sure, you could say that callee should not have expectations of the caller, but what if those functions are already coupled?
If there is coupling that exists for good cause and demands a list as the type of the data structure to be passes around, then, fine, use list. But usually Iterable or Sequence makes more sense; coding to interfaces which impose only what is actually required is better than to unnecessarily specific concrete types.
Because I'm a human, not a robot. If I can describe something with fewer words by sacrificing a little bit of accuracy, I might go for it when I don't deem that accuracy to be important.
If wife sends you to buy eggs, are you the type of person that buys caviar because "technically they're eggs"?
> coding to interfaces which impose only what is actually required is better than to unnecessarily specific concrete types
Of course. Although practicality beats purity.
Did you even read my comments?
Black reformatting causes more steps in bisecting. It's quite easy that a test suite takes 5+ minutes.
When I've used bisection, I've always had a targeted test that I was trying to fail, not the entire test suite. This is because the test suite at the time of that commit wasn't good enough to detect this failure. Otherwise it would have failed with that commit.
Instead, a new failure mode is detected, an automated test developed, and that used to probe the history to identify the commit.
Why are your bisections doing the full suite?
> Black reformatting causes more steps in bisecting
Yes, of course it does. But it's log2(n).
The worst-case analysis I did assumed there was a black commit after every human commit. This is a bad practice. You should be using black as a pre-commit hook, in which case only your new collaborator's commits will cause re-formats. And once they are onboard, you can explain how to use requirements.txt in a virtualenv.
If only 10% of the commits are black reformattings, which is still high!, then a bisection of 100 human commits (plus 10 black commits) goes from about 6.64 tests to 6.78 tests, which with a 5 minute test suite takes an additional 42 seconds.
If it's 3% then your bisection time goes up by 13 seconds.
If you are so worried about 13 seconds per bisection then how much time have you spent reducing the 5 minute test suite time? I presume you run your test suite before every commit, yes? Because if not, and you're letting CI run the test suite, then you're likely introducing more commits to fix breaking tests than you would have added via black, and taking the mental task switch hit of losing then regaining focus.
I would reject such commits in review.
A human might add one or two items to a list and black might decide it's now too long, and make 1 line into 10 lines.
Now I have to manually compare the list item by item to figure out what has changed.
So I normally require formatting to be done in a separate commit, because I don't want to review the larger than necessary diffs that come out doing it within the same commit.
> Actual Python houses typically don't use static typing.
If you had ever developed python professionally, you'd know this to be untrue.
Also your "paper" points to a 404 page.
Given that type hints are a new language feature in Python, please explain to me how these Python houses used static typing when type hints didn't exist?
I'm all ears.
They've existed since 2015… please explain me how 2015 is "new"?
As does integration testing, user acceptance testing and whole system testing (QA engineers, frontend testing, etc.)
Nothing has been forgotten. Python isn't Java, nor should you develop your Python code as if it were Java. Python has it's own software development practices that take advantage of the language's strengths including dynamic typing.
It you don't know how to work with a dynamically typed language properly, that's on you. And I guarantee you will get poor results pretending it's a statically typed language.
Insulting me won't make you a better developer nor a better person :)
Getting started with type hints is not easy, but it can be done incrementally and it is worth it.
If you haven't used them, please learn before insulting people who are more experienced than you.
Homebrew, windows, arch all have very very relaxed processes to enter. There is no QA, you can just do whatever you want. I mean more like Fedora and Debian.
You've now moved on to talking about something else, which is "how much Rust software is packaged." Well, apparently enough that Debian has Rust packaging policy[1]. I'll give you one guess at what isn't mentioned in that policy. Give up? Lock files!
My mistake, seems rust packagers gave up on decent packaging. It isn't so for the python policy, I can assure you :)
A human might add one or two items to a list, decide it's now too long, and make 1 line into 10 lines.
Including the same hypothetical first contributor you mentioned earlier, who you think will find using requirements.txt as being too big a barrier to entry.
Onboarding occurs either way.
I get that you don't like using black - and that's fine! I don't use black on my project either.
But it seems like you're trying to find some other reason to reject black, and constructing hypotheticals that don't make any sense.
Just say you don't like black's choices, and leave it at that.
At which point I tell him to split formatting and actual changes into different commits (see https://mtlynch.io/code-review-love/).
> I get that you don't like using black - and that's fine! I don't use black on my project either.
Well according to this comment, it's because we are noobs: "the people that disagree just haven't figured out that they're wrong yet"
> But it seems like you're trying to find some other reason to reject black, and constructing hypotheticals that don't make any sense.
After the n-th time I have to re-setup the entire virtual env on a different branch just to re-run black and isort to backport a fix to a release branch… it does get tiring.
I presume most people here just do websites and don't really have a product that is released to customers who pay to support old versions for years, so black changing syntax is a 1 time event rather than a continuous source of daily pain.
But it seems the commentators here don't have the experience to know there might be a use case they didn't think of.
My main product is 12 years old, with paying support customers, and with bugfix branches for older releases.
> just to re-run black and isort to backport a fix to a release branch
Great! That's an excellent reason. But it has nothing to with bisection.
But sure continue to hammer in nails with a screwdriver, it's only your own time you are wasting.
And I've noticed you failed to provide any sources whatsoever for your nonsense.
As opposed to your 404 "peer reviewed paper"? :D
> Using static typing in a dynamically typed language will always make you a poor software engineer
Making broad statements about things you've never used yourself just makes you an arrogant guy on the internet who is quite likely to be a poor software engineer with a big ego.
My wages disagree with that statement. But hey, we can't all be 10x.
Did it ever occur to you to think about why people use scripting languages or what advantages they have over regular programming languages?
Of course not, you are used to statically typed languages and are blinded to the idea that there are other ways to develop software.
I'm guessing you did a search for sources to backup what you are saying and found out pretty quickly that they don't exist.
Static typing in Python is a practice based entirely in common ignorance not reality.
No shame, but you are not the most qualified person to comment on typing.
Also python3 is from 2008, and it's effectively a different programming language.
I'll go further and tell you the most common reason for using static typing is to allow the codebase to be a monolith like it's still the 90s. You shouldn't be trying to build a monolith in a scripting language it's a recipe for disaster.
I'm a polyglot, I'm exactly the sort of person who should be commenting.
Does it surprise you that static typing is often a poor choice?
Static typing is something to be used when the performance of your code is important. I've done 40 Gbits/sec network traffic processing, certainly static typing is used for that.
Your standard business CRUD app? Usually dynamic is the better choice.
The poster I was responding to was literally posting false information. I'm correcting it. This doesn't need to turn into a huge long sprawling discussion about packaging Rust programs. The main point that I was making is that lock files do not prevent Rust programs from being packaged. bombolo then went off on their own little tangents spouting nonsense without bothering to acknowledge their mistake.
> My wages disagree with that statement. But hey, we can't all be 10x.
Just a hint: this screams insecurity.
Wealth does not correlate with knowledge.
Especially since you don't even use type hints, so you have no professional experience with them. Just an ill formed opinion by reading blogs and comments.
> Did it ever occur to you to think about why people use scripting languages or what advantages they have over regular programming languages?
No compilation time? Very complete standard library? Opt-in typing? Good introspection? Numpy?
> Of course not, you are used to statically typed languages and are blinded to the idea that there are other ways to develop software.
I was doing python long before type hints existed, and I assure you they are an advantage. Now, I understand you lack the expertise to realise that. I'm just saying that insulting me won't make you correct.
> I'm guessing you did a search for sources to backup what you are saying and found out pretty quickly that they don't exist.
I did not bother. The fact that microsoft, google and facebook invest money into it is proof enough. You reject it because you're being irrational.
> Static typing in Python is a practice based entirely in common ignorance not reality.
I wish I could sound so bold and certain when being wrong!
Basically, you are saying you are a cargo cultist rather than a serious software developer.
You fail to understand that the software practices of large multi-national companies are rarely good.
Good luck flying your plane: https://www.abyssapexzine.com/2020/03/cargo-cult/
No. I'm saying I tried both ways and I know advantages and disadvantages and I'm capable of deciding by myself.
You on the other hand did not try both but feign expertise.
> Good luck flying your plane: https://www.abyssapexzine.com/2020/03/cargo-cult/
Yes everybody knows what a cargo cult is. It's not some sort of intellectual remark. More of a random thing to say on the internet when you have no real arguments.
But good to know you can find sources… when they happen to exist :)
Now try to get something using an obsolete version of some python module into Fedora or Debian and let me know how it goes… It would not be accepted as it is. It'd be patched to work with a current one or just rejected.
Just stop spreading misinformation. And the courteous thing to do is to acknowledge an error when it's pointed out instead of doubling down and redirecting as if no error was made.
> And how much rust software is packaged in distributions? Almost none.
> They haven't figured out the procedures
You're clearly talking about Rust in the second two comments. Your original comment was just a general pronouncement about lock files. You could perhaps be given the benefit of the doubt that you were only thinking about Python, but someone else interpreted your comment broadly to apply to any language with lock files. If you really only meant it to be specific Python, one would reasonably expect you to say, "Oh, sorry, I was only talking about Python. Not Rust. Their situation might be different."
But no. You doubled down and started spouting nonsense. And you continue to do so!
> Where dependency pinning is the norm, there is a culture of breaking API compatibility.
Rust does not have this problem. It is the norm to use lock files for Rust programs, but there is no culture of "breaking API compatibility" without appropriate signaling via semver.
This entire exchange is a classic example of Brandolini's law: https://en.wikipedia.org/wiki/Brandolini%27s_law
It's easy for you to go off and spout bullshit. You've even been corrected by someone else who maintains distro packages. But it's a lot harder to correct it. You wriggle and squirm and rationalize and deflect.
Maybe they just were inferior projects?
I've used a library where every function just accepted "args, *kwargs" and no documentation was given. In that case it's not really the fault of the language that it sucks. It could be a similar case for you.
> I'm a polyglot, I'm exactly the sort of person who should be commenting.
Ok I speak 3 languages fluently and 1 more so-so… But what does this have to do with python typing????
In fact most people who study literature and languages don't know much about python types.
> Does it surprise you that static typing is often a poor choice?
No because it isn't true.
> Static typing is something to be used when the performance of your code is important.
We can all agree that python isn't something to use when performances are very important.
> Your standard business CRUD app? Usually dynamic is the better choice.
Ok. That's not what I do though.
Have you done any C and C++? You know how people prefer doing a list in C++ with a template rather than a list of void in C? Same thing in python. But perhaps you haven't experience in this field either?
Polyglot as in multiple programming languages.
I'm afraid it is true. Knowing the limitations of the tools you use is important. And you clearly do not.
Yes, I've done programing expensive networking hardware in C. I think you are missing that I know a lot more than you.
Tagged values aka dynamic typing is an excellent approach to doing a list from the developer's perspective.
If you think dynamic typing is like using void pointer in C you are very much mistaken.
You ever tried to read haskell's documentation? You search functions by their type signature.
> Polyglot as in multiple programming languages.
lol ok.
> I'm afraid it is true. Knowing the limitations of the tools you use is important.
I agree. But to know the limitations one must know the tools first.
> And you clearly do not.
Having used both things gives me more knowledge than NOT having used both things.
> I've done programing expensive networking hardware in C.
I don't think expensiveness of the rig and skill correlates… are iphone developers better than android developers because iphones cost more?
> I think you are missing that I know a lot more than you.
Ah yes a "polyglot" who speaks just english. A true renaissance man.
> If you think dynamic typing is like using void pointer in C you are very much mistaken.
It's kinda-like-it, conceptually. I've written a compiler and an interpreter, I know how they work.
This is so true. Static typing shines when you have a very complicated deeply nested system. And that's what most teams are naturally end up creating.
But how about not building the complicated system in the first place? Most of complexity in modern software is accidental.
Your best arguments are: "big guys are doing it therefore it's good for me too" (argument from authority) and "I tried both and decided only one is good" (argument from authority/anecdotal evidence). Plus a ton of ad hominems.
To be fair, your opponent isn't very different.
I asked for sources and got a 404 link as a response.
I'm told that I earn less money, hence I'm not as good (despite the other person having no idea of how much i earn).
> To be fair, your opponent isn't very different.
And yet, you felt the need to respond to me…
No one who has really tried both would conclude that static typing is better hence I know you are lying. :p
I did no such thing. But after hours of you writing lies, I'm not surprised you wrote another one.
> No one who has really tried both would conclude that static typing is better hence I know you are lying. :p
In 10-15 years, when you will actually have some non-made-up programming experience, you will agree with me :)
Nah, I'm more qualified than you. You just don't want to hear that you are wrong.