Python exceptions considered an anti-pattern(sobolevn.me) |
Python exceptions considered an anti-pattern(sobolevn.me) |
Python chose untyped exceptions, period. How is this surprising, given that its basis is untyped parameters?
If you don't like that, use Java with its checked exceptions. Or remove exceptions from the implicit monad altogether and use C or Rust. Just don't then go on to write some lengthy post about how any one of those is too explicit.
The mistake I see people make is to use built-in exceptions without subclassing, making it impossible to explicitly catch specific errors. Or the opposite, catching Exception, which will catch everything indiscriminately.
The Python runtime is dynamically typed. The Python language is untyped.
> So, the sad conclusion is: all problems must be resolved individually depending on a specific usage context.
Why is that sad/revelatory? If the wrapper simply reproduces the behavior of a single infix operator for floating point numbers, then almost by definition one would need to reimplement NaN in order to get a general purpose exception handler.
> Exceptions are hard to notice
No. In Java they are easy to notice. In Python we have a dynamic language, so there is no safety net for that. It's a limit of the language that we chose, by design, with pros and cons. It's not exception related.
> So, the sad conclusion is: all problems must be resolved individually depending on a specific usage context. There’s no silver bullet to resolve all ZeroDivisionErrors once and for all
That's a good thing. There are many reasons to have a ZeroDivisionErrors, and they beg all for a different solution. Sometime it's you, sometimes it's the user, sometime it's just how things are. Again, it's not related to exceptions in any way.
> For example, the system might notify the user to change the input, because we can not divide by 0. Which is clearly not a responsibility of the divide function
Indeed, you are supposed to sanitize your inputs at the intersection between your program and the rest of the world. No surprise, this has nothing to do with exceptions.
> Now we just need to check where this exception is actually caught. By the way, how can we tell where exactly it will be handled?
Because you write the handling code. If you don't write error code handling, how is that the exception system fault ?
> There’s no way to tell which line of code will be executed after the exception is thrown.
Yes, there is. Either you handle it, and the next step is in the try/except, or you don't, in which case your program crashed. Same with any way of dealing with errors. It's not about exceptions.
> We have two independent flows in our app: regular flow that goes from top to bottom and exceptional one that goes however it wants.
Nope, the exception flow goes from the bottom of the stack to the top. It's well defined, we even have a beautiful stack trace for that. Don't blame exceptions if you don't even know the basics.
> Exceptions are not exceptional
Compared to the rest of the lines of your program, they are. Exceptions are called that way because they represent a special case. The fact computing is full of them is, you know it now, not related to exceptions.
> How to be safe?
Handle the error. Like in any language, with any error handling tool. There is a difficulty in Python: the exceptions are not listed in the function signature. That's hardly the fault of the concept of exception, and is just a design compromise.
> Now you can easily spot them! The rule is: if you see a Result it means that this function can throw an exception. And you even know its type in advance.
So basically you rewrite Java in Python. Again, not a problem with the exceptions. And oh, no, don't do that. If you want that, use Java, not Python. It's perfectly reasonable, but don't turn Python into Java. The fact exceptions are not written in the function signature is, I repeat, a design decision. It has pros. It has cons. But don't use a screw driver like a hammer, that's bad.
> ow to work with wrapped values?
Ok, now you are trying to implement Haskell in Python.
Use Haskell then.
Python is not made for this. It's an easy to write and read language. You are supposed to be able to edit Python code with notepad if you have to. Every line should be short and to the point, with a well defined role.
And we expect exceptions to bubble, that's how we like it.
> But how to unwrap values from containers?
See what I mean ? The author just opened the pandora box. Now he or she has to write long chunks of text just to explain the basic of error handling.
Without this system, I can just pdb.set_trace() in there, now it's full of inlines, chaining, anonymous callbacks and wrappers.
That's just... no.
Maybe, but there are ungodly big and crufty codebases in Python all around the world, and a lot of stuff depends on them. And it turns out that Python was never meant to be the thing that is being depended on. That's unfortunate. Hence the whole gradual typing (mypy) and other kinds of safety efforts.
You're not forced to use this. But it seems pretty useful, even for scripts.
The key thing to understand here is that post is really about an attempt to implement the "Railway Oriented Programming" paradigm [1] by Scott Wlaschin, but in Python. I would suggest reading the Scott's post on ROP and at least skimming the video before going on.
So ROP is, as Scott himself states, a way to take all those nice Haskell concepts and techniques and apply them in F# in a way that won't overwhelm those who are new to them.
The problem with the original post is that it presents two problems with exception in Python and then offers the "returns" library as a solution which, ultimately, doesn't end up solving either of those problems.
The first problem the post describes is that the exceptions are not part of the function signature. The second problem is that exceptions are essentially gotos and that this makes reasoning about the execution flow very difficulty.
To tackle both problems, the returns library offers its own implementation of the Either monad in form of the Result container. Having presented that solution, the post promptly decides that monadic code in Python is unreadable and offers the @pipeline decorator which allows you to write code that looks imperative, which partially defeats its purpose as a solution to the problem about the flow reasoning. I say partially, because it replaces implicit gotos with implicit returns, which is a marginal improvement over exceptions.
The post then decides that using the Result class in the signature is also ugly and offers the @safe decorator which allows you to write the same code you would write without ROP, but now it wraps everything in Result behind the scenes. Even worse than that, this produces the return type of Result[whatever_your_success_type_is, Exception]. For those familiar with Java, this is very similar to putting "throws Exception" in your method signature, except that it's hidden and implicit.
I'll end this with a bit of advice for the author, preceded by an apology, because I'm not sure if I have managed to find a way of phrasing it that doesn't sound harsh. You might want to do some serious reading about functional programming and monads and how these concepts have been carried over into and grafted onto mixed-paradigm languages like Java. I mention Java specifically, because one of the complaints in your post is that Python won't be supporting checked exceptions in the nearest future. A lot has been written about checked exceptions in Java -- on both sides of the discussion.
[1]: https://fsharpforfunandprofit.com/rop/ [2]: https://en.wikipedia.org/wiki/Tagged_union
"Typing failure modes, not just successful returns" or something would sound like a much more appealing article to a much wider audience.
And for one specific reasons: Exceptions are not important per se, but also for the stacktrace they take with them, and that is given to you when you need to handle it.
I lost too much time in this last month trying to decipher where the hell stuff happened looking at the stacktraces in the logs.
Coming back to the python approach: I like python :) and it's philosophy it's to have unchecked exception, with all what it concerns. The idea is that if you don't know an exception, you really don't know what to do. So let the code blows up - ideally in tests :) - and understand what realistically can happen and how to deal with it.
def update_password_view(session_cookie):
values = request_params(['old_password', 'new_password'])
user = get_user(session_cookie)
verify_password(user, values['old_password'])
update_password(user, values['new_password'])
return 200
def request_params(param_names):
values = {}
for key in param in param_names:
try:
values[key] = request.params[key]
except KeyError:
raise BadRequestError('Missing parameter', key)
return values
def get_user(session_cookie):
users = db.get_user_with_session(session_cookie)
if len(users) != 1:
raise NotFoundError('No user with that session')
return users[0]
def verify_password(user, old_password):
if hash(user.old_password) != hash(old_password):
raise BadRequestError('Bad password')
def update_password(user, new_password):
user.password = hash(new_password)
db.update_user(user)
Our actual implementations are more subtle. We have our own exception hierarchy with classes like `UserNotFoundError` or `BadPasswordError` that subclass the corresponding HTTP error classes, so you can still write code like:
def upsert_user(data):
try:
user = get_user_by_email(data.email_address)
except UserNotFoundError:
user = User(data)
db.save(user)
do_something_with(user)
In practice, we've found this coding style to be much easier maintain than idioms like `if not_found(user): return None` where you spend half your lines of code explicitly checking return values for error sentinels. Life's too short to live like that.
You could pick Julia, or R. A community could create their own tools. It's not like JS, which has an absolute monopoly and people were forced to use it.
Python __became__ popular for that purpose.
Not Java. Not Haskell.
So don't try to implement Java or Haskell error handling in Python if you are using Python in a niche where it shines: it shines here because people working in that field decided it fits well.
I'm part of the people who advocate automated testing and insanely high coverage. Does that mean I want to write all tests manually ? No, I use fuzzers, light test code against many autogenerated fixtures and the like, and get no runtime exceptions.
Duck typing means "bad language" for some people, for me it means "freedom".
It re-implements a 4 line function as a 13 line class, but the logic at the caller doesn't get any simpler:
try:
result = get_user_profile(id)
except:
# handle any exceptions...
result = FetchUserProfile(id)
if (result is a Failure):
# handle the failure
# do something with the resultWe should have language-level mechanisms for being explicit about what's supposed to happen when unexpected situations happen.
This is why unit-tests are really important in Python.
> We should have language-level mechanisms for being explicit about what's supposed to happen when unexpected situations happen.
That's implemented as a try-except block. You're not supposed to catch errors at the low level unless you're explicitly handling them. If there's an error, bubble it up to the main().
Python's try-except can catch classes, sub-classes, and/or groups (tuples) of classes. It's important to categorize your errors.
- imo it effectively requires using type checking (mypy)
- Too out of place with the rest of the python ecosystem (even if I like something, I'd rather not force non-standard practices on others dealing with my code)
try as method1:
method1()
except@method1 try as method2:
method2()
except@method2 try as method3:
method3()
except@method3:
raise NoMethodWorked() methods = [method1, method2, method3, method4]
for method in methods:
try:
method()
except Exception:
pass
else:
break
else:
raise NoMethodWorked()
methods = [(method1, TypeError), (method2, KeyError)]
for m, e in methods:
try: m()
except e: ...
try:
method1()
return
except Exception: # or a more specific exception type
pass # try the next method
try:
method2()
return
except Exception:
pass
# ...
try:
method10()
except Exception:
raise NoMethodWorked()
try:
DispatchController.dispatch()
except:
raise DispatchFailed()
A very strong and emotional point. Accidentally working code will have exceptional handling because of problems during development.
The whole article is depreciated by naive library implementation because unwrap() hides source of original exception. Sadly it's not even a POC.
Edit: Failure doesn't capture trace information at all. Library users will get unusable error in a wrong place.
...and that's a good thing.
Recognising that specific usage contexts require specific recovery strategies is a key part of effective program design.
Division by zero is an exception, yes. That's basic math.
Getting to the point where one of your inputs is "bad" (zero in this case) shows that you have a problem. You should either catch zero up front with a conditional or catch it when it blows up, with an exception. Python favors the latter.
But without context you don't know what that bad input means and so you cannot "fix" division by zero in the general case because it isn't the division by zero that is the cause of the problem.
Asking "what should a division by zero actually return?" is the wrong question asked at the wrong point with the wrong information. Does the zero indicate lack of initialization? Does it indicate an empty container or volume? Does it indicate absence? How much of a problem to the logic of the program is this particular zero? How much of a problem is it for the person running it?
So while I personally dislike exceptions and prefer return codes, exceptions are just the messenger here and they are an effective messenger. Don't shoot them.
It's hyperbolic and frames your position in a way that's likely to create poison. The people without enough experience & knowledge to evaluate your argument critically are likely to end up parroting it and looking like fools. The people who can evaluate your argument critically are going to scrutinize it much more carefully because you've taken an absolutist position.
Unless you have the knowledge & authority to really make such an absolute statement you do yourself and your readers a disservice.
Python could use a "raises" keyword though. Whether that has real benefits is debatable.
Lazy is good (sometimes)
The point is that it is just not possible to get to perfect reliability. Your actual reliability is always going to be less than 100%. You can invest money and effort to get closer to 100%, but obviously you are going to get diminishing returns.
The correct analysis is to decide where the optimal trade-off is between investing in reliability and the return on that investment.
Example 1: You are calling a web service that checks the weather. The service might be down. If it is down you wait a few seconds and try it again. The 'cost' of it being down is that a user doesn't see the current weather. Is it worthwhile to carefully try to determine whether the error when calling the service is due to a server returning a 500 status code versus invalid json?
No, it's not worth it. Either way the client can't use the response. In fact, it doesn't matter what causes the exception, since anything that goes wrong can't be corrected by the client. Whether it's bad json, a network failure, dns failure, the server is being rebooted, the webserver is misconfigured, or the device is in airplane mode, the resolution is always the same, wait and try again in a few seconds. Exceptions work pretty much ideally in this case, you only have to code the 'happy' path and handle all exceptions the same way generically.
Example 2: You are writing code to update a database containing financial transactions. If something goes wrong in an unexpected way you need to make sure the financial data isn't updated or left partially updated.
Again, you don't care about unexpected exceptions. For failures you expect and are coding to work around them, possibly by catching the generic exception where it happens deep in the call stack, and then raising your own exception class which properly identifies the error and contains the context necessary to perform the recovery. For example, if you need to send an email via receipt for the transaction, you call some function which formats and sends the email. That function fails due to the email server being unreachable. The network exception is caught and an EmailCantBeSent exception is raised with the relevant details in it (the user_id you were emailing, the transaction_id the email is for). The resolution is to log a critical error and insert a record to the database with the relevant details of the email so that someone can make sure it is sent later. Then it continues with the transaction. If something unexpected happens the database transaction is never committed.
My point is that there are two kinds of exceptions you run into, the ones you are being careful to trap and resolve as part of your applications design, and the ones that you aren't trying to resolve and so result in just a generic 'this failed' situation.
So finally getting back to finding the optimal tradeoff between investment to improve reliability and payback on that investment, you just need to make sure your generic failures are rare enough that you aren't pushed far from that optimal point, which is almost always going to be the case, even if you basically don't ever handle any exceptions and only code for the happy path. Obviously there are tons of counter-examples and sometimes you need to make sure things work even when something goes wrong (if you are working on an autopilot you will require much higher reliability and so much more careful planning to reach it compared to a twitter client, where you just need to not lose what the person typed).
Ok that's a lot longer than I intended.
TLDR; If you do any kind of analysis on why code fails and what you should do about that, you quickly realize that this library doesn't help at all. This library isn't even bad, the problem it is meant to solve is not well posed.
"X" implies "X considered harmful"
Thus
"X considered harmful" considered harmful
"'X considered harmful' considered harmful" considered harmful
...
QED
But at the language level, unchecked exceptions are not part of method signatures (as you said), therefore not type- checked/declared/inferred. Therefore appropriately described as untyped -
When calling a method, you have no (formal) list of unchecked exceptions that might be raised.
If you think of exceptions as being an implicit union type around every function return (ie monad), analogous to how you have to explicitly check for errors in C/Rust, you'll see what I mean. Java's unchecked exceptions are akin to calling a function in Python that you expect to return objects of only one type, but not being "sure" that it can't return something else.
Of course you get to deal with a bit of FFI when calling built-in functionality of the browser.
(See CoffeeScript, TypeScript, Elm, etc. Or Transcrypt if you want pythonic semantics.)
I understand this being a business requirement made by a business droid. Other than that, there is really no reason to be married to JavaScript.
In java you need to specify what your function can throw and if it tries to throw something that it has not specified as being able to throw, then your program won't compile.