When REST isn't Good Enough(braintreepayments.com) |
When REST isn't Good Enough(braintreepayments.com) |
We spent many months ripping every client library out down to the raw requests. We are in a unique position because of how many APIs we have to support, but the overarching advise we give to people designing APIs:
If you are deviating from the norm, you're doing it wrong.
REST APIs are nice because you can infer how to communicate with the API out of the gate. APIs are a pain and you only introduce headaches by not doing things in a sane, conventional fashion.
SSL: It is not the onus of the vendor to ensure people are properly securing requests. I have seen horrendous things done in the name of "security" but it only adds headaches. Fix this at the client level. SSL everywhere is sufficient. By extension, vendors often introduce mechanics because they want to make the world better: http://xkcd.com/927/
Platform Support: I think the counter-argument is community client libraries, which the OP is correct: they are worse. The stability + scaling + support issues mentioned by OP can usually be solved by better API design and better architecture behind the scene.
Backwards Compatibility: OP claims you never have to update your code if you use their client library. This is likely true. Good API design would dictate not breaking exposed endpoints unless there is new functionality to be had. In this case, to take advantage of any new API features, you'd still have to update your code. No cost savings are had either way.
Even looking at Stripe; they provide a JS library so you can do client-side encryption. However, if you include that JS library on an incorrectly SSLed page, a malicious party could replace it without you knowing. Even with extensive hand-holding, if a user is going to implement SSL badly, there will be security risks.
The only sure-fire solution is for the developers of SSL libraries to come together and implement sane default options, and clear documentation of how not to do things. Looking at the docs for OpenSSL, it`s impossible to easily discern what counts as a sane configuration. The same problem propagates itself into language-specific libraries, where the dev wasn`t quite sure what options to tick to begin with. And so it goes.
The biggest question I wish the various tutorials would address is: why is REST a good idea? Why is this particular way of doing things better than others?
And here's one thing I believe is true, that I have literally never seen in a REST tutorial: sometimes REST is not the best way to go. Sometimes an RPC architecture is better.
It's amazing, really. It's not just a recipies list, it explains a lot of very important REST cases, and also when to use or not to use REST.
It's not because we'd like to document the API less, but because there are related things - http connections, auth, etc - that need to be handled and it's easier to have the library do them. Further, the library serves to wrap the API in the idioms appropriate for the language. This may include data structures, but also includes things like variable names.
That way you can focus on your app in your chosen.. and only dig into the abstraction if you choose to.
BrainTree and you may not make the same decision, and you both could be right.
If the boilerplate for manually getting an SSL connection right in a given language is that obtuse, the "just use our library" pitch is even more compelling.
All this said, I have used Braintree's libraries and they are extremely well executed.
Not exposing REST API documentation externally (it surely exists internally, right?) just feels like a cop out.
(An argument I'm surprised was left out: it's easier for support staff to work with customers integrating with a good library than some home rolled REST client. I can imagine this to be true, but maybe a case of premature optimization if the majority of big API players expose and document their REST APIs anyway?)
REST version: UPDATE /course/324234 { description: "This is a level 1 course" }
Improved version: POST /course/UpdateDescription { courseId: 324234, description: "This is a level 2 course" }
In this case POST is always used for api calls. Course is the namespace, UpdateDesciption is the method name, and parameters are kept all together as JSON.
Security - agree with them on this, the more they can help their users make their systems secure the better. Not sure if it should preclude a public REST API but certainly motivates for having a good client library.
Platform Support - another good reason to have the client library, essentially encoding best practice in the client. I've certainly seen customers abuse features of our APIs. Again, not sure if it should mean keeping the REST API private. Certainly it's a good idea to be defensive on both the client and server (e.g. for queries that request too much data, rate limiting etc.).
Backwards compatibility - Here I disagree with Braintree. I think it should be just as easy to manage backwards compatibility purely on the server side.
Making a client library is also a great way to "dogfood" your REST API - you know a developer will write code, not HTTP calls, to interact with your service. The client API allows you to test out that code and make sure your API is well designed.
What kind of boilerplate would that be? For me one of the big advantages of a RESTful interface is that they are really easy to call from cURL without much need for building up a complex context.
Contrast that with a client library which can do all of that for you, for example: taking a native model object and calling one of two callback functions provided, letting you concentrate on the business logic of your app.
I'm not arguing that a client library is the right way to go, but I certainly understand where the grandparent is coming from RE: boilerplate code. Also, this is going to be heavily influenced by the language and libraries you're using.
The OPTIONS method is a really underused part of HTTP and would be great for this purpose: http://zacstewart.com/2012/04/14/http-options-method.html
That said as a clojure dev I'd much rather work with a nice clojure http library like clj-http than have to use their provided java library.
On the other hand, the security argument is quite strong. Improperly implemented SSL handshaking, especially when dealing with payment transactions like they do, has the potential to be devastating. But this is where interacting with the community that is making the api's would be key. There's a lot of value to be had when company and community development work together. They could contribute the ssl code themselves to the open source client projects.
The biggest benefit comes when building a cache architecture. RESTful API's (when properly implemented) come with built-in assumptions about idempotency. You end up in a place where you can easily cache GET requests while reasoning in a very consistent way about where changes to a resource will be made.
It's a pattern that comes with a lot of really useful benefits.
Now all of those same qualities can be built into other architectural styles (such as the one you propose). However, I find that it becomes much harder to reason about the role of operations and what they're actually doing when you lose that clear distinction that REST enforces.
I do want to quibble with one thing as well:
In REST the parameters are spread over 3 places, the action, the url and the post parameters themselves
That's true for any modular system isn't it? You have the object you are operating on, you have the operation, and you have the data. Object-oriented systems and even most structured languages (like Javascript for instance) make this distinction at some level. Why is that not appropriate for web architecture?
Sure, but people can implement that pattern without buying into the whole clumsy REST edifice.
For example, a strict reading of REST (and RESTers support such readings!) would tell me that if I have an API that takes two cities and returns the distance between them, then I must expose a URI pointing to every combination of cities.
See here: http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hyperte...
>>A REST API should be entered with no prior knowledge beyond the initial URI (bookmark) ... From that point on, all application state transitions must be driven by client selection of server-provided choices that are present in the received representations or implied by the user’s manipulation of those representations. ... [Failure here implies that out-of-band information is driving interaction instead of hypertext.] [emphasis mine]
So, 1000 cities, you must send a million links over the network rather than 1000 possible parameters plus the formatting.
Now, some RESTers assure me that, no, you can somehow communicate to the user that [prefix] / [first city] / [second city] will get you that answer; that a server can "instruct clients on how to construct appropriate URIs".
Fine, but then we're right back to custom, author-documented APIs and RPCs: "to do that, format the call this way". Right back where we were before trying to force-fit everything into a CRUD mold with increasingly bizarre tables just to make all the calls work.
Or maybe a REST purist would tell me that I'm supposed to link a URI for the starting city: [root]/distances/[start city]/, and then from there link them to possible choices of the second city: [root]/distances/[start city]/[end city]/ .
Fine, but why jump through two pointless hoops, when I know what I want, and I prefer to just send one request rather than pointlessly spend bandwidth navigating a path I don't care for?
What you've described is RPC, which has been around for ages. Why do you think REST became a popular alternative to RPC?
Nothing to do with one man's phd thesis or anything (that came YEARS after he wrote the HTTP spec and after most RPC systems). If I had a penny for all the theses that proposed something reasonable which was ignored.
With your design, if your goal is to "combine all the parameters in one place to keep things simple", you could go one step further and have:
POST /course/update {courseId: 1234, field: "description", description: "hello"}
Taking this to its logical extreme, you can truly combine all the parameters in one place:
POST / {object: "course", operation: "update", courseId: 1234, field: "description", description: "hello"}
In the end a route/url maps to a method with parameters anyway. I wish JSON-RPC was a bit more popular. http://www.jsonrpc.org/specification
You can make calling a function on a remote machine look and seem (superficially) like calling a local function, but they will never have similar behaviour. A network is very different from a motherboard.
http://www.tbray.org/ongoing/When/200x/2009/05/25/HTTP-and-t...
It makes for predictable APIs for consumers and it means API producers have a checklist of things to implement to consider their interface 'complete'. It also provides a consistent way to think about how to expose an interface (or in fact, how to expose entities).
- UPDATE/course/324234 - GET /course/324234 - DELETE /course/324234
with predictable results.
Of course, it works better when you have a clear hierarchy of resources, and/or it make sense to do UPDATE/GET/DELETE to the resources with clear results.
I think it works for some cases very well, but I don't think is always the way to go. That said, there are a sensible number of "RESTful APIs" that are not RESTful at all, they just uses HTTP.
I've seen a couple of cases where backwards compatibility can be broken in unexpected ways. For one application that we built at Braintree, we had a client that was sending us an application/x-www-form-urlencoded POST body without the Content-Type request header. We upgraded the version of Rails that this app was using, and it broke that integration because Rails made a change where it wouldn't parse the POST body without the Content-Type header. Unfortunately, we didn't have any test cases in our test suite that made POSTs without a Content-Type. We were able to identify the issue and resolve it quickly, but it was a surprising bug. With client libraries, we can test every version against the upgraded app and know that all clients will continue to work.
Are there interesting request profiling techniques that can be executed on production traffic to analyze requests? I think the challenging part of backwards compatibility is making sure unintentional use cases, that were never intended to be supported, continue to work.
That said, these are some of the things that I've done to help with backwards compatibility through only the serverside API:
* Track production requests and use them as test cases
* Build a large suite of test cases against the API
* Build the API in a statically typed language (yeah, I know, contentious, I love Rails but there's something about an externally facing API that makes me want to use a statically typed language).
* And then the ultimate -- build the API as an app that talks back to the business logic... essentially, it's your 'client library' but deployed on the server between the actual code and the client. Then, never change the part of it that faces the client, only the mappings on the the real business logic.
Did you evaluate whether a RPC pipe or distributed filesystem was a better fit?
Two responses: first, Braintree customers may appreciate knowing that Braintree provides a safer implementation than whatever the (third)contractor that they hired to build thier web app might throw together.
Second, fraud does directly impact their business, regardless of the entry point used by the attacker. Even if they don't get stuck for the money directly, they are going to lose time talking to customers and helping with investigations.
They don't have to do either, but they may find that they can make some types of customer relationships profitable that would not be for a competing business that doesn't do as much hand-holding.
Maybe a kind of chaos-monkey for API connections could help? Every now and then the API returns a wrong SSL certificate. If the client ignores this, they could notify the customer, or directly block further access until the problem is solved? :)
Unless that is what you mean by (b).
I'll be going through it tomorrow, if I remember I'll come back to post a mini review.
I think that the full versions of there acronyms make a pretty good job of explaining what is best when.
"REpresentational State Transfer": obviously, it transfers state, i.e. information about a certain resource at a given moment. "Remote Procedure Call": obviously, it calls a procedure, which may involve several state changes and other activities. Of course, a procedure may be masked behind a REST endpoint (e.g. when you POST some data and do some procedure before you end up with a certain state) and vice versa (e.g. a simple getter), but you may pretty much view REST as the SQL of the Web, and RPC as stored procedures.
Indeed, but that just raises the question: what would possess you to write an app out of SQL calls rather than general functions? So then why do the Web equivalent thereof?
You can most definitely write a GET request that contains side-effects, but that's because you're doing it wrong (tm).
Imagine if your function names had data in the call path MyCourse.32423.Delete(); that doesn't look very good does it? 32423 would look alot better inside Delete().
That's not data, that's an object. "MyCourse.32423" is the actual resource. It would actually look something more like:
MyCourse32423.Delete();
Which seems pretty reasonable to me if we're trying to map this into programming language semantics (which I'm not even sure we should be doing).
You basically have said "I want to make it work the way I want it and I don't care if it's non-standard and unintuitive.". You're welcome to do that, but everyone who needs to work with it will hate you. And when you've been working on it for 2 months, or take a two month break, you'll come back and hate yourself too...
http://en.wikipedia.org/wiki/Representational_state_transfer
> 2. instruct clients on how to construct appropriate URIs
> 3. link a URI for the starting city and then from there link them to possible choices of the second city
All three of these are possible RESTful solutions, and I can imagine situations where each of them might make sense.
But what you're probably looking for is something like this:
<form><select name="city1">...</select><select name="city2">...</select></form>
There are tradeoffs involved in using this style. Are they worth it? That really depends on the larger system and all kinds of details which you've left out.
You have to come up with some added, unnecessary, implementation-exposing abstraction like "CityPair". (In which case "delete" could be ill-defined if, as would be wise, the distance is computed from a lat/long or road table lookup, and so there is no actual database entry that corresponds to the distance between the cities.) That's the problem with REST: it doesn't avoid the complexity of RPC; it just crams it into ever-more-creative resource types.
Most API users (sorry, "consumers") would prefer they just be told the format in which to ask for the data, not have to re-discover it through gradually-exposed paths each time.
The distance between the two cities.
> You have to come up with some added, unnecessary, implementation-exposing abstraction like "CityPair".
This is a really weird thing to say. No, you don't. I have no idea why you would think that.
> What are the four CRUD operations for it? [...] In which case "delete" could be ill-defined
Not every method has to be valid for every resource. It's a total non-issue that you can't delete a distance.
(And POST/GET/PUT/DELETE is a very different concept from CRUD.)
"REST provides a set of architectural constraints that, when applied as a whole, emphasizes scalability of component interactions, generality of interfaces, independent deployment of components, and intermediary components to reduce interaction latency, enforce security, and encapsulate legacy systems."
The existence of these properties should be fairly evident when you examine the constraints.
I think large part of the confusion comes from not recognizing the difference between "Any system can be built in a RESTful style" (this is true) and "Every system should be built in a RESTful style" (this is a straw man).
GET /distance?cities=San+Francisco,Los+Angeles would suffice. city1=modesto&city2=fresno might be preferred if you wanted to support form submissions, too.
Query strings are perfectly restful.
I agree that it's much more convenient to have well-defined fill-in-the-blank. I disagree that it matches the requirements of REST, whose proponents always tell me "If you document the URIs or how to build them, you're doing it wrong."
http://stackoverflow.com/questions/1164154/is-that-rest-api-...
The crucial thing that you're missing is that it's being told how to construct that URI in-band. Not in a piece of documentation that has to be hard-coded into the client.
I've found that REST design is great for problems where the most visible abstraction is a document or an object (with operations or functions as secondary). It doesn't really fit problems where the most common abstraction is a function call (with the arguments a secondary concern).
You're talking about wire formats and structures that feel natural to use for particular kinds of calculations. I'm talking about recognizing properties that we might need a particular system to have (ease of use being only one of them), and ensuring them.
The line I quoted tells us exactly which problems (in the sense that I'm talking about) REST is suited for: the ones where those benefits are needed (and where the tradeoffs aren't to costly).
Not-REST: You can algorithmically generate requests for the information you want based on documentation.
REST: You can algorithmically generate requests for the information you want based on documentation that's provided in the session. (Oh, and forget all that crap we said about URIs having to be presented to the API consumer as part of session's hyperlink navigation.)
So, now I can turn RPC into REST just by slapping a link to the documentation on every response (and maybe moving verbs to the appropriate HTTP request type, even though that has nothing to do with CRUD[1])? Gee, why didn't you say so?
You're not listening, and you're not arguing in good faith.
Alternatively, explain how REST can simultaneously meet the constraints of "avoid combinatorial explosion of possible URIs to explicitly present" and "every resource is accessible by following server-provided links" and "avoid unnecessary bandwidth usage".
This merely happens to overlap significantly with the requirements of public APIs.
I have difficulty imagining your distance calculator needing any of those things, though.
>I have difficulty imagining your distance calculator needing any of those things, though.
Right, because no one's stupid enough to use (or stick to the use of) REST for a distance calculator or any other algorithmically generated information. But make no mistake, scalability of component interaction is an issue, just a solved one (for those that see REST for what it is).
The solution is: specify an input scheme (for a hand calculator: put the first number, then plus, then the second number, then equals) and let the user choose the inputs. This saves you from the (intercomponent-unscalable) combinatorial explosion in which you have to give the user a link to every possible computation as they navigate the interface, and which is the REST method.
So, any exposed function in which you can't feasibly blast every possible input set over the network is REST-incompatible, so I guess the serious RESTers don't think you should do it. Which kinda makes it little more than a footnote.
>The distance between the two cities.
Don't be cute. What server resource is it acting on? The value of the distances is (potentially) computed by some encapsulated algorithm -- you're not acting on that resource. The server resources that are touched are the two cities, and then whatever it does behind the scenes.
>This is a really weird thing to say. No, you don't. I have no idea why you would think that.
Because I'm not GETing a city, so I can't use the GET operation on the city resource; I have to make another resource to GET. Fine, it doesn't have to be a city pair: but there is a many-to-many mapping, which in REST-favoring frameworks (e.g. Rails), requires a separate table. By reducing everything to CRUD, you must create a new resource (type) for each new operation.
>(And POST/GET/PUT/DELETE is a very different concept from CRUD.)
That's a non-standard definition of "very different", considering that POST is create, GET is retrieve, PUT is update, and DELETE is delete.
Also, while POST can mean "create", it can also mean "append" or "process some arbitrary request."
Yep! Works like a charm, until you have to expose a URI pointing to every combination of cities (or indeed, combination of any parameter set).
And I know (like I said before) you can fall back on "no, just tell the user where to put the parameters and you won't have to do that!" ... which is just re-inventing the RPC -- and satisfying users that don't want to navigate a long session just to find the URI they want, every time they send a request.
>Also, while POST can mean "create", it can also mean "append" or "process some arbitrary request."
Used correctly, it doesn't mean (that the sender is requesting that you) "process some arbitrary request"; it should only be used for non-idempotent operations. Close enough to summarize as "create" (appending is certainly creating something in this context!), and generally, for something to have different effects when repeated, you have to create something. PUT/update and DELETE/delete are idempotent specifically because the changes they make aren't creations.
In any case it's clearly an abuse of the term "very different concept from" in the GGP's comment "And POST/GET/PUT/DELETE is a very different concept from CRUD".
Some kinds of apps (esp those that can't tolerate the overhead of REST, like for mobile) need to know how to format a Google search request without navigating through a session on Google site, but just knowing what it should look like, and formatting it that way. REST would restrict you to pointing them to google.com and following links; it prohibits you from saying, "hey, you can have your app just point to google.com, then '?q=', then your search terms connected by +'s".