How I write HTTP services in Go after 13 years(grafana.com) |
How I write HTTP services in Go after 13 years(grafana.com) |
Only thing I agree on is putting all the paths in one file.
In most other programming languages I've done a lot of research how to make it nice and clean.
Was hoping this was it for Go because I'm cleaning up a big project.
But my very basic no nonsense current setup seems better to me than this in many ways.
If anybody has another example that is a lot better (and I don't mean complexer I don't have those ego issues), I am very interested.
But this I want to hide as best as I can from my dev team this is all wrong. It's clever in a lot of ways but it's wrong.
It does not have unit testing at all, all these tests would be duplicated in the end-to-end test.
I also like end-to-end tests better but why put them here, way better to put them in postman for example then you have the most up to date documentation always auto-generated.
Passing the config, man I had so many discussions with junior developers about this, don't do that you'll make things dependent on the config and cannot reuse them in other programs. But that was already mentioned a lot here.
There are also a lot of functions with like 10 arguments passed. If you have that many arguments just pass a stuct containing a lot of the arguments it's always super confusing when people make functions with 12 arguments. I'm always counting them an after 14 times counting I rewrite their function.
It's a matter of style so keep doing it this way if you like it, but it's not my style at all it makes no sense at all to me.
If anybody knows a better example please tell me.
I used to do this, but ever since reading Lexi Lambda's "Parse, Don't Validate," [0] I've found validators to be much more error-prone than leveraging Go's built-in type checker.
For example, imagine you wanted to defend against the user picking an illegal username. Like you want to make sure the user can't ever specify a username with angle brackets in it.
With the Validator approach, you have to remember to call the validator on 100% of code paths where the username value comes from an untrusted source.
Instead of using a validator, you can do this:
type Username struct {
value string
}
func NewUsername(username string) (Username, error) {
// Validate the username adheres to our schema.
...
return Username{username}
}
[0] https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
1. Or maybe a map? Those keys might have significance I didn't tell you about.
Types limit you from making some mistakes, but it also impacts your extensibility. Imagine an enum with 4 values and you want to add 1 because 10 level deep one of the services need new value. How does it usually go with strongly typed languages? You go and update all services until new value is properly propagated to lowest level who actually needs that value.
Now imagine doing same with strings, you can validate at the lowest level, upper levels just pass value as it is. If upper layers have conditionals based on value, they still can limit their logic to those values
See Steven Witten's "I is for Intent" [1] for some ideas about the use of unvalidated data in a UI context.
The legitimate insight that they have is that anchoring the state as close as possible to the user input is valuable—I think that that is a great insight with a lot of good applications.
However, there's nothing that says you can't take that user-centric state and put it in a strongly typed data structure as soon as possible, with a set of clearly defined and well-typed transitions mapping the user-centric state to the derived states.
Edit: looks like there was discussion on this the other day, with a number of people making similar observations—https://news.ycombinator.com/item?id=39269886
Parse, don't validate (2019) - https://news.ycombinator.com/item?id=35053118 - March 2023 (219 comments)
Parse, Don't Validate (2019) - https://news.ycombinator.com/item?id=27639890 - June 2021 (270 comments)
Parse, Don’t Validate - https://news.ycombinator.com/item?id=21476261 - Nov 2019 (230 comments)
Parse, Don't Validate - https://news.ycombinator.com/item?id=21471753 - Nov 2019 (4 comments)
I personally find being robust to errors and having clear error messages is the best option.
Don't focus so hard on getting things right, but rather dealing with things when they go wrong.
I've tried it with constructs like `Data` and `ValidatedData` and it definitely works, but you do end up with duplicate fields between the two objects or worse an ever growing inheritance tree and fields unrelated to either object shared by both.
For example, consider data looking like
Data {
value string
}
ValidatedData {
value int
}
The often misinterpreted part of DRY is conflating "these are the same words, so they are the same", with "these are the same concept, so they are the same". A Username and a String are conceptually different.
It’s almost always better to repeat code. It’s obviously not something that is completely black and white, even if I prefer to never really do any form of inheritance or mutability, it’s not like I wouldn’t want you to create a “base” class with “created by” “updated by” and so on for your data classes and if you have some functions that do universal stuff for you and never change, then by all means use them in different places. But for the most part, repeating code will keep your code much cleaner. Maybe not today or the next month, but five years down the line nobody is going to want to touch that shared code which is now so complicated you may as well close your business before you let anyone touch it. Again, not because the theoretical concepts that lead to this are necessarily flawed, but because they require too much “correctness” to be useful.
Academia hasn’t really caught on though. I still grade first semester students who have the whole “Animal” -> “duck”, “dog”, “cat” or whatever they use into their heads as the “correct way” to do things. Similar to how they are often taught other processes than agile, but are taught that agile is the “only” way, even though we’ve seen just how wrong that is.
I’m not sure what we can really do about it. I’ve always championed strongly opinionated dev setups where I work. Some of the things we’ve done, and are going to do, aren’t going to be great, but what we try to do is to build an environment where it’s as easy as possible for every developer to build code the most maintainable way. We want to help them get there, even when it’s 15:45 on a Thursday that has been full of shit meetings in a week that’s been full of screaming children and an angry spouse and a car that exploded. And things like DRY just aren’t useful.
It causes what I call the lumpy carpet syndrome - sweeping the complexity under the carpet causes bumps to randomly appear that when squashed tend to cause other bumps to pop up rather than actually solving the problem.
...Easy!
[1]: https://fsharpforfunandprofit.com/posts/designing-with-types...
But copy-pasting the same lines of code in literally every function is the Golang Way.
It makes code "simpler".
>If you are doing the validation inside of a constructor, you are still doing validation instead of parsing.
Why that would be considered validation rather than parsing?
From the original post:
>Consider: what is a parser? Really, a parser is just a function that consumes less-structured input and produces more-structured output.
That's the key idea to me.
A parser enforces checks on an input and produces an output. And if you define an output type that's distinct from the input type, you allow the type system "preserve" the fact that the data passed a parser at some point in its life.
But again, I don't know Haskell, so I'm interested to know if I'm misunderstanding Lexi Lambda's post.
type Username string
return Username{username}
return Username(username)A caller can just say:
// This is returning an error for some reason, so let's do it directly.
// username, err := parsers.NewUsername(raw)
username := parsers.Username(raw)
var u Username
u = "<hello>" // Implicitly converts from string to Username1. Create your Schema using https://zod.dev or https://github.com/sinclairzx81/typebox or one of the other many libs.
2. Generate your types from the schema. It's very simple to create partial or composite types, e.g. UpdateModel, InsertModels, Arrays of them, etc.
3. Most modern Frameworks have first class support for validation, like Fastify (with typebox). Just reuse your schema definition.
That is very easy, obvious and effective.
If I have a user type, inferred from a Zod schema:
> { username: string; email: string }
And a function which takes that type:
> storeUser(user: User)
There is absolutely nothing that guarantees that the user object has been parsed by Zod. You can simply:
> storeUser({ username: “”, email: “no” })
And Typescript will not shout at you.
The only way to comparably solve it with Typescript is to inject a symbol into the object during parsing which confirms it has been passed through the correct parser function.
Personally, I just do basic type parsing on input data (usually request data) and more strict parsing where constraints like “is this a valid username, is this a valid email” during output (usually sending to the database). What happens in between I/O doesn’t matter much in many projects (CRUD), and in the places it does you can enforce more rigidity.
This is annoying to translate later. At least also include some error code string that is documented somewhere and isn't prone to change randomly.
type UsernameError struct {
name string
reason string
}
func (e *UsernameError) Error() string {
return fmt.Errorf("invalid username %q: %s", e.name, e.reason)
}
This is fine for lots of different cases, because it’s likely that your code wants to know how to handle “username is invalid”, but only humans care about why.
I have personally never seen a Go codebase where you parse error strings. I know that people keep complaining about it so it must be happening out there—but every codebase I’ve worked with either has error constants (an exported var set to some errors.New() value) or some kind of custom error type you can check. Or if it doesn’t have those things, I had no interest in parsing the errors.
Parse, don't validate:
string ParsedString
untrusted source -------> parse --------------> rest of system
UnvalidatedString string
untrusted source ------------------> validate -------> rest of systemIf you default to treating primitive types as untrusted, it's hard for someone to accidentally convert an untrusted type to a trusted type without using the correct parse method.
In the example I gave above, clients outside of the package can instantiate Username, but they can't access its "value" member, so the only way they could get a populated Username instance is by calling NewUsername.
I push my team to use this pattern in our (mostly Scala) codebase. We have too many instances of useless validations, because the fact that a piece of data has been "parsed"/validated is not reflected in its type using simple validation.
For example using String, a function might validate the String as a Username. Lower in the call stack, a function ends up taking this String as an arg. It has no way of knowing if it has been validated or not and has to re-validate it. If the first validation gets a Username as a result, other functions down the call stack can take a Username as an argument and know for sure it's been validated / "parsed".
one of my biggest pet peeves is when people take a Config object, which represents the configuration of an entire system, and pass it around mutably. When you do that, you're coupling everything together through the config object. I've worked on systems where you had to configure the parts in a specific order in order for things to work, because someone decided to write back to the config object when it was passed to them. Or another case was where I've seen it such that you couldn't disable a portion of the system because it wrote data into the config object that was read by some other subsystem later. The pattern of "your configuration is one big value, which is mutable" is one of the more annoying patterns that I've seen before, both in Go and in other languages.
The one weak spot for me is this aspect:
>NewServer is a big constructor that takes in all dependencies as arguments... In test cases that don’t need all of the dependencies, I pass in nil as a signal that it won’t be used.
This has always felt wrong to me, but I've never been able to figure out a better solution.
It means that a huge chunk of your code has a huge amount of unnecessary shared state.
I often end up writing HTTP handlers that only need access to a tiny amount of the shared state. Like the HTTP handler needs to check if the requesting user has access to a resource, and then it needs to call one function on the datastore.
I'd love to write tests where I only mock out those two methods, but I can't write simple tests because the handler is part of this giant glob where it has access to all of the datastore and every object the parent server has access to because it's all one giant object.
Nothing against Mat Ryer, as his pattern is the best I've found, but I still feel like there's some better solution out there.
> My handlers used to be methods hanging off a server struct, but I no longer do this. If a handler function wants a dependency, it can bloody well ask for it as an argument. No more surprise dependencies when you’re just trying to test a single handler.
For HTTP services in any language, your handlers will usually end up with a lot of business logic, logic which probably has many dependencies. I see single handlers using all of the following on a regular basis: DB, cache, blob storage, some kind of special authz thing specific to your endpoints, maybe some fancy licensing checker, a queue or two, a specialized logger, and specialized metrics client. Many of those (metrics, request/response logging) can live in middlewares most of the time, but in every code base there will be times where you need to do something custom with one or the other. As time passes, the more I wonder "why aren't these all just function parameters?"
Yes, that would be a lot of function parameters (9+ for a single handler, before even getting into the request or custom params themselves), and we all have many rules of thumb and linter rules which try to keep us from having lots of function parameters. But it's not like we're not writing code which depends on all those dependencies, instead we're just sticking them on the "server" class/struct and pretending that because the method signature is shorter, we have fewer dependencies!
As time passes, I find myself wishing more and more for code that takes all its dependencies in the function/method signature, even if there's 20 of them; at least then we wouldn't be lying about how complex the code's getting...
* I would pass a waitgroup with the app context to service structs. This way the interrupt can trigger the app shutdown via the context and the main goroutine can wait on the waitgroup before actually killing the app.
* If writing a CLI program, then testing stdout, stdin, stderr, args, env, etc. is useful. But for an http server, this is less true. I would pass structured config to the run function to let those tests be more focused.
* I disagree with parsing templates using sync.Once in a handler because I don't think handlers should do template parsing at all. I would do this when the app starts: if the template cannot be parsed, the app should not become ready to receive any requests and should rather exit with a non-zero exit code.
Write openapi definition, it'll do routing, definition of structs, validation of JSON schemas, etc.
All I need to do is implement the service.
Validating an integer range for a querystring parameter is just too boring. And too easy to mistype when writing it manually.
Anyways, so far only been playing, so haven't found the bad parts yet.
Agree it doesn't fix the "root" problem that the overall syntax is not ergonomic.
I thougt it would be boring writing the spec, but it was not nearly as bad as I thought. Also, a spec is needed, so might as well write it up front.
All the advice in the article is still helpful, but it takes the "how do I make sure X is initialized when Y needs it" part completely out of the equation and reduces it from an N*M problem to an N problem, ie I only have to worry about how to initialize individual pieces, not about how to synchronize initialization between them.
I've used quite a few dependency injection libraries in various languages over the years (and implemented a couple myself) and the simplicity and versatility of fx makes it my favorite so far.
I gotta say, I hate these dependency injection frameworks.
In a well designed system this should be trivial. Making sure something is initialised when you want to use it is just a matter of it being available to pass in a constructor as a parameter.
stockService := NewStockService()
orderService := NewOrderService()
orderProcessor := NewOrderProcessor(stockService, orderService)
srv, err := newTestServer()
require.NoError(t, err)
defer srv.Close()
resp, err := http.Post(fmt.Sprintf("http://localhost:%d/signup/json", srv.Port()), "application/json", strings.NewReader(` {"email":"test@example.com", "password": "p@55Word", "password_copy": "p@55Word"} `))
In my newTestServer, I spin up a server with fakes for my dependencies. If I want to test a dependency error, I replace that property with a fake that will return an error. I can validate my error paths. I can validate my log entries. I can validate my metric emission. I can validate timeouts and graceful shutdowns.
After the server starts, I inspect to determine which port it is running on (default is :0 so I have to wait to see what it got bound to).
My "unit" tests can test at the handler level or the http level, making sure that I can fully test the code as the users of my system will see it, exercising all middleware or none. I can spin up N instances and run my tests in parallel.
I never want to see another (esp. Python) Quick Start guide that treats dependencies as implicit/static/untestable.
The amount of work you can get done on low-end shared hosting is really quite impressive.
Whose dreams come true in this scenario?
The second issue involves passing an unused *http.Request, which will likely cause the linter to flag it.
I think this is a pretty good example of a real-world implementation:
https://github.com/mtlynch/picoshare
Particularly these files:
https://github.com/mtlynch/picoshare/blob/2cd9979dab084ca781...
https://github.com/mtlynch/picoshare/blob/2cd9979dab084ca781...
And I plan to move to rv64 assembly once I can get reasonably performant hardware (it is already here, but it extremely hard to get some where I am from and how I operate). I dunno if it will be bare metal or with a linux kernel first (coze a minimal TCP stack is already a big thingy).
package config
type options struct {
name string
}
type Option func(o *options)
func Name(name string) Option {
return func(o *options) {
o.name = name
}
}
type Config struct {
opts *options
}
func New(opts ...Option) *Config {
o := &options{}
for _, option := range opts {
option(o)
}
return &Config{opts: o}
}
func (c *Config) Name() string {
return c.opts.name
}
cfg := config.New(config.Name("Emanon"))
fmt.Println(cfg.Name())It's a lot of boilerplate to create something that's not actually immutable. It also makes it harder to figure out which options are available, since now you can't just look at the documentation of the type, you have to look at the whole module package to figure out what the various options are. If one of the fields is a slice or map you can just mutate that slice or map in place, so it's not really immutable. The pattern as Pike describes it has the benefit that supplying an option returns an option that reverses the effect of supplying the option so that you can use the options somewhat like Python context objects that have enter and exit semantics, but in practice I've found that to be useful in a small portion of situations.
Update: What's funny was, in our design the config object was kinda immutable. You have to use the WARNING_DO_NOT_USE api to make modification. We did mutate the object and we caused a sev
Once you've loaded it and mutated it for testing purposes or for copying from ENV vars into the config, you can then freeze it before passing it down to all your app level code.
Having this wrapper object that can be frozen and has a `get()` method to read JSON like data make it effectively not mutable.
type Server struct { val bool }
type Config struct { Val bool }
func NewServer(... config *Config ...) http.Handler { if config == nil { config = &Config{} } return &Server{ val: config.Val } }
It took me a long time to settle on this pattern and I admit it's tedious to copy configuration over to the server struct, but I've found that it ends up being the least verbose and maintainable long term while making sure callers can't mutate config after the fact.
I can pass nil to NewServer to say "just the usual, please", customize everything, or surgically change a single option.
It's also useful for maintaining backwards compatibility. I'm free to refactor config on my server struct and "upgrade" deprecated config arguments inside my NewServer function.
How do you create immutable structs in Go? I didn't think you could, which makes this more a Go problem than a `passing around Config object` problem.
(One of my pet peeves, coming to Go from C, is how little of stronger typing there actually is. In C, I pass and return const objects everywhere I can, my enums are not just ints because the compiler can warn when I forget one in a switch statement, etc).
Where possible plugins are a great strategy to lay down these code seam points that don't force all possibilities upon some body of code, because fundamentally with plugin architectures you pick and choose what you want. Plugins are opt out by default, you must explicitly opt into a plugin for it to manifest. I've been calling software that has this quality going as being an "a la carte" style.
But in general you do what you need to do to avoid "doing everything so you can do anything".
func HandleX(w http.ResponseWriter, req *http.Request) {
// code
}
func HandleX(store *DataStore, dep1 Foo, dep2 Bar, commonDep Common) http.HandlerFunc {
//
// maybe some initialization
//
return func(w http.ResponseWriter, req *http.Request) {
// code
}
}
Simple example is adding a logger. If you add it as a dependency to the constructor, the object starts doing a bit more than initial simple implementation. It's fine to do it, but shame to not figure out how to log without editing the implementation of a simple thing.
Higher order functions (a logger decorator) get there to allow composition, but even they have their drawbacks.
It's still some form of structure that you can deal with, not a mistake.
But if I really feel I can't avoid the need to pass a good amount of external context, I create a dedicated "options" struct and pass that into the constructor as a pointer. The purpose of the pointer (rather than pass by value) is if I want default arguments, I can pass nil.
type ServerOptions struct {
logger *magic.Logger
secretKey string
}
func NewServer(options *ServerOptions) (*Server, error) {
...
}FWIW, I really tried to make the functional option pattern work for me, as many others have suggested, but eventually abandoned it. I felt it was a little too clever and therefore difficult to read, while requiring more boilerplate than the config struct + validate and copy pattern.
https://dagger.dev/hilt/testing-philosophy.html
The biggest problem with the "pass nil for unused dependencies" approach is that when you modify some code to actually use one of those dependencies when it didn't before, you have to go back through every test and populate it.
* https://dave.cheney.net/2014/10/17/functional-options-for-fr...
* https://github.com/uber-go/guide/blob/master/style.md#functi...
Can you explain that a little more?
Which chunk of code has what shared state, and why is it unnecessary?
While I may prefer a struct for this instead of separate arguments, I do agree it's useful to capture "the world" as the set of all dependencies, even if some handlers don't use them (yet).
type CreateUser struct {
store storage.Store
cache caching.Cache
logger logging.Logger
pub events.Publisher
// etc
func (op CreateUser) ServeHTTP(ctx, req, rw) {}
// or if you have custom handlers
func (op CreateUser) ServeHTTP(ctx, input) (output, error) {}
And in my main.go, or where I set up my dependencies, I create each operation, passing it its specific dependencies. I love that because I can keep all the helper methods for that specific operation/handler on that specific struct as private methods.
It does get tedious when you have one operation needing another, as you might start passing these around or you extract that into its own package/service.
Whether all the deps are in the method receiver (the parent struct) or in a struct that's a param; it's all just more indirection to hide all the "stuff" that we need cause we think it's ugly. I dream of a world where we don't do that.
I love this! I will use this as well.
There are so many situations where I have a feeling that people are solving problems that don’t exist. In code I run into at work, code and projects I see online, etc
The “whose dreams are you making come true” really applies here, because dreams are exactly what they are.
I spent quite some time writing an automatic image resizer and optimiser for my blog. Does it matter? No! Should I have spent that time writing blog posts instead? Yes! Still I was chasing some dream.
Thanks for this image
Abstracting things away from main makes it less readable and is general pointless for bespoke services that will be deployed in exactly one configuration.
main() is the only place where you can't return an error. In order to keep as much of the code as idiomatic as possible you just call something like run() where you can do so.
In addition there is the testing aspect. You can't invoke main() from your tests.
If you forget to initialize a field and the tests don't notice, you didn't need the field in the first place, so it won't matter if it is left in an invalid state.
You just don't get the squiggly lines in your text editor. That's the tradeoff.
Text has more possible states than the equivalent AST, many of which are useful when you haven't typed in all the code yet. Incomplete code usually doesn't parse.
This suggests that drafts should be represented as text, not an AST.
And maybe similarly for drafts of other things? Drafts will have some representation that follows some rules, but maybe they shouldn't have to follow all the rules. You may still want to save drafts and collaborate on them even though they break some rules.
In a system that's not an editor, though, maybe it makes sense to validate early. For a command-line utility, the editor is external, provided by the environment (a shell or the editor for a shell script) so you don't need to be concerned with that.
BTW kudos on zanzibar. Love the tech and the code).
A number of bad decisions were made and later discovered over time, and it was much easier to address them with automatic dependency injection available than without. Using as "big" of a test as you can without regressing speed or flakiness gives you really solid, non-fragile tests, which makes large changes tractable to do safely.
I don't know whether certain poor designs would have been spotted sooner, but this certainly made the code easier to change and resulted in more useful and lower maintenance tests.
It’s not perfect, but software evolves through many imperfect stages as it gets better, and this is one such imperfect stage that your software may evolve through.
Including a human-readable version of the error is useful because the developers / operators will want to read through the logs for it. Sometimes that is where you stop, because not all errors from all backends will need to be localized.
And more accurate than LLMs.
Feels like whenever an LLMs could code it, you'd be better of not having the boilerplate code at all.
I was agreeing with parent that some spec formats frankly suck to write by hand and openapi yaml is IMHO one of those (as opposed to say .protos which are nice for humans to read/write).
I use LLM as glorified interactive autocomplete to speed up writing the yaml specs (not the code! Use deterministic generators for that!) and it works great for me, personally (n=1 anecdote).
I'd like a world that sacrifices a bit of aesthetics in order to erase ambiguity or confusion. So instead of putting your deps in a struct that's a param, or putting your deps in a parent closure, I'd like to put them in the function params.
Though I will admit that if I had to choose, I'd use (and have used) the closure approach most often.
Both cases appear to depend on there being a "checkpoint" all data must go through to cross over to the rest of the system, either at parsing or at UnvalidatedString construction.
>Both cases appear to depend on there being a "checkpoint" all data must go through to cross over to the rest of the system, either at parsing or at UnvalidatedString construction.
The difference is that if string is the trusted type, then it's easy to miss a spot and use the trusted string type for an untrusted value. The mistake will be subtle because the rest of your app uses a string type as well.
The converse is not true. If string is an untrusted type and ParsedString is a trusted type, if you miss a spot and forget to convert an untrusted string into a ParsedString, that function can't interact with any other part of your codebase that expects a ParsedString. The error would be much more visible and the damage more contained.
I think UnvalidatedString -> string also kind of misses the point of the type system in general. To parse a string into some other type, you're asserting something about the value it stores. It's not just a string with a blessing that says it's okay. It's a subset of the string type that can contain a more limited set of values than the built-in string type.
For example, parsing a string into a Username, I'm asserting things about the string (e.g., it's <10 characters long, it contains only a-z0-9). If I just use the string type, that's not an accurate representation of what's legal for a Username because the string type implies any legal string is a valid value.
GP is pointing out that go specifically makes it an issue.
But there are simpler and cheaper solutions that might be good enough.
I guess my intention was also to point out that there're mature frameworks for many languages, but somehow most people in the Go community keep reinventing the wheel and unfortunately more often worse than better. Some years ago I wrote a Go web service. Of course I found the first two versions of OPs series. They're great to read and even greater to watch on YT, but I preferred the approach ardanlabs (Bill Kennedy). It was for sure interesting going through all of this, but incredible time consuming.
Now, if your complaint is rather that you can call whatever method and pass in your bogus data, I don't see the point in arguing that. It's your code, the only person who can stop you is you.
This entire comment thread is a discussion about how to prevent that from being a possibility. The person I responded to threw their hat in with a Typescript solution that doesn’t achieve the goal being discussed. I was simply pointing this out.
>This entire comment thread is a discussion about how to prevent that from being a possibility.
No, this thread is also about how much you need to invest to be safe enough, when time and resources are limited.
DRY's goal is to avoid defining behaviors in duplicity, resulting in having multiple points in code to change when you need to modify said behavior. Code needs to be coherent to be "good", for a number of of the different quality indicators.
I'm doing a "side project" right now where I'm using a newcomer payment gateway. They certainly don't DRY stuff. Same field gets serialized with camel case and snake case in different API, and whole structures that represent the same concept are duplicate with slightly different fields. This probably means that Thursday 15.25 the dev checked-in her code happy because the reviewer never cared about DRY, and now I'm paying the price of maintaining four types of addresses in my code base.
God no. Stop the copy pasta disease! It's horrible, mindless programming.
When reviewing code, I'm astonished anything was accomplished by copy pasting so much old code (complete with bugs and comment typos).
Incidentally, OOP encourages you to copy a lot. It's just an engine for generating code bloat. Want to serialize some objects? Here's your Object serializer and your overloaded Car serialize and your overloaded Boat serializer, with only a few different fields to justify the difference!
OOP is bad. Copy pasta is bad. DRY is good. All hail DRY, forever, at any cost.
The idea is that your parsed representation and serializer are likely produce a much smaller and more predictable set of values than may pass the validator.
As an example there was a network control plane outage in GCP because the Java frontend validated an IP address then stored it (as a string) in the database. The C++ network control plane then crashed because the IP address actually contained non-ASCII "digits" that Java with its Unicode support accepted.
If instead the address was parsed into 4 or 8 integers and was reserialized before being written to the DB this outage wouldn't have happened. The parsing was still probably more lax than it should have been, but at least the value written to the DB was valid.
In this case it was funny Unicode, but it could be as simple as 1.2.3.04 vs 1.2.3.4. By parsing then re-serializing you are going to produce the more canonical and expected form.
But yes usually you do want to split something into it's elemental components, should it have any.
But even with that understanding and from re-reading the post, that seems to be an extra safety measure rather than the essence of the idea.
Going back to my original example of parsing a Username and verifying that it doesn't contain any illegal characters, how does a parser convert a string into a more direct representation of a username without using a string internally? Or if you're parsing an uint8 into a type that logically must be between 1 and 100, what's the internal type that you parse it into that isn't a uint8?
IP address would be about the minimum amount of structure. Something else would be like processing API requests. You can take the incoming JSON and fully parse it as much as possible, rather than just validate it is as expected (for example drop unknown fields)
Just for the sake of example, your internal representation might start from 0, and you just add 1 whenever you output it.
Your internal type might also not be a uint8. Eg in Python you would probably just use their default type for integers, which supports arbitrarily big numbers. (Not because you need arbitrarily big numbers, but just because that's the default.)
How is not actually immutable? How could cfg.opts.name be modified after New() returns?
> It also makes it harder to figure out which options are available
I find it easier, the go tooling helps a lot. For example, all the options are grouped together at https://pkg.go.dev/github.com/go-kit/log/level#Option
It's also easy to use "Find Usages" in my editor, and filter on return type.
Functional options have some niceties - largely their ability to evolve without breaking changes - but as the GP points out, completely break discoverability with intellisense. Having to do some dance with filtering usage to find the options available is just worse than a static config struct with zero values that are meaningful for anything not set.
Sure you can. Option func is a constructor for option type, and constructors are auto-included above methods in the docs.
PLS completion works for them as well.
- main constructor is easily available from the main type's docs,
- option type is easily available from the main constructor's docs,
- all option funcs are easily available from the option type's docs (because in fact these option funcs are constructors for the option type).
Excerpt from grpc godoc index:
type Server
func NewServer(opt ...ServerOption) *Server
...
...
type ServerOption
func ChainStreamInterceptor(interceptors ...StreamServerInterceptor) ServerOption
func ChainUnaryInterceptor(interceptors ...UnaryServerInterceptor) ServerOption
func ConnectionTimeout(d time.Duration) ServerOption
func Creds(c credentials.TransportCredentials) ServerOption
etc...
Yeah, thats what I figured. Im not sure if I want the tradeoff of calling .GetValue in multiple places just to save calling validate in maybe 2 or 3 places.
Not to mention I cant easily marshal/unmarshal into it and next week valid username is a username that doesnt already exist in the database.
Maybe this approach appeals to people and Im hesitant to say “that’s not how Go is supposed to be written” but for me this feels like “clever over clear”.
The tradeoff is not that you save calling validate, it’s that you avoid forgetting to call validate in the first place, because when you forget to validate, you get a type error.
IMO it’s a little more clear this way:
type Ticket struct {
requestor Username
assignee Username
}
The original example was more “wrap a simple type in an object so it’s always validated when set” which looks beautiful when you don’t have the needed getters in the example nor show all the Get call sites opposed to the 1 or 2 New call sites. All in the name of “we don’t want to set the username without validation” but without private constructors Username{“invalid”} can be invoked, the validation circumvented and I’m not convinced the overhead we paid was worth it.
It's funny seeing this beside the DRY thread. Seems programmers taking things a bit too literally is a common theme.
“Just do the right thing everywhere and you don’t have to worry!”
You can’t stop consumers of your libraries from creating zero-valued instances.
Coq exists if you really feel you need a complete type system. But there is probably good reason why almost nobody uses it.
Unfortunately "This is the same character string" is all a DRY purist needs to start messing up the code base.
I honestly believe that "DRY" is an anti-pattern because of how often I see this exact behavior trotted out or espoused. It's a cargo cult thing to some devs.
When you add complexity to DRY out your code, you're adding a readability regression. DRY matters in very few context beyond readability, and simplicity and low cognitive load need to be in charge. Everything else you do code-style wise should be in service of those two things.
DRY is about maintainability - if you repeat rules (behavior) around the system and someone comes along and changes it, how can you be sure it affected all the system coherently?
I've seen this in practice: we get a demand from the PO, a more recent hire goes to make the change, the use case of interest to the PO gets accepted. A week later we have a bug on production because a different code path is still relying on the old rule.
DRY applied as a dogma will eventually fail, because it's not a verified mathematical proof of infallible code, it's just a practice that gives good results inside its constraints, people just don't learn the constraints until it explodes in their faces a few times.
Like any wisdom, it's hard it will be received and understood without the rite of passage of experience.
DI still has other advantages, but that’s not one
Especially in Go, where you don't have destructors to help with shutdown, having common structure in place to help tear down components has always been a net benefit for me.
But when micro-services are so common, it seems like people use them (Spring) because everyone else does, not because they actually provide needed value.
It should be inserted literally right next to it's first use case. Your IDE will literally point it to you with red squigglys because the places where you've added a dependency will be missing a parameter. Go to the highest one and add it on the line above.
What do you do on shutdown? In languages with destructors, that can automatically give you a call order in reverse of the construction order, but in Go you end up manually ordering things or just not having panicless shutdowns.
If your loggers are needed everywhere, then you just pass them as a constructor to the objects that need them. You're literally doing this with fx anyway.
Like, a logger is probably the first thing you new up in main(). So now you can pass it down as a dependency in constructors.
For shutdown you just defer your shutdown functions. Have a basic interface where your services have a Shutdown() method and then you can push them onto a stack and pop them off during shutdown.
There's no manual ordering involved. Your initialisation is a linear top down process, your shutdown is bottom up. It can't be any simpler. If you keep code as close to usage sites then there's only 1 possible order.
Doing something twice is not an anathema, that's my point, not when doing it twice is a cognitively easier and practically faster task.
In almost every case, bugs are the result of human error, and keeping cognitive load as low as possible reduces the likelihood of human error in all cases. As DRY as possible is very rarely the lowest cognitive load possible.
I value the hours wasted on diagnosing a bug far more than the extra keystrokes and couple of seconds required to avoid it in the first place.
In my opinion, this pattern breaks when the validation must return an error and everything becomes very verbose.
Except for all those times you're the consumer of someone else's library and there's no way for them to indicate that creating a zero-valued struct is a bug.
Again, it's the philosophy of "Just do the right thing everywhere and you don’t have to worry!" Sometimes it's nice to work with a type system where designers of libraries can actually prevent you from writing bugs.
Nonsense. Go has a built-in facility for documentation to communicate these things to other developers. Idiomatic Go strongly encourages you to use it. Consumers of the libraries expect it.
> Sometimes it's nice to work with a type system where designers of libraries can actually prevent you from writing bugs.
Well, sure. But, like I said, almost nobody uses Coq. The vast, vast, vast majority of projects – and I expect 100% of web projects – use languages with incomplete type systems, making what you seek impossible.
And there's probably a good reason for that. While complete type systems sound nice in theory, practice isn't so kind. There are tradeoffs abound. There is no free lunch in life. Sorry.
…where, "what GP seeks" is…
> way for [library authors] to indicate that creating a zero-valued struct is a bug
I'd say that's a really low and practical bar, you really don't need Coq for that. Good old Python is enough, even without linters and type hints.
Of course it's very easy to create an equivalent of zero struct (object without __init__ called), but do you think it's possible to do it while not noticing that you are doing something unusual?
Go's zero-values are the problem, not any other lack of its type system.
Nobody is suggesting that Coq should be used, so stop bringing it up (strawman). And yes, Coq might have an even stricter and more expressive type system than Rust. But nobody is asking for a perfect type system (no true Scotsman). People are asking to be able to prevent users of your library to provide illegal values. Rust (and Haskell and Scala and Typescript and ….) lets you do this just fine whereas Golang doesn’t.
And personally I would much rather have the compiler or IDE tell me I’m doing something wrong than having to read the docs in detail to understand all the footguns.
My personal opinion is that - even though I’m very productive with Golang and I enjoy using it - Golang has a piss poor type system, even with the addition of Generics.
No, Python is not enough to "...work with a type system where designers of libraries can actually prevent you from writing bugs." Not even typed Python is going to enable that. Only a complete type system can see the types prevent you from writing those bugs. And I expect exactly nobody is writing HTTP services with a language that has a complete type system – for good reason.
> Of course it's very easy to create an equivalent of zero struct
Yes, you are quite right that you, the library consumer, can Foo.__new__(Foo) and get an object that hasn't had its members initialized just like you can in Go. But unless the library author has specifically called attention to you to initialize the value this way, that little tingling sensation should be telling you that you're doing something wrong. It is not conventional for libraries to have those semantics. Not in Python, not in Go.
Just because you can doesn't mean you should.
After playing with Rust, I changed my tune. The type system just forces you into the correct path, that a lot of code became boring because you no longer had to second guess what-if scenarios.
A lot of languages certainly don't make it easy. You shouldn't have to make a Username struct/class with a string field to have a typed username. You should be able to declare a type Username which is just a string under the hood, but with different associated functions.
If you want to implement what Yaron Minsky described as "make illegal states unrepresentable", then you use a username type, not a string. That rules out multiple entire classes of illegal states.
If you do that, then when you compile your program, the typechecker can provide a much stronger correctness proof, for more properties. It allows you to do "static debugging" effectively, where you debug your code before it ever even runs.
Which is exactly what's going on here. A username has a string as a payload, but that payload has restrictions (not every string will do) and methods which expect a username should get a username, not any old string.
Meanwhile the real world usages of this term I've seen in the past have all been things like enums as strings, lists as strings, numbers as strings, etc... Not arbitrary textual inputs from the user.
Typescript, you say?
const bar: Foo = {} as Foo
Your example doesn’t even satisfy the base case, let alone the general case. Get back to us when you have actually read the thread and can provide something on topic.
But let's assume the programmer knows what they are doing and there is no code coming from any other source. When would said programmer write code that isn't deliberate? What is it about Go that you think makes them, an otherwise competent programmer, flail around haphazardly without any careful deliberation?
No, you do. Anywhere the type system is incomplete means that the consumer can do something the library didn't intend. Rust does not have a complete type system. There was no relevance to mentioning it. But I know it is time for Rust's regularly scheduled ad break. And while you are at it, enjoy a cool, refreshing Coca-Cola.
> Go's zero-values are the problem
"Sometimes it's nice to work with a type system where designers of libraries can actually prevent you from writing bugs." has nothing to do with zero-values. It doesn't even really have anything to do with Go specifically. My, the quality of advertising has really declined around here. Used to be the Rust ads at least tried to look like they fit in.
The topic at hand is about preventing library users from doing things the library author didn't intended using the type system, not "what happens if a language has zero-values". Perhaps you are not able to comprehend this because you are hungry? You're not you when you are hungry. Grab a Snickers.
> > > Crazy that actually using your type system leads to better code.
> > There's a name for this anti-pattern: "Stringly typed"
> I don't think a reasonable person would consider storing a username in a string to be "stringly typed".
#1 was saying that the root comment shows better code using the type system.
#2 was clearly referring to the case where you don't do this as being an anti-pattern.
#3 is saying that storing a username in a string, without wrapping defining a distinct type for it, was not stringly typed. But as I pointed out, it certainly is.
If you doubt my interpretation of #3, the same commenter said this in another comment: "Is it really more 'programmer friendly' to create wrapper types for individual strings all over your codebase?"
By having stronger types this won’t come up as a problem. You don’t have to rely on having the best programmers in the world that never make mistakes (tm) to be on your team and instead rely on the computer making guard rails for you so you can’t screw up minor things like that.
I would probably type to the level of Url, Email, Name but not PersonProfileTwitterLink.
That can be handled transparently in languages that have good support for strong type systems, like Rust or Haskell, using traits or type classes.
What you're saying is essentially that addressing stringly typing can only be taken so far in weakly typed languages, without becoming inconvenient.
> Meanwhile the real world usages of this term I've seen in the past have all been things like enums as strings, lists as strings, numbers as strings, etc... Not arbitrary textual inputs from the user.
The definitional question is not that interesting. The point is that the concept applies just as much to a username represented as a string as it does to any other kind of value being represented as a string.
The reason is simple, which is just that "string" is a general type that can represent anything, whereas "username" is a subset of all possible strings. If you're trying to use your type system to ensure correct code, you want to be able to type check a function signature like `f(user, company, motto)`, just to take a simple example.
Maybe it's time for you to finally try rust? Or any other language without zero-values, since rust seems to irritate you in particular.
The original claim was that with go, doing certain pattern "[...] guarantees that you can never forget to validate the username through any codepath". Which is not true. It is not true, because go has its own billion-dollar-mistake called zero values.
In our codebase, we use wrapper classes and the only time we had a defect with this is when one developer got lazy and used 3 primitive Strings in a class instead of wrapper classes. Another developer needed to update the code and populated the wrong wrapper class as they were not as familiar with that part of the codebase. Had the original developer simply used wrapper classes, the person maintaining the code wouldn't have had that confusion.
I get it: You were in such a rush to fill your marketing quotas that you didn't bother to read the entire thread. Maybe the lesson here is don't use HN as an advertising platform next time? You should have known better from the get go.