Ask HN: Do you set the user-agent on your scripts?

3 points by kfdm 5 years ago | 2 comments

Typically I try to remember to set the user-agent on tools/scripts I make, to avoid default agents like `Go-http-client/version` or `python-request/version` but it seems like a lot of projects do not always set it.

Sometimes I will submit patches to projects to fix this, though sometimes I wonder if I'm in the minority of developers who prefer to properly set user-agent.

(I am fully aware and agree that the user-agent for most web browsers is terribly complicated and almost useless at this point. My focus is more on bots/scrapers/etc)

pitched 5 years ago |

There’s a dev-time vs. reliability debate here, I think. I don’t think it’s worth worrying about most of the time but I would approve a PR if someone did do the work.

When scripting things that probably shouldn’t be scripted but I care about them (like covid stuff), I’ll take the time to find a good browser user agent. I would be worried about bot-prevention measures catching a non-standard user agent and taking it down. Or even worse, someone else made a terrible script posting dos-level traffic and mine gets banned along with it...

The next level is using a custom but specific-to-the-script user agent. I do this whenever I also own the backend, to make tracking down logs way easier.

For everything else, I use the default! In this case, taking the extra time to think of an appropriate user agent isn’t helpful to the project. Sometimes, I might still send up Star Wars references for fun.

alexeldeib 5 years ago |

Production-grade software: yes.

Hobby: not always.

It’s incredibly useful, IMO. Not everyone knows about it, or strictly needs it. But I think it’s very useful.

In Go, I often find a lower level SDK-type library will let you append to the user agent, which for large web services is nice since you can basically see how all your callers are using your API (in my case usually looking at volume of calls from language SDKs vs raw HTTP calls vs CLI vs 3rd party tools)