It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - IIRC some clients measure average packet latency and if it starts going up, put some backpressure), resolve magnet URLs, set up port forwarding, reconfigure firewalls, offer API for *arr stack, be a good netizen (report stats correctly, send packets within the specs, do not spam - otherwise other clients will blocklist your in their code or config), implement many BEPs (mutable torrents are cool), be able to recover from interrupted state based on only the data that's on disk, have configurable downloading order (people want to start playing videos before they finish downloading, so you may want to e.g. download header and footer of each file first), and ideally detect duplicates between torrents (cross-seeding). And then there will be people throwing 2TB+ torrents at it (e.g. TLMC) to benchmark it, and saying your client is "literally unusable" if it doesn't handle it.

So, building a "perfect" torrent client from the ground up is a daunting task. But the "good" news is that nobody built such a "perfect" client just yet, so if you have some spare months of your time, you can take a shot at it. Or even better yet, open the issue tracker for one of popular clients or libraries, and add one of the missing features from the list above.

This looks simple not only because the article is written well but also because Go is the go-to-language for complex networking situations. Doing things in parallel, even pipelining? This would make quite some spaghetti algorithm in C/C++, even async rust/python world would not look so clean as in Go. This is clearly a big strength of the language.

Very cool, and surprisingly simple. I was always mystified as a kid how these file-sharing protocols actually work, but if you just look at a torrent as a collection of fixed-size pieces you are asking peers managed by a tracker for, it becomes really straightforward to implement.

Now I wonder how clients protect themselves against abusers (i.e.. people who never upload a single bit but only download). I often noticed that when I set the maximum upstream to 0, clients would stop sending me pieces. Do clients share a predefined list of configuration parameters with each other or with the tracker (Max upstream, max downstream, etc.)? Or is it something more sophisticated?

CodeCrafters has a BitTorrent client building challenge that I watched Jon Gjengset do on a stream, if someone wants to try and build their own client in an iterative and checked way.

https://app.codecrafters.io/courses/bittorrent/overview

I was just thinking about how incredible BitTorrent was the other day, how it really does outperform non-distributed solutions in many cases and how it is one of the rare examples of that.

One thing I was thinking though was that the finding of Peers seems to be the bottleneck - that if peer resolution could be almost instantaneous, the BT protocol could be used for so many more use cases.

Does anyone know if this part of the process could ever be improved or does that just come with the territory?

This looks surprisingly simple. A perfect project to learn programming language by creating thing you can interact with and not get overwhelmed by domain complexity.

Do you know more such problems?

One thing that I find hard to understand is how you're able to connect to a random peer online with just an IP and port combination. The peer is likely behind a NAT proxy, and never communicated with you to begin with. In my head, NAT shouldn't allow this connection through, or be able to associate the port with the individual peer machine since the connection source (you) is arbitrary.

This is possible with port forwarding. But that's a niche set of peers, who have the power to configure port forwarding on a NAT proxy.

This was post number 5 when I started reading it and 45 when I finished, I am guessing some people from the RIAA and MPA found their way to the flag button.

i wanted to do the same when i got fed up with bt clients. but then i found there already are go clients and i found i like qbt. but it is a good read nevertheless.

Sufficiently evolved torrent clients follow a tit-for-tat protocol:

- If you're nice with me I'll be nice with you

- If you're mean with me I'll be mean with you

This is the best answer for the Prisoner's dilemma (https://en.wikipedia.org/wiki/Prisoner's_dilemma) on the long run, ie a situation where peers don't trust each other but will both gain if both cooperate.

In bittorrent it's typically implemented as follows: peers start by sending a very minimal content, and see what the other replies. If they reply with low enough latency then slowly increase the amount of content that is sent and see if they reply with the same increased amount of content; if they do, continue up to the max of what the network link allows (in combination with other peers of course). If at some point the other peer doesn't send something equivalent (even though we know they have it and we asked for it) then that peer can be cut off for not being cooperative.

Situation is different for seeds of course, because they have everything and want nothing, but they can have a similar behaviour -- start sending a little, increase slowly over time

> Now I wonder how clients protect themselves against abusers (i.e. people who never upload a single bit but only download).

It's called leeching, and it depends. It's typically considered good etiquette to upload as much as you download and that's usually enough, but it can be enforced

https://en.wikipedia.org/wiki/Leecher_(computing)

IIRC, when both peers only have some subset of pieces, they are meant to “trade” with each other, and so will eventually drop low value peers for higher value ones. Aside from the adversarial case, this helps optimise for being nearer to the people you are sharing with, since torrents are global.

In practice, BitTorrent really needs seeders who have downloaded the entirety of the file to be fast for everything except really popular downloads, Seeders don’t really check for fairness and will typically upload to whoever they can the fastest (with a limit on number of peers).

There’s an even more adversarial case because the unit of validation (a piece with a fixed hash from the spec) might be bigger than the chunks that are being shared individually. So it’s possible for a peer to fake having pieces and upload garbage data instead, and they wouldn’t be caught since different chunks came from different peers.

There's private tracker which forces their user to upload.

Most torrent clients I know support UPnP and/or NAT-PMP to request port forwarding from routers. As you say, it just isn't going to work for many people without this.

I run a BitTorrent service for an academic institution, to disseminate research data. We have a regular routable IP address, but still need to navigate the institutional firewall.

There are a couple ways to do this:

1. Manual port forwarding, it's likely that there are at least a few power users who already have the torrent and are seeding who have this set up. Seedboxes are a notable example, they're often simple servers that actually have a public IP.

2. UPNP, a protocol that lets you ask your router to set up a temporary port forward for you. Again, not all peers support this, but some do, and you can just connect to those.

3. Hole Punching. Imagine Alice is sending data to Bob, and her router ends up sending it from port 1234. Her router needs to send the packets it receives on port 1234 back to Alice's computer, to allow her to receive Bob's responses. Some routers will do this no matter which IP the packets are coming from. If Bob tells alice her router is sending from port 1234, she can spread that fact to others and let them contact her that way.

I don't know if BitTorrent clients take advantage of this specifically, but it's a very common way of doing NAT traversal in general.

NATs is why private trackers have the concept of "connectability", if you're "connectable", it means you can accept connections from other clients. Crucially, if just one of the peers is connectable, they can both communicate, so connectability is heavily encouraged but not required.

Most clients try to set up port forwarding using UPnP IGD, NAT PMP, or PCP protocol, a lot of residential routers support one of them.

If you enable WebTorrent as a transport protocol (enabled in gotorrent, disabled by default in libtorrent), it should be possible to use existing public STUN/TURN infrastructure, but I don't know if any client does it yet.

In practice, you just have to accept that many connections will simply fail, and make your client move on to try a different peer.

There's a less commonly known feature of NAT (maybe called cone NAT or something), where the remote endpoints are not checked for incoming packets and connections. You communicate with trackers and DHT and as usual your outbound packets are mapped to some port. But everyone use sees your public IP and port and talk about your client using that pair. They then also communicate with you over UDP to your public port. As long as you have regular outbound traffic running through that mapping, NAT will keep the hole alive. I think with testing I found at least half of people on NAT had this. It's less common on mobile and fibre.

The other answers are correct about only needing a single peer among any pair of peers to be addressable.

However there is a built-in hole punching mechanism in BitTorrent where peers ask for a third peer to assist in hole punching.

It's implemented in my client. It was very painful to implement. I think someone privately funded the feature which was very nice.

the answer to this problem in general is NAT hole punching, but BitTorrent doesn't actually have an answer to your problem. if you are behind a NAT, you can only connect to peers that are not behind a NAT or have port forwarding set up. for popular torrents this is good enough because you don't have to connect to all peers.

> This is possible with port forwarding. But that's a niche set of peers, who have the power to configure port forwarding on a NAT proxy.

yes it's niche but I guess this means BitTorrent isn't as P2P in practice as one wants it to be, but held up by seedboxes.