Endlessh: An SSH Tarpit

290 points by elliebike 5 years ago | 105 comments

xoa 5 years ago |

Are tarpits still of use these days? I sort of figured that even modern script mass attackers have gotten professionalized and sophisticated enough that they can deal with trivial timeouts and the like. I could see actual honeypots still being of use for researchers or blue teams at organizations that are real targets, and ML might even open up some interesting new ways to make those more engaging for longer. But a tarpit doesn't seem like it'd cause bother for drive-by or APT, the former are all about volume so if something takes more than a few seconds just skip it (and maybe flag it as a tarpit for punishment) and an APT will instantly recognize it too.

For individuals and smaller orgs I've sort of felt like keeping your head down, running a wg/ssh bastion with a non-standard port maybe along with single packet auth or even plain old port knocking to reduce log spam from random drive-by is more effective and attainable for places without any sort of dedicated security or even constant in-house IT staff. Running a tarpit on a VPS seems like it'd fail to bother most these days, and running it on an actual IP seems like at best it'd have no effect and at worst if it ever actually held up a scanner and the operator noticed they might decide to direct some actual attention to that IP, or at least throw a mild ddos at it for a bit. Am I wrong or out of date on that? I'm all for sticking it to bad actors and efforts to reduce the economic incentives, but in 2020 tarpits strike me as kind of obsolete with some risk to boot.

belorn 5 years ago | |

For individuals and smaller orgs the easiest and by experience the best practice is to use a certificate (or generated and never to be reused password) for ssh authentication, install server monitoring, and then simply observe if the spam from random drive-by causes enough resource drain that would validate further work. Most likely it won't.

Running a tar pit is a bit like installing a trap on a bike in order to teach bike thieves a lesson. It won't really reduce the problem, but for a lot of people the idea of vengeance gives a bit of a warm happy feeling.

toyg 5 years ago | | |

> Running a tar pit is a bit like installing a trap on a bike in order to teach bike thieves a lesson.

It's more like approaching a thief and persuading him to steal some bike "just around the corner", then guiding him around endlessly. While he's following you, he's also not stealing anything from anyone, his attention (which is naturally finite) gets drained - even just a little bit - to the benefit of the community as a whole. It's not necessarily about vengeance.

dheera 5 years ago | | |

> installing a trap on a bike in order to teach bike thieves a lesson. It won't really reduce the problem

Maybe it is reducing the problem, but not enough people are installing traps to make a noticeable difference? Or the number of new thieves is cancelling out the number of thieves being put out of business by traps? Is there data for this?

Or maybe the traps just aren't sophisticated enough?

LinuxBender 5 years ago | |

Yes, well, mostly just for entertainment. My sftp server acts in a weird way like a tarpit. Instead of tarpitting or blocking the bots, I create accounts for them with null passwords. I was hoping they might upload something interesting, but no... they just try to get a shell or try to forward ports to other sites, which I do not allow. Some of them have been connecting several times an hour for the past several years.

narrator 5 years ago | | |

When I used to run my own MX on a home server, I would have bots connect and try to send mail to <random whatever>@mydomain. They never used the same email or IP and they did it for several years straight. I tried IP blocking, but the IPChains list got so long it started slowing down my server.

floatingatoll 5 years ago | | |

If you rewrite those portforward destinations to your own server, do they get stuck in an endless loop?

chromaton 5 years ago | |

I've successfully used an HTTP tarpit to cut down on registration spam. The attacks were being launched from only a handful of (presumably compromised) hosts. If I blocked them, they'd switch to a different attacking host. But I discovered if I tarpitted them, they'd be slowed down to the point where they weren't a problem any more.

jchook 5 years ago | | |

How did it work?

pnutjam 5 years ago | |

Don't forget fail2ban or something similar. 2 hour lockout after 4 bad tries does wonders to discourage bots.

boring_twenties 5 years ago | | |

I don't like it because it opens up the possibility of someone on the same network as me locking me out of my own server.

Sure, it's unlikely, but I don't see what I'd be gaining using fail2ban in the first place. I don't leave password authentication enabled, of course.

Log spam is a bit annoying, but at the end of the day, who cares? Even with the ongoing attempts, my authlog is like 300K uncompressed today and 60-120K per day gzipped. Whatever.

If I cared about that I would prefer to just block Chinese IP ranges outright.

protomyth 5 years ago | | |

Well, if you use certificates, you can immediately ban anyone trying to do password authentication.

jdc 5 years ago | |

Depends on your threat model and how playful you are.

spc476 5 years ago | |

About 15 years ago, I ran a tarpit at my then current job (small ISP for commercial clients and web hosting) that tarpitted all TCP traffic on unused IP blocks. It did slow down network scans and had I gotten around to it, I could have probably set up some way of automatically blocking the IP addresses hitting the tarpit (as no legitimate traffic should hit it).

pbhjpbhj 5 years ago | | |

fail2ban would be the go to solution for blocking IP address used in failed port access attempts.

kachnuv_ocasek 5 years ago | |

> if something takes more than a few seconds just skip it (and maybe flag it as a tarpit for punishment)

Good. Isn't that the point? Also, I like your use of the word punishment as something negative in this context.

cornstalks 5 years ago | | |

> I like your use of the word punishment as something negative in this context.

Some people decide to launch a DDOS attack or something like that in retaliation. It doesn't always happen, but there have been instances of an attacker being thwarted and then trying to punish the victim (who successfully defended themself) in some other way.

DarkWiiPlayer 5 years ago |

Reminds me of the dungeon I built for web crawlers to have fun collecting email addresses at https://darkwiiplayer.com/bot-dungeon xD

tptacek 5 years ago |

I'm sure this was fun to put together and it seems like it's fun for people to talk about, but you can put this along with fail2ban, port knocking, and nonstandard SSH ports in the back of the attic and just (1) turn off password authentication entirely and (2) put SSH behind WireGuard. Even if you don't do step (2), step (1) eliminates the rationale for all the silly stuff people do to obfuscate their SSH installs.

teddyh 5 years ago | |

I you put it behind WireGuard, why use SSH? Why not simply use telnet instead? And use FTP for file transfers.

Drdrdrq 5 years ago | |

Could you elaborate on WireGuard part? Do you mean that users must first VPN, and only then can SSH, or something else?

tptacek 5 years ago | | |

Yes. This is how SSH access to prod works in most large companies: you have to be behind the VPN to get it.

Lex-2008 5 years ago |

discussion of a blog post about this tool: https://news.ycombinator.com/item?id=19465967

Freaky 5 years ago |

One I made in async Rust: https://github.com/Freaky/tarssh

I currently have 22 clients stuck it in across three machines. When I started out it was more like a thousand, so seems they've largely adapted.

nickcw 5 years ago |

Great idea!

I'm not sure we should be writing new network connected daemons in C though.

klodolph 5 years ago | |

> I'm not sure we should be writing new network connected daemons in C though.

In general, yes. However, in this case--no, that's not helpful advice--because this program doesn't actually receive input from clients! Kind of hard to trigger exploitable behavior on a program that only sends output.

codeulike 5 years ago | | |

Kind of hard to trigger exploitable behavior on a program that only sends output.

It wouldn't suprise me to find there were still possible exploits

kmbfjr 5 years ago | |

Your concern is well founded, but what are you going to use that doesn't end up touching libraries written in...C?

We're a long way from "Smashing the Stack", people are aware of mitigation and the care that needs to be taken, precautions have been made inside operating systems and compilers.

MCOfficer 5 years ago | | |

I take the bait... rust.

young_unixer 5 years ago | |

Until there's a better alternative to C at its level of performance, people will keep using C.

geocrasher 5 years ago |

I have to admit that I tried this and it was rather lackluster. Log output:

https://pastebin.com/4FTHRF3f

Not a lot of activity over the time I ran it, and I know that the port gets hit more than that. I had a much better time when I ran a honeypot with Kippo:

https://github.com/desaster/kippo

It was much more useful as it gave me a great list of IP's to block from all my systems ;)

mdaniel 5 years ago | |

The top of the readme for that repo advises to use the fork: https://github.com/cowrie/cowrie

k33n 5 years ago |

The tarpit approach is a double-edged sword. Sure, you're keeping some script kiddie's machine locked up (maybe), but you're also keeping socket connections open and wasting resources on the machine they are targeting. A much more efficient approach is using fail2ban and a firewall to just drop traffic from offenders.

mtlynch 5 years ago | |

Tarpits aren't really a defense mechanism. They're meant to waste attackers' time and study their techniques, making attacks more expensive.

It's sort of like those YouTube channels where they waste phone scammers' time in an entertaining way. [0] Obviously, the easiest thing for the callee to do is hang up the phone, but their goal is to make phone scams less profitable.

[0] https://en.wikipedia.org/wiki/Jim_Browning_(YouTuber)

a1369209993 5 years ago | | |

> where they waste phone scammers' time in an entertaining way.

This can also be automated, so the defender doesn't even need to waste their own time on it. Eg: https://old.reddit.com/r/itslenny/ .

spc476 5 years ago | |

Depends upon how you are doing the tarpitting. Back when I did this (15 years ago) I used a program that created a raw socket to handle all TCP traffic that just responded with 0-byte sized windows to all TCP packets (so overhead was minimal). I placed this software on a dedicated server (an old box that wasn't being used otherwise) and routed all our unused IP space to this system. It could keep thousands of connections "live" with minimal resources.

prussian 5 years ago | |

You could also do a combined approach with the tarpit + fail2ban parser that could just ban people stuck for longer than x amount of seconds.

golem14 5 years ago | | |

In my experience, fail2ban does only help if there is a small number of IP addresses requests come from. I usually observed a trickle of requests from huge number of IP addresses, at most 1-2 request from each IP over the course of days.

tarpit will likely hurt yourself as the system ties up sockets for a long time and you'll run out eventually. You'd have to combine the tarpit with something to limit the number of connections you accept.

IMO, setting up ssh on another port has been useful, especially combined with port knocking. And of course turning off password auth.

dclaw 5 years ago |

Hah, I love endlessh.... been running it for a few years now on one of my digital ocean droplets. Better to fuck with these bots. My personal record was somewhere around 23 days having one stuck.

nirui 5 years ago |

What got my inspired here is, if a simple delay strategy can make attack harder, why not add this as a common feature in SSH?

It can be called "Initial Connection Delay": Once a new TCP connection is established, wait for an uncertain number of n seconds before read and respond to the handshake request.

password4321 5 years ago |

One of the simplest ways to block unwanted connections is to filter on client id. I haven't seen anyone willing to change it even though I've blocked libssh, sshgo, and paprika.

Of course, this functionality is only available in non-standard SSH servers such as the one from Bitvise.

clon 5 years ago |

This is like a self-administered "slow lori attack" then - making it easier for an attacker to keep connections up until things start getting tight on port 443.

heavenlyblue 5 years ago | |

I can imagine this is so easily overcome by the attacker. Why would they even need machines that take 10 seconds to return a single line over SSH?

ivanbakel 5 years ago | | |

Tarpits trap dumb animals. An intelligent attacker won't fall for it, but they aren't meant to.

earthboundkid 5 years ago |

https://github.com/carlmjohnson/heffalump

ryankrage77 5 years ago | |

This seems like it would use a lot of bandwidth?

earthboundkid 5 years ago | | |

Oh, it’s a terrible idea. It’s basically a practical joke.

seqizz 5 years ago |

I'd rather have a trusted common list of known abusers' IPs. But I think that's harder to maintain.

mortehu 5 years ago | |

That's called a DNSBL, and there are many of them, mostly for email spam though.

https://www.dnsbl.info/

geocrasher 5 years ago | | |

What I find infuriating is when an admin gets the bright idea to block all connections from IP's that are in mail dnsbl's. It's a great way to alienate people from using your services. I can't remember which company was doing it, but a customers API calls to a vendor failed because the vendor blocked the IP, which was blacklisted somewhere.

Here's the kicker: The server wasn't even used to send mail and hadn't been for a long time. So we had to apply for a delisting from a mail blacklist for a server that didn't send mail so that a customer could use an API.

The admin thought they were being clever, but instead they were just being difficult.

beatrobot 5 years ago | |

There's https://iplists.firehol.org/

verroq 5 years ago |

This would have been fun to put onto production machines. We had a botnet that was running ssh bruteforce with 10s requests per second with unique IPs. It stopped after we disabled password auth.

creeble 5 years ago | |

Wait, I think I'm an idiot - does disabling password auth entirely prevent openssh from generating a password prompt?

VWWHFSfQ 5 years ago | | |

yes