TIL: You can make HTTP requests without curl using Bash /dev/TCP(mareksuppa.com) |
TIL: You can make HTTP requests without curl using Bash /dev/TCP(mareksuppa.com) |
It's a TCP client
curl is an HTTP client
I prefer TCP clients to HTTP clients. Simpler, easier to modify, faster to compile
There are many to choose from. For example, I use a modified version of tcploop
For generating HTTP, I use own utilties. This is more flexible than curl. There are some things curl cannot do, even though it has too many options
timeout 5s bash -c "echo >/dev/tcp/google.com/443" && echo "port open" || echo "port closed"
Using /dev/tcp was also handy in getting that initial low-priv shell.
I discovered it for myself some years ago, when I wanted to make simple network test scripts run without depending on curl or telnet, or other executables outside of bash.
bash -i >& /dev/tcp/IP/PORT 0>&1
to talk to their friends(' computers).
shame it's not a real device so the surface is limited to bash only
I wonder what software might be vulnerable to this attack surface
This is why we can’t have nice things. This feature is complex and obscure enough that you are unlikely to be able to use it manually without consulting a reference, and poorly supported that any script you write with it is unportable.
Bash is so powerful and so frustrating for this reason all the time :(
Simple get: GET / HTTP/1.1 Content-Type: text/html User-Agent: l33t hax0rs lol X-Funny-Monkey: farts
For sending a mail message on port 25: HELO mail-from: whoever@whatever.com mail-to: sysadmin@yaya.com <other headers> <blank line> Body of the message yay. <two blank lines to end>
POP3 was so long ago I forgot but you could list the mailboxes then get individual messages and so on.
This revelation was the beginning of "there is no magic" for me. The realization that every part of the computer was built by human beings and was at some level understandable if one undertook the effort.
Perhaps most people in the future won't bother. They'll just let agents do it all. I'm sure that will leave some interesting holes in various systems for people willing to actually learn how they work without the filter of a model (or its safety rails).
But at my first work (begining of 2000s) there was one person that made a fun email, using From of head of company (or was it head of that particular division) to his coworker with congratulations for pay increase and promotion. It would be all great, but that coworker didn't catch the joke and replied to it (person in the From wasn't amused). Author of the joke was fired (which is not easy thing to do in Europe), some people don't catch jokes.
> most SMTP servers would accept email from anyone anywhere to anyone anywhere (i.e. 'open relay').
to date that claim, I'd say that by the late 90s at least, true open relays ("from anyone to anyone") were still numerous but carried a huge assumption of being part of spam operations (willingly or through ineptitude), and the most basic spam filtering would reject mail that came out of one.
That said, (before things like SPF) it was easy enough to deliver email to anyone you wanted even if you didn't have your own real email account and SMTP server; you could just look up the destination's MX and connect to it with telnet like that. Since your own random IP probably wasn't blocklisted it would generally be accepted and delivered.
Back then it was still basically considered bad form to reject email simply because the server didn't know where it was from... sadly, if we were still playing by those rules today, I can only imagine how useless email would be. Now it's definitely guilty-till-proven-innocent.
She stood behind me and watched bemused as I fired up telnet, connected to my ISP's pop server and started reading emails from friends. I think I did manage to send some emails back via SMTP but I was not as good with that protocol.
If you could bottle the creativity and enthusiasm of a bored teen, I'm pretty sure you could take over the world
One day I realized even databases were just text files and I had to sit down.
Cat homework.ps > /dev/lp0The realization that (on DOS) "copy con file.txt" as the world's worst text editor or "copy file lpt1:" was treating physical devices as files. Everything on the computer was arbitrary so you could make anything behave like anything else (for some definition of behave like)!
Not a great insight I'll grant but a key one that everyone has to go through to be any good.
You also can't do that with TLS (and a lot of servers won't talk HTTP other than redirects). openssl s_client instead of telnet might allow you to tunnel text inside TLS, but that feels like a cheating.
And many other modern protocols, sadly, prefer binary encoding, which makes it impossible to tinker with it on wire level, not without specialized tools anyway.
I think people in the future will bother. I tried to make a fire with sticks once, I tried to burn a clay brick, these old things can be a lot of fun and sometimes of real use. If anything, AI actually makes tinkering a lot more easier. You don't need to dig into RFC to check your mail, you can just talk to LLM about it and it'll help you with most typical IMAP commands, for example.
You're welcome. Works like netcat plus TLS. Kind of inconvenient though. Hey someone should write tlsnc.
Good times.
But the joke's on him--it led directly to me meeting a lifelong friend & mentor.
I am not sure why this is a revelation. Any college level networking course would cover this?!
As an actual kid it's easy for it to be a revelation, no? At least it was for me, with no college level networking course experience.
But can you imagine the look on some young teen’s face when they train their own GPT on their local computer for the first time?
exec 3<>/dev/tcp/example.com/80
printf 'GET / HTTP/1.1\r\nHost: example.com\r\nConnection: close\r\n\r\n' >&3
cat <&3
HTTP/1.1 200 OK
Date: Tue, 16 Jun 2026 17:37:45 GMT
Content-Type: text/html
...
I open my web browser and go to http://example.com and get redirected to the captive portal page again and retry completing what they need from me to get internet access.
https://gist.github.com/skull-squadron/edb8c0122f902013304c0...
My only concern would be that example.com doesn't promise to never do the 'required SSL' thing.
exec 3<>/dev/tcp/example.com/80
printf 'GET / HTTP/1.1\r
Host: example.com\r
Connection: close\r
\r
' >&3
cat <&3
9front lets you play with that on Linux.
Some Plan 9 like /net things are visible in Go libraries... (Rob Pike legacy)
This is awesome.
No, it can not. Bash lets you open TCP sockets.
What you are doing here is trying to speak HTTP yourself, which is fine for testing and debugging, and hella cool for fun to do by hand, but you will shoot yourself in the foot if you try to use this pseudo http client unattended in reality. This toy code does not parse HTTP properly and will break.
You could of course write a full http/1.1 client in bash, you can even do a full http server in pure bash: https://github.com/bahamas10/bash-web-server
For less insane, non-bash shells there is always nc which is usually probably the wiser choice.
Zsh has its own zsh/net/tcp and zsh/zftp modules.
https://zsh.sourceforge.io/Doc/Release/TCP-Function-System.h...
https://zsh.sourceforge.io/Doc/Release/Zsh-Modules.html#The-...
https://zsh.sourceforge.io/Doc/Release/Zftp-Function-System....
The main surprise was that Bash has /dev/tcp which lets you do the equivalent of an HTTP request with a bit of shell magic, for instance:
exec 3<>/dev/tcp/service/8642
printf 'GET /health HTTP/1.1\r\nHost: service\r\nConnection: close\r\n\r\n' >&3
cat <&3
Pretty cool!
<https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=146464#37>
As others have mentioned, there are numerous other ways to directly access network features using shell tools, including curl (noted in TFA's title), wget, the HEAD and GET commands (from Perl), netcat (nc), socat, telnet, and I'm quite sure others.
FROM openjdk:11-jre-slim HEALTHCHECK --start-period=10s --timeout=3s --retries=5 \ CMD perl -e "use IO::Socket; $sock = IO::Socket::INET->new(Proto => 'tcp', PeerAddr => 'localhost', PeerPort => '8888') or die $@; $sock->autoflush(1); print $sock 'GET /actuator/health HTTP/1.1' . chr(0x0a) . chr(0x0d) . 'Host: localhost:8888' . chr(0x0a) . chr(0x0d) . 'Connection: close' . chr(0x0a) . chr(0x0d) . chr(0x0a) . chr(0x0d); while (my $line = $sock->getline ) { if ($line =~ /UP/) {exit;} }; close $sock; exit 1;"
The routers were very basic model with very limited flash memory (~4MB?). I was brought in to build firmware for those routers. I ended up customising openwrt - removed all kinds of packages to make their packages fit on those routers. At the end, I had less than 4KB space, And I needed to implement a "heart beat" service. A lot of routers were behind firewalls that only allowed http, https and a couple of other protocols. Libcurl was too heavy. So I ended up writing a shell script that used this feature of bash to send out heart beats.
Fun times...
docker inspect -f '{{.State.Pid}}' container-name
# let's imagine that outputs 814538
nsenter -t 814538 -n curl example.com
I was really just trying to see if intra-container connectivity works, and this ended up being a very quick way of doing so. (The alternative being building and deploying a new image, which would likely take significantly longer.)
You said the image was Python, though? Using that is way easier and faster. https://news.ycombinator.com/item?id=48558763
If all you need to know is that it can connect:
python3 -c 'import socket as s;s.create_connection(("8.8.8.8",53))'
or http:
python3 -c 'from urllib.request import*;print(urlopen("http://example.com").status)'
# Connection successful:
$ timeout 1 bash -c 'cat < /dev/null > /dev/tcp/google.com/80'
$ echo $?
0https://github.com/ksh93/ast-open-archive/blame/master/src/c...
Just like parsing HTML with regexes can be fine too - for instance if you know the sender.
Just like repeating code can be fine too, even though it violates DRY.
Mixing markup and code can be fine (call it Locality of Behavior).
But separating markup and code is fine too (Separation of Concerns).
goto’s can be a lifesaver for deeply nested error conditions in C.
The point is all these “you shouldn’t do this” comments are just generalities. Use your judgement, decide if the tradeoffs are right and make a deliberate choice.
Why they didn't lock those down by using different creds per client in the computer lab I still don't know. Maybe it was a VAX limitation (at the time)?
bash-web-server project builds a C language socket listener [0] that is dynamically loaded at run-time as a "built-in" module that makes the functionality available.
[0] https://github.com/bahamas10/bash-web-server/tree/main/loada...
cd src/bash-5.3/examples/loadables
make accept
enable -f ./accept accept
(accept -r RHOST -v SOCKETFD -b 127.0.0.1 8000;
read -u $SOCKETFD SOCKETDATA;
printf "%s: %s\n" "$RHOST" "$SOCKETDATA";
printf "goodbye, world\n" 1>&${SOCKETFD} ) &
nc 127.0.0.1 8000 <<< "hello, world"
Edit: https://news.ycombinator.com/item?id=39369749 (2024)
Very fair pushback -- I did get carried away and will update the article to be more precise. Thanks for raising it!
> For less insane, non-bash shells there is always nc which is usually probably the wiser choice.
For completeness, `nc` or any netcat equvialent I could think of was not available in the image I was trying this with. It would certainly be a better option though.
Common misconception, if you want to replace a dependency on a swiss knife you don't need to implement a swiss knife, sometimes you can just implement the last helix of the corkscrew.
it is insane to use it for anything serious (also the opposite, implementing webservers in bash), but for quick testing it's pretty great!
https://sdomi.pl/weblog/15-witchcraft-minecraft-server-in-ba...
Though I did also notice they didn’t claim it was pure bash themselves. That’s a flare you added.
I thought you had to use a program called netcat for that--if not then what is the point of that binary? And for that matter, can't you also use telnet to manually send HTTP?
So we start at compiling the codebase (Rust) against MUSL. That way we can run it with FROM scratch images.
If we need more tooling available at runtime, then we look at alpine, but still using MUSL.
If MUSL itself is proving problematic, or if some of the libraries we use need glibc then we can look at using some locked down image.
The cool part about FROM scratch images is that you'll never have to update your base image to address CVEs. Only your software and its (compiled) dependencies.
For what its worth, this container used `python:3.12.2-slim-bookworm` and I really would not expect that sort of an image to bundle `curl` -- even if it is intended for production.
In theory it has a couple of benefits. You don't have to re-deploy your image to patch CVE's in OS components if you don't have any OS components. And it provides some measure of defence-in-depth - one could certainly theory-craft a scenario where an attacker gains some limited control over your application and then uses some OS component to escalate.
These days if a security engineer is proposing my team adopt distro-less containers to receive these benefits, I would point out that we need to weigh them against the very real drawbacks of not having standard debugging tools available where and when we need them. And also to consider the relative impact of other defence-in-depth measures they could be pursuing instead - such as any sort of network policy to limit network traffic.
Probably get confusing again when people start referring to 'the 20s' not as the 1920s.
One can have curl as a bash loadable routine without the need to recompile bash. Headless or not, much more sane[4] than ctypes.sh[1] as curl binary epilog/prolog handle the non-http protocol items within the context of curl module/program. More relevant logic level related to topic beyond http protocol only curl : socat[2] and gnu netcat[3].
Ah, HN doesn't provide a way to conjure up * data tapes (comments not withstanding). Ebnf being battle tested for years & grepping the ANTLER[7] by the horns:
awk interpeting http stream via http protocol[5] via ebnf of http protocol likely to better handle the different protocal layers more gracefully than bash script. Would need to make sure awk is in bit mode and not character/byte mode to avoid line discipline issues / 'stty sane'[4]. Note:
Perl/python provide better layers of abstraction than shell coding for the different levels of protocols used in/with http protocol/ ebnf,Kleene* coding.[6] In shell, having a different command programm for every ebnf set/protocol level vs. awk, different ebnf for each Kleene* protocol level)
---------------------------------------------------------------------------------------------
[0] : Your trerminal is not a terminal: An Introduction to Streams : https://www.lucasfcosta.com/blog/streams-introduction
[1] : ctypes.sh : https://github.com/taviso/ctypes.sh
[2] : socat : https://github.com/3ndG4me/socat/releases
[3] : gnu netcat: https://netcat.sourceforge.net/
[4] : stty sane : https://real-world-systems.com/docs/stty.1.html
https://jaredgorski.org/notes/unix-tty/
api, GET url, content-type aplication/json, parse json
you can even invert it and make a server
There are rules now, but the concept is still almost intact, random people writing to the servers disk - to be later read by someone
I think it's neat that you can still find echoes of this. MAIL worked by just appending to MLFL, separating records with CRLF.CRLF - which is still how Data segments are terminated in SMTP.
Yup, this was also a fun exercise. Use nslookup, find the MX records, then telnet to them and deliver some mail destined for domains that email server handled. At that point you're just a slower version of a mail daemon.
In 2006 when FIOS first rolled out they assigned ARIN IP blocks to anyone who requested a static IP and I hosted my own webserver and email server on the web. So for a long time I had my own netblock and my email server had been around for so very long it was in everyone's legacy trusted list. Even gmail never bounced emails from me, despite not having SPF or DKIM and whatever else. Though I did eventually set those up. Only authed users could send email and that was a limited list so my netblock never actually delivered a spam message.
Funny enough I also setup default routing so all non-registered addresses went into a separate mailbox for me and I used the "companyname@mydomain.net" for everything. For many years I knew about every single data breech before anyone else. Often months or years ahead as spam would arrive out of nowhere to that address.
When I finally moved to the bay area to work in tech for real the entire world had changed. You couldn't just get a netblock assignment anymore. And lord knows even if your ISP wasn't deliberately sabotaging your ability to run a server every other system on the internet would assume you're a scamming spammer. I had to give up self-hosting on-prem.
(Not that I'm claiming anything's original here, ftp & smtp are both in the nwg/ietf family tree, which makes it easy to draw parallels. There's probably 100 other influences.)
- http://connectivitycheck.gstatic.com/generate_204
- http://detectportal.brave-http-only.com/
Plus, it feels nice to depend on the reserved domain name example.com instead of relying on a domain that any one specific corporation has to maintain :D
Add an ephemeral container to an already running pod, for example to add debugging utilities without restarting the pod.
https://kubernetes.io/docs/reference/kubectl/generated/kubec...
Keeping in mind that containers are merely a bunch of namespaces, there's nothing stopping you from entering the same PID namespace with a different mount namespace in order to debug.
To summarize, in my experience there is immense value to having basic shell tools available in the environment where you need them with zero extra friction. Stripping those out provides a security benefit only in specific nebulous and niche scenarios.
I agree, however assuming you maintain a chroot for debugging this can be accomplished with a shell command that takes a single argument to target a running container by name.
Your linked comment suggests being limited to kubernates but nsenter and a chroot are entirely runtime agnostic.
Reminds me of the time on libera IRC when someone told me "cloud storage does not exist" because they were hung up on some ultra-purist word definition that nobody else shared.
As the calendar rolled from 1999 to 2000, we entered a new millennium, century, decade, year, day, ...
Every Jan 2 I start saying "last year" and every Dec I say "see you next year"
no, that all happened when we rolled from 2000 to 2001.
smh, even paedants today aren't what they used to be.
I think that's more or less when I lost faith in humanity.
That probably wouldn't be a useful distinction because almost every module can be built in
What's the benefit really, though? If you still need to be able to rapidly deploy a new image in response to a dependency CVE, what have you gained?
Add an ephemeral container to an already running pod, for example to add debugging utilities without restarting the pod.
https://kubernetes.io/docs/reference/kubectl/generated/kubec...
But when Docker scans my image and notices that there is a CVE in one of those binaries, my image is currently out of compliance.
FROM scratch just reduces the surface.
The actual attack surface of your application? Or the attack surface of you and your team's attention from a busybody security org.
It's important not to confuse the two.
> to not have any dependencies outside of the code.
> ... FROM scratch images is that you'll never have to update your base image to address CVEs...
So a FROM scratch image, basically doesn't have things like a package manager to install things, and maybe also libraries that things like curl would depend on? Sorry for my ignorance, I've heard of FROM scratch but never tried them.
If you want to run as another user, you need to manually add an /etc/group & /etc/password (or generate them in a stage before that and copy them over).
If you need ca-certificates, you need to install ca-certificates in a stage before that and copy over /etc/ssl/certs/ca-certificates.crt from that stage to your current one.
Was grandparent comment written by an LLM?
Or is this a human who copies a style they saw in a blog post, unaware that they’re copying an AI?
Or is this a human who spent too much time talking to an AI and now they just talk like this?
Or is this an organic human response and we’re all paranoid by now?
I don’t know which would be worse.
For many in the next generation of language learners, this reference will be Claude.
But certainly with smarter AI I do believe it'll become more fluent with choosing more diverse idioms and phrasing, rather than repeating one thing over and over, to a point of being a comically similar. So people who learn on AI-generated text, will not learn from just one recurring style.
(For what it's worth I did write the message above manually but I understand why no one would believe that now. At least I did not call netcat "load-bearing" [https://mareksuppa.com/til/load-bearing/] or something...)
Before that would just made you top 5% (or maybe top 1%) of the nicest people to talk too.. know ppl think you are Claude.
We are all going crazy s a sibling comment said.
I notice myself getting afflicted with llm-isms after a full workday. And I didn't always notice, sometimes I only realize the day after...
Like it slowly siphoned out my soul, which then reconnected with me over night
export http_proxy=http://your.proxy.server:port/
export HTTPS_PROXY=https://your.proxy.server:port/
curl -x http://proxy_server:proxy_port --proxy-user username:password
or $socks-wrapper curl # [2]
[2] - torsocks, tsocks, wireproxy, shadowsocks-rust, proxychains-ng, etc...
and a server behind it like
``` mkfifo /tmp/myfifo cat /tmp/myfifo | nc -l 12345 > /tmp/myfifo ```
so if you manually type out
CONNECT host:12345 HTTP/1.1
host: host:12345
curl -x proxy:3333 telnet://host:12345
telnet was always there though. it also worked for speaking all the other plaintext internet protocols. (imap, pop, smtp, etc)
Use nc or this TCP Bash technique if you really want to ensure decent compatibility when doing hacky solutions, otherwise a random 0xFF somewhere from a terminal console color change (or other control character) might really screw you over.
EDIT or ya know, use the correct tool like Curl.
Like if my server replied with ‘HI PLEASURE TO MEET YOU 127.0.0.1 THAT NAME SOUNDS FAMILIAR ARE YOU BY CHANCE FROM BOSTON MY MOTHER IS FROM BOSTON WELL QUINCY ACTUALLY BUT DO YOU KNOW 127.0.1.1 THEY ARE A REALLY GOOD FRIEND OF MINE YOU SHOULD MEET I HEAR THEIR DAUGHTER IS A DOCTOR DONTYAKNOW AND YOU COULD…”
etc, etc?
smtp grew up to be an antisocial curmudgeon. extended smtp starts with EHLO.
When there is no corresponding level of restraint in the libraries that we add to most applications, does it really make a difference to leave out the likes of curl, nano, ping, etc compared to how frustrating it is to operate in just busybox (etc)?
I'm not just ranting, I'd actually like someone who swears by always shipping alpine images (etc) and never installing any basic utilities in them to share their reasoning.
Thanks to `kubectl debug`, you don't need to install debugging utilities into your production image.
The current year counting was based on the same way years were counted in the past, so you had things like "the 10th year of Caesar's reign". So the year 1 A D. was the first year of Christ's reign.
The year 2000 was the 2000th year of Christ's reign so that's what is celebrated.
Alternatively, you don't really care about that and it's called Common Era now anyway, but in that case it's entirely arbitrary. So either way, calling 2001 the New Millennium is wrong!
The first century is years 1 through 100. Therefore, the second century is years 201 through 300.
The first millenium is years 1 through 1000. Therefore, the second millenium is years 2001 through 3000.
There is no year zero.
The hundredth year of Christ's reign is the year 100.
The thousandth year of Christ's reign is 1000.
The two thousandth year of Christ's reign is 2000.
There is no year zero.
And even that's only true if you assume kubernetes is the only place your container runs where you might want to also debug it.
How wasteful though? I have to admit, I envy the person whose codebase itself they have to support is so lean and space-conservative that the size of the gnu coreutils, curl, nano, etc., would show up as anything but a rounding error in the image size.
I see it like putting a thermometer in a turkey before I stick it in the oven. Sure, the thermometer adds thermal mass itself, making the turkey take a few seconds longer to cook, but the value of it being there is greater than the cost imposed.
For the rare cases where you find yourself needing to attach a debugger to your pods running in staging/prod, a debug container is absolutely the right tool to reach for.
The amount of languages are decreasing on the earth, I would also say that dialects and accents are decreasing as well. I think this is a problem.
I need to try this with a Squid SSL Bump MitM proxy just dont have one up at the moment.
curl -vvv -A Mozilla -H "Accept-Language: en_us" -H "Sec-Fetch-Mode: navigate" --url 'https://nochan.net/.env'email will become so unusable, next one will have to be HELNO i guess
"EHLO" still sounds friendly. It just sounds like a different accent or something. Know someone that used to answer calls with a friendly "Jello?".