# in sshd_config:
AuthorizedKeysCommand /usr/bin/php /etc/ssh/auth.php %u
# in /etc/ssh/auth.php
$user = $argv[1] ?? '';
$user = rawurlencode($user);
echo file_get_contents("https://gihub.com/{$user}.keys");
This saves a lot of trouble when off/on-boarding folks, since you can simply add/remove them from a github group to revoke or grant access to your machines.
In theory it's kinda nice because it can let you do fancy things¹, but my actual experiences with it breaking basic functionality even for people who don't use those fancy things has ultimately made me trust Amazon Linux less.
It was especially frustrating because when I first encountered this, I was trying to SSH into a box owned by one of our cloud-first DevOps guys. I couldn't diagnose the box because I didn't have hands on it. He couldn't diagnose the issue because he knows AWS better than he knows Linux and didn't know where to look. He'd chosen Amazon Linux because it's by the owner of the cloud platform, so it must be 'more compatible', right? But here, 'more compatible' actually meant 'more full of stupid surprises'.
Bleh.
--
1: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-...
Other languages I know would require imports/boilerplate that would distract (Go/C#/C/Python/Scala/JS) from the example. Bash might be more familiar with devops but less familiar to regular programmers — part of the illustration I wanted to make was that it didn’t need to be a devops thing. Also PHP (along with JS) is a language most devs know a little bit about, whether they want to or not.
Can valid Unix usernames have special characters that need escaping? I should know that by now.
This may seem inconsequential, but IME when changing defaults in, e.g., /etc/ssh/sshd_config, people and software tend to append their changes to the end of a file or directive block, not the beginning, expecting those changes to be effective. Even security companies and organizations get this wrong, including various SSH bastion products I've seen. CIS Benchmarks recommendations (IIRC) and most (all?) third-party CIS audit suites don't consider precedence at all or get it wrong--e.g. by recommending appending a value, or by providing a compliance test that accepts a broken configuration. FWIW, the proper way to check whether an OpenSSH configuration directive is defined as expected is to use `sshd -T` or `ssh -G` to dump the derived, internal configuration, not by directly inspecting the configuration file(s).
This is the most important, succinct statement made in this piece. -L and -R confused me from the get-go. Having which port instance L or R is "local" change is in some ways, annoying. I "get" that -L and -R change the direction of intentionality, where initiator and responder are, but I think it might have been sensible to make a port:address:port phrase ALWAYS refer to local:binding:remote and have -L and -R define which was listen and which was send.
Then the rest (the host:port) is just the normal way to tell where to connect.
Since we're doing port forwarding over an SSH tunnel, it's obvious that the host is contacted from the other side of the tunnel than where the listening port is.
The big thing with it is that you don't have to do a full auth for subsequent sessions - nice if you don't have tmux (etc) on the remote, and do multiple panes via multiple terminal windows. Particularly when auth involves a passphrase and hsm touch or similar that can take several seconds.
It also has a "connection persistence" setting so when you're bouncing around between a handful of servers you don't have to auth each and every time you switch between servers.
Overall I think of it as one of those features that's nice to have, but not really life changing or anything - Some servers I connect to have it turned off and I notice it's absence more than I notice when it's working.
More info: https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing
# in ~/.ssh/config
Include config.d/*.conf
# in ~/.ssh/config.d/work.conf
host work
hostname myoffice.example.com
user myuser
# in ~/.ssh/config.d/client1.conf
host client1.dev
hostname dev.client.example.net
user someuser
host client1.prod
hostname prod.client.example.net
user someuserFor example, add `host *_work ` and then some stiff that is the same for all work hosts like host1_work.
# in ~/.ssh/config
Host proj1.*.corp
Include ~/.ssh/proj1.conf
# in ~/.ssh/proj1.conf
...
-t is a cool trick, didn't know about that one.
An important note that's easy to overlook in the ~ escape command list is you can nest the escape when in nested sessions (i.e. if you're not using -J for whatever reason).
Cool list, it definitely lines up with what I've found useful and had a few more.
$ ssh -t my-dev-vps 'tmux new-session -A -s main'
So each time I run it I'm right back where I left off.
That kinda still is a problem when you have multiple shells open to the target server. I wish SSH exported it in any reasonable way aside from trying to get it myself from the process list...
ssh host.example -N -D 1080 && wall "Socks proxy to host.example closed."
Why did rlogin, rsh use tilde? because cu used it.
Why cu? Because if you had a modem or serial line, cu was the way you talked to it, to send Hayes codes, and you can't use Hayes codes breakouts because they will break to the modem, so you need a signal to break to cu.
Why not ^[ ? Because thats telnet. so if you had telnet to a host, to connect to the modem over cu, you needed a distinct break-back for cu, to not break back to telnet.
Its breakout syntax all the way down.
Also, its not actually tilde, it <cr>tilde
This section starts out talking about how the command uploads your public key then seamlessly switches to saying it uploads your private key (which I am guessing are typos).
Also that command does not just upload the key, it appends it to ~/.ssh/authorized_keys which is considerably more useful.
Finally, in the ssh-keygen section, ed25519 is from everything I’ve read preferred these days to ecdsa.
I didn't bookmark that post, and I haven't been able to find it again to my great dismay. If anyone remembers this post and still has it, I'd love to read it again.
You might be interested in how to limit users to nologin shell coupled with subsystem access:
I found this[1] one while looking around, the thread has some additional interesting remarks, including an LD_PRELOAD attack vector.
On Mac, that's easy to do via the Secure Enclave: https://github.com/maxgoedjen/secretive
https://github.com/Foxboron/ssh-tpm-agent
Currently hacking up better support for `HostKeyAgent` and `HostKey` for `sshd`.
I used to use an easily memorizable password, but you said that was wrong, and set me straight. Now my password is so complex, I have to rely upon a 3rd party service, that keeps getting hacked.
Then you insisted I use keys. After, you became irate if I left the keys on my work dir.
Now you want me to lug around a 2U HSM appliance?!
For shame!
And there's a big dog chained to the desk beside it. Biometric security, you see: if you don't smell right to Brutus, you don't get to log on.
Look at our security rituals from an outside view: we sure do seem to spend a lot of time propitiating our idols of one kind or another.
If you don't need a certified HSM that generates keys on device (and you don't, right? You can generate keys on a ramdisk from live media with no persistence and no/encrypted swap), you can use basically any PGP smartcard, including nice little USB ones like Yubikey and NitroKey. And even if you do you can get a little USB HSM.
Edit: Okay now I see that command is supposed to be run from internal-web and not campfire. I guess you would also have to ProxyJump through vuln-server to internal-web to even run that command!
I just wanted to mention that not only does it have subsystems like sftp but you can make up your own subsystem (i use parameko!) to so just about whatever you want. Cool stuff like exposing the remote host's sound or random devices over ssh, cool BBS displays/apps or for the sneaky: command and control protocol between your malware/implant that proxies normal ssh for normal ssh clients as usual.
ssh -L 0.0.0.0:2222:10.0.0.1:22 host
I think this will bind to 0.0.0.0:2222 (allowing remote hosts to connect) and forward all traffic to that port to 10.0.0.1:22 (from the server's perspective).
The biggest gap in this collection of tricks (IMO) is SSH certificate support.
But they can tell the agent to authenticate with any key loaded in your agent, not just the one you used to ssh into the machine you forwarded your agent to
So e.g if you have a distinct ssh key for GitHub and a different one for all other uses and you ssh to a compromised server with agent forwarding, the attacker can then ssh to GitHub as you.
() there was a vulnerability not too long ago involving getting the remote agent to load arbitrary shared objects for remote code execution, which obviously changes things
There are alot if details in this that can go wrong. I seldom use agent forward in unknown/undesigned environments because of this.
People should just use example.com, it’s reserved and the most obvious to the end user that it’s for example purposes.
Whereas ".intranet", ".lan", and some others are what (some) people think ".local" is[1].
But I have to ask: do people really find color schemes like this easier to read? I'm squinting at it throughout.
This is how the sudoers file works as well. I think this is desirable in software that authenticates or authorizes users, and maybe more broadly wherever security concerns are essential. That's because this logic makes it easy to create settings that can't be overridden in by adding a new file in a whatever.conf.d directory: you define those settings in the main config file before you source whatever.conf.d/* and you put some kind of special protections on that file.
Even where you're not worried about somebody evading your controls per se, it can be nice from a configuration management perspective in giving you a soft 'guarantee' that if some new hire who doesn't have the whole picture adds a new file in there, or some package tries to install a stupid default for its own service, your baseline settings can retain priority.
In other contexts you probably see the opposite behavior because what you really want is not a 'baseline configuration' but a collection of defaults in the strict sense: fallback settings to be used in case nothing is explicitly configured by the user, developer, or administrator (as the case may be).
I strongly disagree - the most important trait for security-relevant configuration is that it works like you'd expect.
> That's because this logic makes it easy to create settings that can't be overridden in by adding a new file in a whatever.conf.d directory: you define those settings in the main config file before you source whatever.conf.d/* and you put some kind of special protections on that file.
What is gained by this? The administrator controls both of these locations.
> Even where you're not worried about somebody evading your controls per se, it can be nice from a configuration management perspective in giving you a soft 'guarantee' that if some new hire who doesn't have the whole picture adds a new file in there, or some package tries to install a stupid default for its own service, your baseline settings can retain priority.
Instead the new hire or a package adds important settings that now have no effect. In either case you need to review the whole configuration - reversing the order of precedence doesn't do anyting except confuse users.
If you want to have a configuration file that has the final say then make that explicit - and that absolutely doesn't require the order inside that file to be reversed. If the subdirectory is explicitly included in the main config then you can add your overrides after that include anyway.
But I got bit by something related yesterday when NixOS suddenly changed the merge order and put my AllowUsers from own file below a Match from another file and locked me out :(
Same here. I have anecdotally noted that many people, like myself, who were around in the days of actual green screen terminals and later green monochrome monitors are far less likely to prefer dark mode than younger people.
For me, the advent of color screens that made black-on-white text possible was a huge improvement in terms of readability and eye strain reduction, and I cannot imagine going back to the old ways.
The constant "light mode hurts my eyes!" never made sense to me until I tried using my computer in a completely dark room. So I think the trend over the past decade isn't really dark mode itself, it's more people using screens in dark rooms.
And as someone else mentioned, dark rooms may be part of it too. I find light backgrounds less obnoxious in brighter light, but i avoid brighter lights by preference.
Indeed, for the past decade I also prefer the light themes, though I find these dark themes pretty usable too: https://protesilaos.com/emacs/modus-themes-pictures
As I've noticed, the main problem with dark themes is the low contrast usually. These modus themes were designed scientifically, to meet the contrast ratios recommended by the Web Content Accessibility Guidelines.
Also, back in those days mono-monitor days, people just kept the brightness at the same level they used during the day and complained how tiring is it for their eyes to "use the computer for so many hours".
My eyes never got tired, because I was constantly adjusting the brightness to match the ambient lighting and I've drastically lowered it, when I was coding in dark.
I've noticed that dark-theme users (eg during live streaming) crank up their brightness/contrast to compensate for the low-contrast dark themes, then they are surprised to be blinded when they open a webpage, which is extremely likely to be light themed and they keep complaining about light themes...
On the whole, this actually allows you to achieve the same contrast level with lower total light emission. If you are in fact clinically light sensitive and you also have difficulty with low contrast, you just can't achieve the balance of contrast you need as effectively with light themes.
> then they are surprised to be blinded when they open a webpage, which is extremely likely to be light themed
For me it's well worth it to sacrifice the 'artistic integrity' of web designers' intentions and use something like Dark Reader or Midnight Lizard to force a dark theme across the whole web.
No, and also not the monospace font for prose.
I simply turned off style sheets in Firefox (Alt+V Y M [View menu->Page Style->No Style]) to get rid of the website color scheme and have it render using the colors I have set as default in Firefox.
As it is, I'm much the opposite of our cloud-first DevOps guys: I know how to operate Linux a lot better than I know how to operate AWS. That makes yours a more intimidating proposition to me.
At the same time, this particular box was a shortlived VM that was already behind a VPN, my organization already has tentative plans to implement a different network access system than SSM, it won't be my project to execute, and I don't get to be in the room for architecture decisions related to it.
I would not be surprised if there is some way to attack this into getting a private key, but it would either be a direct attack on the agent code like sending it malformed messages to somehow get remote code execution to then read the key. Or some more complicated attack on the cryptography where you repeatedly force it to auth and can somehow use the results to reduce the key space needed to brute force the key - along the lines of a known plaintext attack.
But I'm also just a hobbiest here who has looked a little into the security model but I am by no means a cryptography expert, so take all of this with a grain of salt.
It's amazing how switching jobs to one where you just write terraform and yaml all day gets your terminal skills all rusty.
Sometimes you do want to set a baseline, and other times you do want to set a default. But using an unconventional ordering for overrides in your config file format isn't the right way to do anything.
Thanks for laying this out.
.example is also reserved for this exact purpose.
That's true but most people don't understand what subdomains are, so we're back to being (more) difficult for laypersons.
> And there’s also example.org and example.net
Well, that's a good point. There are options.
They are probably referring to its special status and special handling.
https://en.wikipedia.org/wiki/.local
>The Internet Engineering Task Force (IETF) reserves the use of the domain name label .local as a special-use domain name for hostnames in local area networks that can be resolved via the Multicast DNS name resolution protocol.[2] Any DNS query for a name ending with the label local must be sent to the mDNS IPv4 link-local multicast address 224.0.0.251, or its IPv6 equivalent ff02::fb. A domain name ending in .local may be resolved concurrently via other mechanisms, for example, unicast DNS
Back in the 1990s, Microsoft distributed a Netware-killer version of NT called Windows Small Business Server. It was aimed at companies that might at best have a dial-up internet service. SBS, being based on Active Directory and Exchange (etc usw) required a domain name, but back then you needed considerable arcane knowledge to register an Internet domain name, which most SBS users lacked.
So Microsoft recommended that their SBS users should pick a name under .local for their AD domain name. I will not relive the many hilarious fuckups this caused, especially when Exchange was trying to use POP3 for incoming email. [tedu’s comment upthread reminds me of an incident when I spotted a company that was clever enough not to use .local for their AD, but not clever enough to understand that corp.int is not an internal subdomain of corporation.com.]
(What MS should have done, instead of squatting on a name that might get created as a real TLD in the future, was tell their customers to make up a subdomain of a properly registered domain of MS’s own; if MS’s customers wanted to turn their fake domain name into a real internet presence, MS would have had a ready-made lever to turn their customers into subscribers. But that was 15ish years before MS realised Azure might be a good idea.)
OK, so part two of this story is the early years of Mac OS X when Apple needed a replacement for AppleTalk that worked with IP over Ethernet. The main gap that needed filling was zero-configuration service discovery, which AppleTalk had enjoyed forever and IP lacked. The solution was called Rendezvous or Bonjour (I forget which name replaced which) and multicast DNS was a foundational part of it. Apple did an incredibly effective job of getting other vendors (especially printer manufacturers) to adopt the new protocol.
HOWEVER, Apple needed to choose a domain name for mDNS so that names of devices on the LAN could be distinguished from names out on the internet. They chose .local because a LAN is a local area network.
Hence, hilarity ensued. So much confusion and failure to interoperate because two large corporations failed to appreciate the importance of a global shared namespace, and foolishly chose the same cute name to mean completely different things.
Possibly the saddest episode was when Apple were in the process of turning mDNS from a de facto standard (with multiple implementations across multiple vendors) into an IETF standard. MS tried to derail the effort by persuading the IETF to spin up a working group to develop LLMNR, link-local multicast name resolution, TOTALLY NOT mDNS HONESTLY. Surprising no one, there was zero interest in replacing a successful working deployed protocol with slightly differently shaped vapourware. Rough consensus and running code wins again.
The upshot of this is that the IETF has a lot of institutional trauma and scar tissue around the question of non-DNS domain names. (see also .onion and others)
As far as laypersons are concerned, I would worry that they wouldn’t recognize foo.example to be a domain name, as opposed to www.example.com or blog.example.com.
But in your example, you're comparing different structures. "www.foo.example" is somewhat more clear.