Freenginx: Core Nginx developer announces fork(mailman.nginx.org) |
Freenginx: Core Nginx developer announces fork(mailman.nginx.org) |
https://web.archive.org/web/20240214184151/https://mailman.n...
see EU CRA
There was a time when I wanted to move away from it and was eyeing HAProxy, but the lack of the ability to serve static files didn't convince me. Then there was Traefik, but I never looked too much into it, because Nginx is working just fine for me.
My biggest hope was Cloudflare's Rust-based Pingora pre-announcement, which was then never published as Open Source.
Now that I googled for the Pingora name I found Oxy, which might be Pingora? Googling for this yields
> Although Pingora, another proxy server developed by us in Rust, shares some similarities with Oxy, it was intentionally designed as a separate proxy server with a different objective.
Any non-Apache recommendations? It should be able to serve static files.
I could have sworn that I've read about Nginx CVEs in the past.
Before you ask why would I do that, Ive got all Ethernet interfaces on dynamically IP created on a on-demand basis and only wanted ONE specific interface (non-public) to host the HTTP/HTTPS protocol.
And no, we do not want to jerry-rig some fancy nginx config file shell -script updater whenever an IP address gets assigned/reassigned.
Here came lighthttpd and Apache to the rescue.
Infrastructure like that should not be run by for-profit corporations anyway, it will always end up like in this case sooner or later
Probably a skill issue but when I last tried to compile Nginx from the Github mirror I spent hours trying to figure it out. I wish there was a GitHub page with an easy to understand build process... and that I could just run "cargo build --release" lol
make
make install
I just ran this to be sure I wasn't delusional and it took only 2 minutes.
https://github.com/nginx/nginx/tree/branches/stable-1.24
I cloned this and it doesn't have a makefile or configure script
Neither does the official repo?
https://hg.nginx.org/nginx/file/tip
Do you run it from /auto/?
OTOH, things that update too often seem to be more than slightly broken on an ongoing basis, due to ill-advised design changes, new bugs and regressions, etc.
Eg: http3 support was stabilized with 1.25.1 , which came out June 2023.
I now believe that every piece of software should be shipped as a container to avoid any system library dependencies.
Too bad every team I've ever worked on as a consultant does the opposite. The biggest piles of shit I've ever seen created have all been the product of 10 people doing 2 people's worth of work...
NGINX is the de facto standard today, but I can remember running servers off apache when I began professionally programming. I remember writing basic cross-broweser spas with script.aculous, and prototypejs in 2005, before bundlers and react and node.
Everything gets gradually replaced, eventually.
This is one reason maintainability is very important for the survival of a project. If it takes an extra person to maintain your build system or manage dependencies or... or... it makes it all the more fragile.
1 or 2 people maintain one particular software implementation of some of these standards.
It's interesting to think of what a large and massive community cheap and reliable computation and networking has created.
If 2 people are operating the plants, that’s terrifying.
(Obviously once bigco buys such a startup's offering, that startup needs to hire, fast)
Traefik Opensource is my go to for almost all of my use cases theses days and I have never stopped and said hmmm I wonder if Apache would do better here. It is that good.
Nginx loves to pretend it’s 1995. It barely has http3 support and does insanely stupid things by default.
No wonder people move to haproxy, Traefik, caddy, etc. Cloudflare doesn’t use it anymore for good reason.
Not sure if serious, but you do realise that free is not at all about having a GitHub page?
Maxim has been working on nginx for years and just forked the project so that he can continue working on it. The license remains the same as the original nginx project and you can already download its sources here: https://freenginx.org/en/download.html
In that case I'd agree with his view, though I think his reaction is a bit over the top.
[0] https://freenginx.org/pipermail/nginx/2024-February/000007.h...
For NGinx I have been able to make use of HAProxy and Apache just fine. Long ago Apache was slower than NGinx but ever since APR 1.7 and Apache 2.4 there are about the same performance wise. Some here don't like the configuration syntax but I am used to it.
There is JetBrains, for example.
But there is also core-js which is a little polyfill library being used by like way more than half of high profile websites. Also written by a Russian national.
If you excise all contributions by Russian nationals to PostgreSQL or the Linux kernel, they will be left in a not very runnable state, I’m afraid.
On the other hand, it’s not like you are giving them money directly, unless you do; I also can see that in, say, both Linux and PostgreSQL there is also enough people from the “geopolitical opposition” so that even if the Russian contributors are asked by some stern people from the Apparat to sneak something backdoory in, it will be sniffed rather quickly and prevented from going much further.
So tl;dr is that there is no simple response.
"I don't always git clone, but when I do, it's hg clone"
Some people just don't have a clue and only know buzzwords
That team changes every 6 month when another company offers more money. If only one or two people are working on a project, that's a high risk for the company.
If you got one or two highly skilled people in that team of 10, you are lucky. Managers don't want them to work alone on their project, they want them to help the team grow.
But that is peanuts and for me basically no difference than B2C and that is not something you can put on "customers that trusted us" banner on your landing page.
If you want big company to rely on your services and have 50-100 users each seat paid $500 a month form a single company, that is not just some manager swiping CC and for that you have to have a team and business continuity.
For example, how much of that code is the mail server component and how much is the http component? How much does http/1, or http/2 or http/3 take up? How much of that is necessary to keep the internet actually running?
I'm not suggesting it's trivial but the original perspective was highly overblown. To think of it another way, if these two men died tomorrow, how much of an impact would it actually have? Some, to be sure, but the internet wouldn't even notice.
Then nothing
(That isn't about Caddy, rather a third-party plugin.)
I'd love to get rid of the part of my clients' codebase that starts with // workaround for broken caddy servers
It’s sometimes beneficial to pose as an EU-based business if a purely Russian business was either sanctioned or considered too risky/dirty/shady to deal with.
So while Czechs don’t like to be equated with Russians, not all of them would quite sing „běž domů, Ivane” or share the feeling.
Unless you mean, having thin clients use SSH as opposed to directly running serial cables throughout a building to VT100 style hardware terminals, and therefore being vulnerable to eavesdropping and hijacking?
But I think when we talk about TTY we mostly don’t refer to that kind of situation.
If someone talks about TTY today, I assume they mean the protocol and kernel interfaces being used. Not any kind of physical VT100 style serial communication terminals.
There is also the mighty bystander effect at play: surely, someone else is going to look at it. Someone else will have time to test it. He's our hero, the Someone-Else-Man!
Mind you, it only takes to catch you once, and your mountain of reputation will poof out of existence in an eyeblink. This is the price.
Mind you, asking to downplay a vulnerability "because it's in an experimental module not built by default" would make me suspicious on the simple grounds that even if a module is experimental, you ship it alongside your stable code, and for sure someone builds it and is using it. Depending on who those users might be, there could be also parties interested in them not patching the vulnerability for as long as possible.
This sounds paranoid for sure, but your being paranoid doesn't mean there's nobody out to get you!
But at least I have the option, dammit! Contrary to the proprietary software where your problem will be diligently filed into a ticket, given a number, and be left to rot.
Which doesn't change the fact that people are lazy and do turn the blind eye... :-( and sometimes the Someone-Else-Man won't come and save the day. But that's just life.
I'm not sure about that, for anything besides static resources, given the rate at which various vulnerabilities are found at and how large automated attacks can be, unless you want an up to date WAF in front of everything to be a pre-requisite.
Well, either that or using mTLS or other methods of only letting trusted parties access your resources (which I do for a lot of my homelab), but that's not the most scalable approach.
Back end code does tend to rot a lot, for example, like log4shell showed. Everything was okay one moment and then BOOM, RCEs all over the place the next. I'm all for proven solutions, but I can't exactly escape needing to do everything from OS updates, to language runtime and library updates.
That and a small collection of other things are standards based and not going though changes.
I’ve been looking for the words to put to that feeling myself but was unable to pinpoint it so well.
I loved GitHub at first. “Look at all the cool stuff I made” was kinda a way of showing my capabilities (and is still a great way today!) but somewhere along the way it became a platform for egos and star stroking and blind following into the nights. They improved their search but it could be so much better. Not everyone has a graphic designer on staff to make pretty README.md’s
Apple routinely holds back changes for a .0 release for advertising reasons. This means that they routinely have big releases that break everything at once. Bugs could come from 4 or 5 different sets of changes. But if they spread out changes… bug sources would be way more easy to identify.
And bug fix velocity going up could mean people stop treading water on bugs, and actually get to making changes to avoid entire classes of bugs!
Instead, people think the way to avoid bugs is to avoid updates, or do it all at once. This leads to iOS .0 releases being garbage, users of non-rolling release Linux distros to have bugs in their software that were fixed upstream years ago, and ultimately to make it harder to actually fix bugs.
I do regularly install updates on my (Linux) desktop/laptop because guess what? It consistently works exactly the same afterward. Occasionally new formats like jxl images just start working everywhere or something. But otherwise it has just continued to work unchanging with no fanfare for the last decade or so. It's amazing to me how much higher quality in that way volunteer software is compared to commercial software.
If you want things not to break, you must slow down.
It isn’t reasonable to ask for these two things at once:
* lots of change
* stability
The current milieu seems dramatically skewed toward churning out low-value changes without sufficiently considering the impact to stability, causing frequent breakage, and resulting in net negative value.
> I wrote the initial version of SSH (Secure Shell) in Spring 1995. It was a time when telnet and FTP were widely used.
> Anyway, I designed SSH to replace both telnet (port 23) and ftp (port 21). Port 22 was free. It was conveniently between the ports for telnet and ftp. I figured having that port number might be one of those small things that would give some aura of credibility. But how could I get that port number? I had never allocated one, but I knew somebody who had allocated a port.
Emphasis mine.
Cheers.
rsh was common on internal networks, but almost never used on the wider Internet. telnet was everywhere all across the net.
ssh was a revelation and it replaced telnet and authenticated/non-anonymous ftp primarily.
And also sometimes rsh, but less importantly.
The command line options were almost identical for an easy switch. ssh even respected the .rhosts file! Last time I checked, that functionality was still in place.
Both the rlogin-family of commands and the telnet/ftp-family were in use across the Internet, certainly in cases where Kerberos was used. I would think telnet was more common, certainly so outside the UNIX sphere of influence, but things like Kermit also existed.
They all got SSL-encapsulated versions in time, but Kerberos solved authentication for free, and for the simpler use cases ssh had already taken over by then. And in the longer run, simple almost always wins!
ssh solved the "pass credentials in cleartext over untrusted networks" problem. Consequently it replaced telnet and ftp. It also duplicated the functionality of rsh and rcp, so those protocols became irrelevant. But that was not the important goal.
> Kerberos solved authentication for free,
This made me laugh. Kerberos didn't do anything for free. :)
Even in Athena, Kerberos had reliability problems. In the wider world, it was very hard to find a well-managed Kerberos implementation. Things are different now!
I have not. It is quite the niche problem. Mostly because web performance is so bad across the board that saving a few milliseconds just isn't meaningful when your page load takes more than a second and mostly is stuck in javascript anyway. Plus everybody just uses cloudflare and having that CDN layer use whatever modern tech is best is very much good enough.
I'm not 100% sure when I started using putty, but I definitely used it in 2004. I still need a ssh client and terminal emulator for Windows. I still don't want to install a unix like environment just to have a terminal. I still don't want tabs in my terminal, lots of windows works just fine. I still need X11 forwarding so I can run programs on remote systems and display them on Windows (VcXsrv is an easier to get going X server than others I've used on Windows).
I might like to have something that can do whatever magic so I can gcloud and aws auth on my remote machine without cutting and pasting giant urls and auth blobs to and fro all the time; but I'm using a auth token that needs to stay connected to the windows machine. In a more integrated corp environment this would probably be keberos/active directory/magic?
(But yeah I'm still using putty, too)
So I'm not using Putty since I guess ~ 2018 or so. Not insisting other should stop using it, of course.
I'm not really a fan of cmd or powershell, although I guess I could use them in a pinch. Wouldn't look like what I'm used to though. :p
Are there any better alternatives?
Then there's things like x11-style copy-paste.
I still use putty because it does what I need for it to do. No need to change just because MS has their own terminals application, which besides I far from trust.
But there's also trust in the rely on sense. Which at least I try to compartmentalize. I can trust Microsoft (or Google) to make an OS I can rely on to run other people's apps. If Microsoft or Google want to provide apps, they'll be evaluated as they are, not with a bias because the OS provider shipped them.
What was meant is Windows Terminal
So this is a pretty impactful fork. It's not like one of 8 core devs or something. This is 50% of the team.
Edit: Just noticed Sergey Kandaurov isn't listed on GitHub "contributors" because he doesn't have a GitHub account (my bad). So it's more like 33% of the team. Previous releases have been tagged by Maxim, but the latest (today's 1.25.4) was tagged by Sergey.
That said, I’m not sure how much leg he has to stand on for using the word nginx itself in the new product’s name and domain…
pretty sure they can't really do anything to him in Russia. Russia and US don't recognize each others patents, same as China.
https://freenginx.org/hg/nginx
I don't see it. Sure, he contributes. But in the last 3-4 years he definitely does not look like he is nginx based on that log. Or am I looking in the wrong place?
(Regardless, if you scroll back past March 2020, the timeline "resets" to this past year, and you see a ton of Dounin commits. Looks like an artifact of how the hg web viewer deals with large, long-lived branches getting merged.)
... to this specific bug in an experimental feature.
Originally I read your comment as Maxim doesn't want to use CVEs at all.
Where was the disagreement hashed out, so I can read more?
If nginx continues to receive more attention from security researchers, I imagine Maxim will have good reasons to backport fixes the other way too, or at least benefit from the same disclosures even if he does prefer to write his own patches as things do diverge.
Though history also shows that hostile forks rarely survive 6 months. They either get merged if they had enough marginal value, or abandoned outright if they didn't. Time will tell.
- nginx is "open core", with some useful features in the proprietary version.
- angie (a fork by several core devs) has a CLA, which sounds like a bait and switch waiting to happen and distro's won't package it
- freenginx is at least open source. But who knows if it'll still be around by June.
I had been an Apache user for quite some time, and thought I'd take a look at the (at that point, a few years old) "new" shiny thing. I found that something as simple as LDAP authentication required a payed plugin; a free Apache module has been available for this for ages. That made nginx a non-starter for this particular use case.
I wonder if the fork will accumulate free plugins for things that the old core required payed plugins for, slowly eroding their business case.
>In particular, they decided to interfere with security policy nginx uses for years, ignoring both the policy and developers’ position.
F5 is a CNA and follows CVE program rules and guidelines, and we will err on the side of security and caution. We felt there was a risk to customers/users and it warranted a CVE, he did not.
On the other hand nginx core developers (the Russians) were arrogant to the point of considering anyone else as inferior and unworthy of their attention or respect, unless they contributed to nginx oss. They managed that project secretively and rewrote most “outside” contributions. They also ignored security issues - one internal developer spotted security issues with NGINX Unit (a failed oss project 20 years out of date before it started) and was told to fix the issues quietly and not to mention “security” anywhere in the issue messages or commit history.
So I can imagine exactly how these meetings would have gone, I’m sure it was the last straw!
For clarity are you referring to CVE-2024-24989 and -24990 (HTTP/3)?
> webserver, llc
From statement: "Instead, I’m starting an alternative project, which is going to be run by developers, and not corporate entities"
Ah, I completely forgot F5 was involved in this, probably most of everyone else and F5 gets no money from this. Shouldn't matter to them, do they even have competition in enterprise load balancer space? I spent 9 years of my career managing these devices, they're rock solid and I remember some anecdotes about MS buying them by the truckloads. They should be able to cover someone working on nginx, maybe advertise it more for some OSS goodwill.
"Simplifying configuration: the location directive can define several matching expressions at once, which enables combining blocks with shared settings."
> Unfortunately, some new non-technical management at F5 recently decided that they know better how to run open source projects. In particular, they decided to interfere with security policy nginx uses for years, ignoring both the policy and developers’ position.
Refers to F5's decision to publish two vulnerabilities as CVEs, when Maxim did not want them to be published.
IANAL, but i strongly recommend reconsidering the name as the current one contains a trademark.
Ingress was forked; the Post fork version of Ingress was called "Post"gres.
So maybe name this new project "PostX" (for Post + nginx).
Though that might sound too similar to posix.
> The INGRES relational database management system (DBMS) was implemented during 1975-1977 at the Univerisity of California. Since 1978 various prototype extensions have been made to support distributed databases [STON83a], ordered relations [STON83b], abstract data types [STON83c], and QUEL as a data type [STON84a]. In addition, we proposed but never prototyped a new application program interface [STON84b]. The University of California version of INGRES has been ‘‘hacked up enough’’ to make the inclusion of substantial new function extremely difficult. Another problem with continuing to extend the existing system is that many of our proposed ideas would be difficult to integrate into that system because of earlier design decisions. Consequently, we are building a new database system, called POSTGRES (POSTinGRES).
All F5 contributions to NGINX open source projects have been moved to other global locations. No code, either commercial or open source, is located in Russia.
yeah, yeah
But then perhaps he also has every right to do it, even though AFAIR the original author was somebody else.
Everyone has a right to forking the project. Only time will tell if they get critical mass of developers to keep it going.
Edit: I see now from the hg history that Igor hasn't been coding on Nginx for a decade actually.
In light of recently announced nginx memory-safety vulnerabilities I'd suggest migrating to Caddy https://caddyserver.com/
Using Caddy instead.
A point came where I realised I didn't enjoy Nginx. Configuring it was hard and it felt brittle.
A particular pain point is certificates/ssl. I absolutely dreaded doing anything with certificates in Nginx.
When I heard that Caddy automatically handles SSL/ certificates I jumped the nginx ship and swam as fast as I could to Caddy.
I didn't compile in fastcgi support in to my build, but it can be enabled.
https://mail.python.org/archives/list/mailman-users@python.o...
when your KPI is CVE's per month every bug looks like a CVE
F5 wants this feature prioritized over what Maxim planned, and Maxim doesn't have to comply, he is a volunteer.
Is the fork going to allow you to change the nginx Server response header (A PAID feature in the current fork...) without requiring you to mod it in and recompile it? :p
Yes - You read that correctly. They refuse to accept PR's to add additional functionality because that functionality is restricted to the paid version :p
Buuuut, they have by far the best support. They’re as responsive as Cisco, but every product isn’t a completely different thing, team, etc. And they work really well in a big company used to having Network Engineering as a silo. I’d only use them as physical hardware, though. As a virtual appliance, they’re too resource hungry.
Nginx or HA-Proxy are technically great for anything reasonable and when fronting a small set of applications. I prefer nginx because the config is easier to read for someone coming in behind me. But they take a modern IT structure to support because “Developers” don’t get them and “Network Engineers” don’t have a CLI.
For VMWare, NSX-V HA-Proxy and NSX-T nginx config are like someone read the HOWTO and never got into production ready deployments. They’re poorly tuned and failure recovery is sloooow. AVI looked so promising, but development slowed down and seemed to lose direction post acquisition. And that was before Broadcom. Sigh.
We'd get completely bogus explanations for bugs, escalate up the chain to VPs and leadership because there was an obvious training, understanding, and support for complex issues problem, and get the VPs trying to gaslight us into believing their explanations were valid. We're talking things like on our IPv4 only network, the reason we're having issues is due to bugs in the equipment receiving IPv6 packets.
So it's one of those things where I've personally been burned so hard by F5 that I'd probably to an unreasonable level look for other vendors. The only thing is, this was awhile ago, and the rumor's I've heard are that no one involved is still employed by F5.
Handling a few thousand RPS is nothing to nginx, and doesn't require fancy hardware.
That said, it replaced Kemp load balancers, which it seems is the next biggest competitor in the hardware load balancer appliance space.
Caddy seems like a wonderful alternative that does load balancing and static file serving but has wild config file formats for people coming from Apache/Nginx-land.
Fork vs "hacked up [Ingress] enough ... Consequently, building a new database system" named Postgres.
As a sidenote I believe the people who start projects that they themselves run in excellent manner, should be praised, supported, noted and there is nothing more for their identities to matter. It very much matters some particular person with weird nick burntsushi created this wonderful tool rg, and kept growing it for long time. Besides, I can bet for projects such as Cosmopolitan C, it absolutely matters that jart started/did it.
I think this because Nginx has a bunch of parsing quirks that are shared with AVI and nothing else.
Yeah, very, very likely one and the same. Since 1989.
Can you say more about the CVE thing? That seems like the opposite of what Maxim Dounin was saying.
I don't know what else there is to say really. The QUIC/HTTP/3 vuln was found in NGINX OSS, which is also the basis for the commercial NGINX+ product. We looked at the issue and decided that, by our disclosure policies, we needed to assign a CVE and make a disclosure. And I was firmly in that camp - my personal motto is "Our customers cannot make informed decisions about their networks if we do not inform them." I fight for the users.
Anyway, Maxim did not seem to agree with that position. There wasn't much debate about it - the policy was pretty clear and we said we're issuing a CVE. And this is the result as near I can tell.
Honestly, anyone could have gone to a CNA and demanded a CVE and he would not have been able to stop it. That's how it works.
for tini you mean https://github.com/krallin/tini? how large is your final docker image, why not just alpine in that case which is musl+busybox
In this case, I didn't need alpine. I generally aim to get the image as minimal as possible without too much hassle. I end up doing stuff like this alot when I feel like a community image maybe too bloated when something like alpine or distroless can be used. Entry point scripts have all kinds of envars and a shell dependency, I'd rather rebuild the image to cater for my needs and execute the binary directly, and mount in any config via k8s.
Unpaid Open Source developers tend to focus on interesting/cool core stuff and ignore all the stuff businesses care about (like LDAP authentication).
IDK could be a way to do it, pay the bills and some, and also limit the negative impacts public business or VC funded growth startup.
If the basketball or soccer team captain were also a ball hog, they'd have trouble keeping the bench full.
When you become lead, you have to let some of the code go, and the best way I know to do it is to only put your fingers into the things that require your contextual knowledge not to fuck up. If you own more than 10% of the code at this point, you need to start gift-wrapping parts of the code to give away to other people. If you own more than 20%, then you're the one fucking up.
Obviously this breaks down on a team size of 2, but then so do concerns about group and team dynamics.
Nginx is one of the most widely used open source projects in the world. It's hard to read this without laughing, as if it's still to be determined whether Nginx could be considered successful.
Or, just follow me here, a open source project really doesn't get that many dedicated contributors beyond the leads and everyone else may casually drive-by.
And in all likelihood if you are expecting a core competency in enough domains for the situation you reference to be true, it's because you have a bad case of NIH, you aren't concentrating your efforts in the areas your company is purportedly focused on. That makes it difficult not only to scale up a team, but also to scale it down. The first major revenue hiccup you encounter may be your last.
If you are concentrating on a narrow domain you intend to be experts in, then that will be 15-25% of the code. Meaning to maintain a decent bus number, you only need to be primary on about 10%, if you have half a dozen people or so.
Crafting the change? Applying the commit?
The former is where we should strive for heterogeneousity. The latter is a janitorial duty that should be guarded and centralized.
Do not underestimate the importance of janitorial duties though! That is the way we build culture and community, and that is the only scalable way to build any quality above that it compiles.
Reluctance to accepting commits and keeping a strong culture is something that is common to all successfully scalable open source projects.
And yes, there are lots od companies with bad management, bad decisions, or even surviving despitr bad management. But most of the time thr management at least gives some direction.
You sound like one of those freeloader types, who dont contribute.
I guess this consultancy-on-a-paid-version model doesn't bother me (and clearly didn't bother the developer of freenginx while they were paying him).
But a double fork can't be good.
Clearly it did, so much so that he gave up all that pay.
How long until F5 submits requests for domain ownership of freenginx.org, and how quickly does Angie get takedown requests for their features that look remarkably similar to Nginx Plus features (e.g., the console)?
Its illegal for products in the same space to have similar features?
I'm not very familiar with the implications, so it seems like a relatively fine hair to split- as though the trouble of dealing with these as CSV would be less than the extra work of forking.
Or he's just a very strange man, and for some reason this pair of CVEs was oddly that important to him.
I can’t imagine them supporting telco gear. The IPv6 thing has me LOLing because I just had a similar experience with a vendor where we don’t route IPv6 in that segment and even if we did, it shouldn’t break. Similarly, a vendor in a space they don’t belong that I imagine we bought because of a golf game.
A thing I dread is a product we’ve adopted being acquired… and worse, being acquired by someone extending their brand into a new area. It’s also why we often choose a big brand over a superior product. It’s not the issue of today, but when they get bought and by who. I hate that so much and not my decision, but it’s a reality.
It’s also a terrible sign if you’re dealing with a real bug and you’re stuck with a sales engineer and can’t get a product engineer directly involved.
I have a list of “thou shalt not” companies as well, and some may be similar where a few bad experiences ruined the brand for me. Some we’re still stuck with and I maaaay be looking for ways to kill that.
Can you share that list?
Generally my rule is “except for their very core product.” But this is full “hate everything” that pops into my mind:
RedHat won’t accept gifted patches for critical bugs in their tools that they won’t troubleshoot themselves. Getting the patch upstream means you get to use it in the next major version years later. That predates IBM. I won’t use their distribution specific tooling anymore. Outside the OS sucks worse. If I hear ActiveMQ one more time… [caveat: I probably hate every commercial Linux distro and Windows because my nonexistent beard is grayer than my age]
IBM… kind of feel sad about it, but they now suck at everything.
Oracle has good support, but they’re predatory and require an army of humans to manage inherently hodgepodge systems. Also creates an organizational unit of certified admins that can’t transition to alternatives because they’ve only memorized the product. Cisco’s the same except the predatory part and without many good alternatives for core DC gear.
CA, Symantec were awful pre-Broadcom and even worse now that they’re Broadcom’s annuity. Where products go to die.
Trellix (ex McAffee) is like the new Symantec or something.
There’s more I wish I could list for you, but can’t for various reasons.
On the other end, Satya has made MS a reasonable choice in so many things. Still a lot that sucks or is immature, but still… I didn’t think that was possible. I had to shift my mindset.
- CVEs are gold to researchers and organizations like citations are to academics. In this case, the CVEs were filed based on "policy" but it's unclear if they are just adding noise to the DB.
- The severity of the bug is not as severe as greater powers-that-be would like to think (again, they see it as doing due diligence; developers who know the ins and outs might see it as an overreaction).
- Bug is in an experimental feature.
I'm not saying one way is right or not in this case, just pointing out my experience has generally been that CVEs are kind of broken in general...
If you run a web app of any sort, and you don't have "X-Frame-Options: Deny" in your headers, you'll get lots of "researchers" (that are probably bots) e-mailing you that you have a CRITICAL security issue.
"Beg bounties", we call them.
That very much depends on what service is being denied. Nginx is _everywhere_. While not a direct security concern for nginx (instead an availablity issue) it could have security or safety implications for wider systems. What if knocking out nginx breaks a service for logging & monitoring security information? Or an ambulance call out management system? Or a payment progressing system for your business at the busiest time if your trading year? There are many other such examples. This sort of thing is why availablity can be considered a security matter and therefore why DoS vulnerabilities, particularly those affecting common software, are handled as security issues of significant severity.
Viewed in this light a bug that enables a successful Denial of Service (DoS) or Distributed Denial of Service (DDoS) attack is a security bug. A bug that causes a DoS or DDoS, but is not exploitable, would not be a security bug (e.g., some idiot added an infinite loop to the startup code). That's where issue triage comes in, a bug should never be assigned before its triaged. Sometimes triage results in 'we don't know enough' and someone gets assigned to evaluate the bug to answer specific questions before triage can finished. After triage is get assigned - or even better, a developer with a matching skill set chooses it to work on for the next release/sprint/etc.
But I agree DoS is kind of a strawman since everything connected to a network is vulnerable to some form of DoS without extensive mitigation.
What about serving certificate revocation list, with another system relying on say one day old cache? (Sure, that's "fail open" - but still...).
Or proxying LDAP for sync to a central auth/authz system?
Ed: proxy giving access to logging system goes down - alert on failed logins silenced, disabling rate limits for brute force attacks?
>And, while the particular action isn't exactly very bad, the approach in general is quite problematic.
When in doubt, err on the side of doing the right thing for the users. I find that's the best approach. I don't consider CVE a bad thing - it shouldn't be treated like a scarlet letter to be avoided. It is a unique identifier that makes it easy to talk about a specific issue and get the word out to customers/users so they can protect themselves. And that's a good thing.
The question I ask is "Why not assign a CVE?" You have to have a solid reason why not to do it, because of default is to assign and disclose.
I don't think having the CVEs should reflect poorly on NGINX or Maxim. I'm sorry he feels the way he does, but I hold no ill will toward him and wish him success, seriously.
I get that CVEs have been politicized and weaponized by a bunch of people, but it seems weird to object that strenuously to something like this.
And appreciate the clarification about the CVE disagreement.
And the customers were, by and large, great.
> Honestly, anyone could have gone to a CNA and demanded a CVE and he would not have been able to stop it. That's how it works.
I recently did exactly that when a vendor refused to obtain a CVE themselves. In my case, I was doing it as part of an effort to educate the vendor on how CVEs worked.
I know there are other mentions - it's been in the commercial product since R30, hence the CVE.
Even if third parties can file CVEs, do you think it hits different when the parent organization decides to do so against the developer's wishes? Why do he and F5 view the bugs differently? It sounds like the fork decision was motivated less by the actual CVEs and more about how the decision was negotiated (or not at all).
(PS. Thanks for participating in the discussion.)
Based on my observation of various NGINX forums and mailing lists, the HTTP/3 feature, while experimental, is seeing adoption by the leading edge of web applications, so I don't think it could be argued that its not being slowly rolled into production in places.
all that said, neither drops existing connections on reload
Now, SSL termination is done at the host level, using a distributed SSL termination proxy developed by S3 called "JBLRelay"
With fairly cheap ddos services you can "just" order you can knock most servers offline anyway. Internet reachability is rarely safety-critical, and if it is, that's probably a huge design flaw somewhere because there's tons of reasons outside of your control that can make the internet not work for either the server or clients.
Is all of this inconvenient and (potentially) a serious problem? Sure. But not "zomg criminals have credit card records / can spoof random domains / read private data / etc. etc." type serious.
A DoS bug and an DDoS attack are very different things. One is a flaw that can bring a service down, the other is a brute force technique for making a service unusable. You can DDoS services without exploiting bugs.
I see this question has remained unanswered for a couple of years.
https://security.stackexchange.com/questions/257823/what-are...
> nginx is already good at mitigating HTTP desync / request smuggling attacks, even without using HTTP/2 to backends. In particular because it normalizes Content-Length and Transfer-Encoding while routing requests (and also does not reuse connections to backend servers unless explicitly configured to do so)
It’s a reason why many large, modern infra deployments have moved away from nginx.
I can see where you're coming from, but it's not unreasonable behaviour, is it? Connections needs to migrated over to the new worker and that's how all major servers do it. If that's a problem then maybe something designed as proxy only instead of a real server is the way to go?
This works for me because I already knew a fair bit about nginx configuration before picking up Caddy but it really kills me to see just how many projects don't even bother to explain the nginx config they provide.
An example of this is Mattermost, which requires WebSockets and a few other config tweaks when running behind a reverse proxy. How does Mattermost document this? With an example nginx config! Want to use a different reverse proxy? Well, I hope you know how to read nginx configuration because there's no English description of what the example configuration does.
Mastodon is another project that has committed this sin. I'm sure the list is never-ending.
This is so real. I call it "doc-lock" or documentation lock-in. I don't really know a good scalable way to solve this faster than the natural passage of time and growth of the Caddy project.
I think you are totally right here - gaining critical mass over the time for battle tested solution. On the other hand, the authors [who prefers Caddy] of docs will likely abandon providing Nginx configs sample and someone else will complain on that on HN.
"Battle tested" can be seen differently of course, but in my opinion, things like the next one,
> IMO most users do require the newer versions because we made critical changes to how key things work and perform. I cannot in good faith recommend running anything but the latest release.
from https://news.ycombinator.com/item?id=36055554 , by someone working at Caddy doesn't help. May be in their bubble (can I say your bubble as you are from Caddy as well?) noone really cares on LTS stuff and just use "image: caddy:latest" and everything is in containers managed by dev teams - just my projection on why it may be so.
This method is also useful for abusive clients that one still wishes to give an error page to. Based on traffic patterns, drop them in a stick table and route those people to your pre-compressed error page in the unique back-end. It keeps them at the edge of the network.
[1] https://docs.haproxy.org/dev/configuration.html#4.4-return
Disclosure: I'm a community contributor to HAProxy.
I'm a community contributor to HAProxy.
I think I recall chatting with you on here or email, I can't remember which. I have mostly interacted with Willy in the past. He is also on here. Every interaction with HAProxy developers have been educational and thought provoking not to mention pleasant.
stockholm syndrome
[0] https://www.nginx.com/resources/wiki/start/topics/depth/ifis...
I can see why you'd want an all-in-one solution sometimes, but I also think a single-purpose service has strengths all its own.
nginx open source does all of these things and more wonderfully:
Reverse proxying web apps written in your language of choice
Load balancer
Rate limiting
TLS termination (serving SSL certificates)
Redirecting HTTP to HTTPS and other app-level redirects
Serving static files with cache headers
Managing a deny / allow list for IP addresses
Getting geolocation data[0], such as a visitor’s country code, and setting it in a header
Serving a maintenance page if my app back-end happens to be down on purpose
Handling gzip compression
Handling websocket connections
nginx can do all of this with a few dozen lines of config and it has an impeccable track record of being efficient and stable. You can also use something like OpenResty to have Lua script support so you can script custom solutions. If you didn't want to use nginx plus you can find semi-comparable open source Lua scripts and nginx modules for some individual plus features.
[0]: Technically this is an open source module to provide this feature.
And I think it's fine.
On the other hand, since the definition of "supported" is specifically designed to help downstreams, if it were known that some bit of code was widely used in production, we'd be open to declaring it "security supported", regardless of whether we thought it was "finished" or not.
The stack included Linux, Java, Chromium, and MySQL. It took multiple person-years of playing whack-a-mole with dependencies to get it into production because we'd have to have conversations like:
Client: there's a CVE in the this module
Us: that's not exploitable because it's behind a configuration option that we haven't enabled
Client: somebody could turn it on
Us: even if they somehow did and nobody noticed, they would have to stand up a server inside your VPC and connect to that
Client: well what if they did that?
Us: then they'd already have root and you are hosed
Client: but the CVE
Us:
In either case, you should probably do something about it.
Really, really dumb. Not at all good security, just checking boxes.
There's tons of reasons why you wouldn't, but the core reason for this fork probably isn't really about the CVEs as such. It's either the final straw in a long line of disagreements, or the entire thing was handled was so badly that he no longer wants to work with these people. Or most likely: a combination of both.
I once quit after a small disagreement because the owner cut off my explanation on why I built something the way I did with "I don't care, just do what I say". This was after he ignored the discussion on how to design it, and ignored requests for feedback when I was building it. And look, I don't mind to re-doing it even if I don't agree it's better better, but I did put quite a lot of thought and effort in to it and thought it worked very well. If you don't even want to spend 3 minutes listening to the reasons on why it's like that then kindly go fuck yourself.
It's not the disagreement as such that matters, it's the lack of basic respect.
This makes sense IMHO: experimental features may be buggy, but they may work in your limited use case. So you may be inclined to use them...except you don't know they expose you in a critical way.
Also, something that keeps getting lost here, the CVE is NOT just against NGINX OSS, but also NGINX+, the commercial product. And the packaging, release, and messaging on that is a bit different. That had to be part of the decision process too. Since it is the same code the CVE applies to both. This was not a rash decision or one made without a lot of discussion and consideration of multiple factors.
But one of our guiding principles that we literally ask ourselves during these things is "What is the right thing to do?" Meaning, what is the right thing for the users, first and foremost. That's part of the job, IMHO. Some vendors never disclose anything, but that's not how we operate. I've written a few articles on F5's DevCentral site about this - "Why We CVE" and "CVE: Who, What, Where, and When" are particularly on topic for this, I think.
The question I ask is "Why not assign a CVE?"
Other hats I wear (outside of my day job) include being on every (literally, every) CVE.org Working Group and being the newly elected CNA Liaison to the CVE Board. This has been a subject of discussion and things are a bit overblown right now, IMHO. Some of the initial communications were perhaps not as clear as they could have been. But it isn't going to be every kernel bug being a CVE - not every bug is a vuln.
I'm also one of the co-chairs for the upcoming VulnCon in Raleigh, NC. Just a plug. ;-)
While I agree the whole Linux CVE thing is a bit overblown, but as an outside observer the new policy [1] does not read like they are super happy with CVE in general.
Too bad the CFP is closed for VulnCon, it might be fun to do a "Assume everything is wrong and you can't do anything the way you do it now - how do you build CVE 2.0" (also that title is too long).
1. https://lwn.net/ml/linux-kernel/2024021314-unwelcome-shrill-...
Found a missing comma in the documentation of a function? Yup - That's a CVE ;p
Could possibly also have been in the issue tracker, which I did help bootstrapping and doing maintenance for quite a while after initially setting it up. Luckily the core team has took over, since I had much less time for HAProxy contributions lately.
The CVE program has grown and changed a lot the past few years, and the rules are undergoing a major revision right now (comment period currently) taking in a lot of the feedback. And the rate of CNAs joining has been picking up rapidly as global interest in the program has increased.
No one thinks it is perfect, but that's why a lot of us are active in the working groups and trying to keep moving things forward.
The sane approach is connection draining - you send a `connection: close` response header on the old worker, then finally remove any remaining idle connections at the end of the drain.
In http/2 it's not an issue as it has a way for the server to explicitly say the connection is closed.
I guess you could send the connection header on draining, but anything less than what the big servers do is bound to cause some compatibility problem with some niche client somewhere. I can certainly see why a web server with millions of installs would be reluctant to change bevaviour, even if it is within spec.
I can only guess at the use case here, but maybe something designed from the start as a stateless proxy and not a general purpose web server would be a better fit.
It's clear from this thread that a) Nginx open source will not proceed at its previous pace, b) the forks are for Russia and not for western companies, and c) Caddy seems like absolutely the most sane and responsive place to move.
It is not released any sense of the word. It is not even a complete feature.
I am actually completely shocked this needs to be explained. Legitimate insanity.
By what definition is that not shipped?
> I am actually completely shocked this needs to be explained. Legitimate insanity.
Right back at you.
Are UML diagrams considered in scope too?
Where did you get this info? It might be the feature is actively being worked on and the DoS is a known issue which would be fixed before merge. Lot of projects have contrib folder for random scripts and other things which wouldn't get merged before some review but users are free to run the script if they want to. Experimental compile time build flags are experimental by definition.
Being the same code it'd be darn strange to have the CVE for one and not the other. We did ask ourselves that question and quickly concluded it made no sense.
> [F5] decided to interfere with security policy nginx
> uses for years, ignoring both the policy and developers’ position.
>
> That’s quite understandable: they own the project, and can do
> anything with it, including doing marketing-motivated actions,
> ignoring developers position and community. Still, this
> contradicts our agreement. And, more importantly, I no longer able
> to control which changes are made in nginx within F5, and no longer
> see nginx as a free and open source project developed and
> maintained for the public good.
The whole drama seems very short-sighted from F5's perspective. Maxim was working for you for free for years and you couldn't find some middle ground? I imagine there could have been some page on the free nginx project that listed CVEs that are in the enterprise product but that are not considered CVEs for the open source project given its stated policy of not creating CVEs for experimental features, or something like that.
To nuke the main developer, cause this rift in the community, and create a fork seems like a great microcosm of the general tendency of security leads to wield uncompromising power. I get it. Security is important. But security isn't everything and these little fiefdoms that security leads build up are bureaucratic and annoying.
I hope you understand that these uncompromising policies actually reduce security in the end because 10X developers like Maxim will start to tend to avoid the security team and, in the worst case, hide stuff from their security team. I've seen this play out over and over in large corporations. In that sense, the F5 security team is no different.
But there should be a collaborative, two-way process between security and development. I'm sure security leads will say that they have that, but that's not what I find. Ultimately, if there's an escalation, executives will side with the security lead, so it is a de facto dictatorship even if security leads will tend to avoid the nuclear option. But when you take the nuclear option, as you did in this case, don't be surprised by the consequences.
http://nginx.org/en/docs/quic.html
"Also, since 1.25.0, the QUIC and HTTP/3 support is available in Linux binary packages."
It's still being tested. It's not complete. It's not released. It's not in the distribution. The amount of people that have this feature in the binary AND enabled is less than the amount of people that agree that this should be a CVE.
CVE's are not for tracking bugs in unfinished features.
Should people file a CVE against that?
Ask yourself why this matters? What is the big deal about having a CVE assigned? A CVE is just a unique identifier for a vulnerability so that everyone can refer to the same thing. It helps get word out to users who might be impacted, and we know there are sites using this feature in production - experimental or not. This wasn't dictating what could or could not go into the code - my understanding was the vuln wasn't even in his code, but from another contributor. So, honestly, how does issuing the CVEs impact his work, at all?
That's what I, personally, don't understand. At a functional level, this really has no impact on his work or him personally. This is just documentation of an existing issue and a fix which had to be made, and was being made, CVE or no CVE. And this is worth a fork?
What you're suggesting is the best thing to do is to allow one developer to dictate what should or should not be disclosed to the user base, based on their personal feelings and not an analysis of the impact of that vulnerability on said user base? And if they're inflexible in their view and no compromise can be reached then that's OK?
Sometimes there's just no good compromise to be reached and you end up with one person on one side, and a lot of other people on the other, and if that one person just refuses to budge then it is what it is. Rational people can agree to disagree. In my career there have been many times when I have disagreed with a decision, and I could either make peace with it or I could polish my resume. To me it seems a drastic step to take over something as frankly innocuous as assigning a CVE to an acknowledged vulnerability. Clearly he felt differently, and strongly, on the matter. Maybe he is just very strongly anti-CVE in general, or maybe he'd been feeling the itch to control his own destiny and this was just the spur it took to make the move.
His reasons are his own, and maybe he'll share more in time. I'm comfortable with my personal stance in the matter and the recommendations I made; they conform with my personal and professional morals and ethics. I'm sorry it came to this, but I would not change my recommendation in hindsight as I still feel we did the right thing.
Only time will tell what the results of that are. I think the world is big enough that it doesn't have to be a zero sum game.